linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH V5 00/18] SC7 entry and exit support for Tegra210
@ 2019-06-28  2:12 Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 01/18] irqchip: tegra: Do not disable COP IRQ during suspend Sowjanya Komatineni
                   ` (17 more replies)
  0 siblings, 18 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch series includes Tegra210 deepsleep support with RTC alarm
wake event.

This series also includes save and restore of PLLs, clocks, OSC contexts
for deepsleep exit to normal operation.

This patch series doesn't support 100% suspend/resume to allow fully
functional state upon resume and we are working on some more drivers suspend
and resume implementations.

[V5]: Changes between V4 & V5 are
	- V4 feedback fixes

[V4]: Changes between V3 & V4 are
	- V3 feedback fixes
	- Removed park bits clear for EMMC pads in pinctrl-tegra driver
	  function tegra_pinctrl_clear_parked_bits as based on V3 feedback
	  parked_bit is updated to parked_bitmask to use with DRV_PINGROUP
	  as well and thierry posted patch series for this.
	- Implemented all peripheral clocks save and restore through their
	  corresponding clk_ops save_context and restore_context and removed
	  all direct registers store and restore in clk-tegra210 driver.
	- Created separate patch for fence_delay update during PLLU init based
	  on V3 feedback.
	- Added more comments in tegra210_clk_resume regarding dfll restore
	  sequence and its dependency on peripheral clocks restore.

[V3]: Changes between V2 & V3 are
	- V2 feedback fixes
	- GPIO restore should happen prior to Pinctrl restore to prevent
	  glitch on GPIO lines. So using resume_noirq for gpio tegra to allow
	  gpio resume prior to pinctrl resume.
	- Implemented save_context and restore_context callbacks for clock
	  plls, pll outs and dividers in corresponding drivers.
	  Note: Peripheral clocks and clock enable and reset need to be in
	  Tegra210 clock suspend/resume as they need to be in proper sequence
	  w.r.t DFLL resume for restoring CPU clock.
	- Removed gpio-tegra changes for hierarchical support to have PMC as
	  parent to GPIOs for GPIO wake event support. Thierry is working on
	  gpiolib for some cleanup before adding hierarchical support. So
	  holding on to GPIO wake support for now.

[V2] : V1 feedback fixes
	Patch 0002: This version still using syscore. Thierry suggest not to
	use syscore and waiting on suggestion from Linux Walleij for any better
	way of storing current state of pins before suspend entry and restoring
	them on resume at very early stage. So left this the same way as V1 and
	will address once I get more feedback on this.
	Also need to findout and implement proper way of forcing resume order
	between pinctrl and gpio driver.

[V1]:	Tegra210 SC7 entry and exit thru RTC wake and Power button GPIO wake
	using hierarchical IRQ with PMC as parent to GPIO.




Sowjanya Komatineni (18):
  irqchip: tegra: do not disable COP IRQ during suspend
  pinctrl: tegra: add suspend and resume support
  clk: tegra: save and restore divider rate
  clk: tegra: pllout: save and restore pllout context
  clk: tegra: pll: save and restore pll context
  clk: tegra: save and restore CPU and System clocks context
  clk: tegra: support for saving and restoring OSC context
  clk: tegra: add suspend resume support for DFLL
  clk: tegra: add save and restore context support for peripheral clocks
  clk: tegra210: use fence_udelay during PLLU init
  clk: tegra210: support for Tegra210 clocks suspend and resume
  soc/tegra: pmc: allow support for more tegra wake
  soc/tegra: pmc: add pmc wake support for tegra210
  arm64: tegra: enable wake from deep sleep on RTC alarm.
  soc/tegra: pmc: configure core power request polarity
  soc/tegra: pmc: configure deep sleep control settings
  arm64: dts: tegra210-p2180: Jetson TX1 SC7 timings
  arm64: dts: tegra210-p2180: Jetson nano SC7 timings

 arch/arm64/boot/dts/nvidia/tegra210-p2180.dtsi     |   7 +
 arch/arm64/boot/dts/nvidia/tegra210-p3450-0000.dts |   7 +
 arch/arm64/boot/dts/nvidia/tegra210.dtsi           |   5 +-
 drivers/clk/tegra/clk-dfll.c                       |  78 ++++++++++++
 drivers/clk/tegra/clk-dfll.h                       |   2 +
 drivers/clk/tegra/clk-divider.c                    |  23 ++++
 drivers/clk/tegra/clk-periph-fixed.c               |  31 +++++
 drivers/clk/tegra/clk-periph-gate.c                |  34 +++++
 drivers/clk/tegra/clk-periph.c                     |  43 +++++++
 drivers/clk/tegra/clk-pll-out.c                    |  28 ++++
 drivers/clk/tegra/clk-pll.c                        | 121 +++++++++++++-----
 drivers/clk/tegra/clk-sdmmc-mux.c                  |  30 +++++
 drivers/clk/tegra/clk-tegra-fixed.c                |  14 ++
 drivers/clk/tegra/clk-tegra-super-gen4.c           |   4 -
 drivers/clk/tegra/clk-tegra210.c                   | 128 +++++++++++++++++--
 drivers/clk/tegra/clk.c                            |  94 ++++++++++++++
 drivers/clk/tegra/clk.h                            |  45 ++++++-
 drivers/irqchip/irq-tegra.c                        |  20 ++-
 drivers/pinctrl/tegra/pinctrl-tegra.c              |  52 ++++++++
 drivers/pinctrl/tegra/pinctrl-tegra.h              |   3 +
 drivers/pinctrl/tegra/pinctrl-tegra210.c           |   1 +
 drivers/soc/tegra/pmc.c                            | 141 ++++++++++++++++++++-
 22 files changed, 858 insertions(+), 53 deletions(-)

-- 
2.7.4


^ permalink raw reply	[flat|nested] 111+ messages in thread

* [PATCH V5 01/18] irqchip: tegra: Do not disable COP IRQ during suspend
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support Sowjanya Komatineni
                   ` (16 subsequent siblings)
  17 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

Tegra210 platforms use sc7 entry firmware to program Tegra LP0/SC7 entry
sequence and sc7 entry firmware is run from COP/BPMP-Lite.

So, COP/BPMP-Lite still need IRQ function to finish SC7 suspend sequence
for Tegra210.

This patch has fix for leaving the COP IRQ enabled for Tegra210 during
interrupt controller suspend operation.

Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/irqchip/irq-tegra.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/irqchip/irq-tegra.c b/drivers/irqchip/irq-tegra.c
index e1f771c72fc4..851f88cef508 100644
--- a/drivers/irqchip/irq-tegra.c
+++ b/drivers/irqchip/irq-tegra.c
@@ -44,6 +44,7 @@ static unsigned int num_ictlrs;
 
 struct tegra_ictlr_soc {
 	unsigned int num_ictlrs;
+	bool supports_sc7;
 };
 
 static const struct tegra_ictlr_soc tegra20_ictlr_soc = {
@@ -56,6 +57,7 @@ static const struct tegra_ictlr_soc tegra30_ictlr_soc = {
 
 static const struct tegra_ictlr_soc tegra210_ictlr_soc = {
 	.num_ictlrs = 6,
+	.supports_sc7 = true,
 };
 
 static const struct of_device_id ictlr_matches[] = {
@@ -67,6 +69,7 @@ static const struct of_device_id ictlr_matches[] = {
 
 struct tegra_ictlr_info {
 	void __iomem *base[TEGRA_MAX_NUM_ICTLRS];
+	const struct tegra_ictlr_soc *soc;
 #ifdef CONFIG_PM_SLEEP
 	u32 cop_ier[TEGRA_MAX_NUM_ICTLRS];
 	u32 cop_iep[TEGRA_MAX_NUM_ICTLRS];
@@ -147,8 +150,20 @@ static int tegra_ictlr_suspend(void)
 		lic->cop_ier[i] = readl_relaxed(ictlr + ICTLR_COP_IER);
 		lic->cop_iep[i] = readl_relaxed(ictlr + ICTLR_COP_IEP_CLASS);
 
-		/* Disable COP interrupts */
-		writel_relaxed(~0ul, ictlr + ICTLR_COP_IER_CLR);
+		/*
+		 * AVP/COP/BPMP-Lite is the Tegra boot processor.
+		 *
+		 * Tegra210 system suspend flow uses sc7entry firmware which
+		 * is executed by COP/BPMP and it includes disabling COP IRQ,
+		 * clamping CPU rail, turning off VDD_CPU, and preparing the
+		 * system to go to SC7/LP0.
+		 *
+		 * COP/BPMP wakes up when COP IRQ is triggered and runs
+		 * sc7entry-firmware. So need to keep COP interrupt enabled.
+		 */
+		if (!lic->soc->supports_sc7)
+			/* Disable COP interrupts if SC7 is not supported */
+			writel_relaxed(~0ul, ictlr + ICTLR_COP_IER_CLR);
 
 		/* Disable CPU interrupts */
 		writel_relaxed(~0ul, ictlr + ICTLR_CPU_IER_CLR);
@@ -339,6 +354,7 @@ static int __init tegra_ictlr_init(struct device_node *node,
 		goto out_unmap;
 	}
 
+	lic->soc = soc;
 	tegra_ictlr_syscore_init();
 
 	pr_info("%pOF: %d interrupts forwarded to %pOF\n",
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 01/18] irqchip: tegra: Do not disable COP IRQ during suspend Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-28 11:56   ` Dmitry Osipenko
                     ` (2 more replies)
  2019-06-28  2:12 ` [PATCH V5 03/18] clk: tegra: Save and restore divider rate Sowjanya Komatineni
                   ` (15 subsequent siblings)
  17 siblings, 3 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch adds support for Tegra pinctrl driver suspend and resume.

During suspend, context of all pinctrl registers are stored and
on resume they are all restored to have all the pinmux and pad
configuration for normal operation.

Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/pinctrl/tegra/pinctrl-tegra.c    | 52 ++++++++++++++++++++++++++++++++
 drivers/pinctrl/tegra/pinctrl-tegra.h    |  3 ++
 drivers/pinctrl/tegra/pinctrl-tegra210.c |  1 +
 3 files changed, 56 insertions(+)

diff --git a/drivers/pinctrl/tegra/pinctrl-tegra.c b/drivers/pinctrl/tegra/pinctrl-tegra.c
index 34596b246578..e7c0a1011cba 100644
--- a/drivers/pinctrl/tegra/pinctrl-tegra.c
+++ b/drivers/pinctrl/tegra/pinctrl-tegra.c
@@ -621,6 +621,43 @@ static void tegra_pinctrl_clear_parked_bits(struct tegra_pmx *pmx)
 	}
 }
 
+static int tegra_pinctrl_suspend(struct device *dev)
+{
+	struct tegra_pmx *pmx = dev_get_drvdata(dev);
+	u32 *backup_regs = pmx->backup_regs;
+	u32 *regs;
+	unsigned int i, j;
+
+	for (i = 0; i < pmx->nbanks; i++) {
+		regs = pmx->regs[i];
+		for (j = 0; j < pmx->reg_bank_size[i] / 4; j++)
+			*backup_regs++ = readl(regs++);
+	}
+
+	return pinctrl_force_sleep(pmx->pctl);
+}
+
+static int tegra_pinctrl_resume(struct device *dev)
+{
+	struct tegra_pmx *pmx = dev_get_drvdata(dev);
+	u32 *backup_regs = pmx->backup_regs;
+	u32 *regs;
+	unsigned int i, j;
+
+	for (i = 0; i < pmx->nbanks; i++) {
+		regs = pmx->regs[i];
+		for (j = 0; j < pmx->reg_bank_size[i] / 4; j++)
+			writel(*backup_regs++, regs++);
+	}
+
+	return 0;
+}
+
+const struct dev_pm_ops tegra_pinctrl_pm = {
+	.suspend = &tegra_pinctrl_suspend,
+	.resume = &tegra_pinctrl_resume
+};
+
 static bool gpio_node_has_range(const char *compatible)
 {
 	struct device_node *np;
@@ -645,6 +682,7 @@ int tegra_pinctrl_probe(struct platform_device *pdev,
 	int i;
 	const char **group_pins;
 	int fn, gn, gfn;
+	unsigned long backup_regs_size = 0;
 
 	pmx = devm_kzalloc(&pdev->dev, sizeof(*pmx), GFP_KERNEL);
 	if (!pmx)
@@ -697,6 +735,7 @@ int tegra_pinctrl_probe(struct platform_device *pdev,
 		res = platform_get_resource(pdev, IORESOURCE_MEM, i);
 		if (!res)
 			break;
+		backup_regs_size += resource_size(res);
 	}
 	pmx->nbanks = i;
 
@@ -705,11 +744,24 @@ int tegra_pinctrl_probe(struct platform_device *pdev,
 	if (!pmx->regs)
 		return -ENOMEM;
 
+	pmx->reg_bank_size = devm_kcalloc(&pdev->dev, pmx->nbanks,
+					  sizeof(*pmx->reg_bank_size),
+					  GFP_KERNEL);
+	if (!pmx->reg_bank_size)
+		return -ENOMEM;
+
+	pmx->backup_regs = devm_kzalloc(&pdev->dev, backup_regs_size,
+					GFP_KERNEL);
+	if (!pmx->backup_regs)
+		return -ENOMEM;
+
 	for (i = 0; i < pmx->nbanks; i++) {
 		res = platform_get_resource(pdev, IORESOURCE_MEM, i);
 		pmx->regs[i] = devm_ioremap_resource(&pdev->dev, res);
 		if (IS_ERR(pmx->regs[i]))
 			return PTR_ERR(pmx->regs[i]);
+
+		pmx->reg_bank_size[i] = resource_size(res);
 	}
 
 	pmx->pctl = devm_pinctrl_register(&pdev->dev, &tegra_pinctrl_desc, pmx);
diff --git a/drivers/pinctrl/tegra/pinctrl-tegra.h b/drivers/pinctrl/tegra/pinctrl-tegra.h
index 287702660783..55456f8d44cf 100644
--- a/drivers/pinctrl/tegra/pinctrl-tegra.h
+++ b/drivers/pinctrl/tegra/pinctrl-tegra.h
@@ -17,6 +17,8 @@ struct tegra_pmx {
 
 	int nbanks;
 	void __iomem **regs;
+	size_t *reg_bank_size;
+	u32 *backup_regs;
 };
 
 enum tegra_pinconf_param {
@@ -193,6 +195,7 @@ struct tegra_pinctrl_soc_data {
 	bool drvtype_in_mux;
 };
 
+extern const struct dev_pm_ops tegra_pinctrl_pm;
 int tegra_pinctrl_probe(struct platform_device *pdev,
 			const struct tegra_pinctrl_soc_data *soc_data);
 #endif
diff --git a/drivers/pinctrl/tegra/pinctrl-tegra210.c b/drivers/pinctrl/tegra/pinctrl-tegra210.c
index 0b56ad5c9c1c..edd3f4606cdb 100644
--- a/drivers/pinctrl/tegra/pinctrl-tegra210.c
+++ b/drivers/pinctrl/tegra/pinctrl-tegra210.c
@@ -1571,6 +1571,7 @@ static struct platform_driver tegra210_pinctrl_driver = {
 	.driver = {
 		.name = "tegra210-pinctrl",
 		.of_match_table = tegra210_pinctrl_of_match,
+		.pm = &tegra_pinctrl_pm,
 	},
 	.probe = tegra210_pinctrl_probe,
 };
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 03/18] clk: tegra: Save and restore divider rate
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 01/18] irqchip: tegra: Do not disable COP IRQ during suspend Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 04/18] clk: tegra: pllout: Save and restore pllout context Sowjanya Komatineni
                   ` (14 subsequent siblings)
  17 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch implements context save and restore for clock divider.

During system suspend, core power goes off and looses the settings
of the Tegra CAR controller registers.

So during suspend entry the context of clock divider is saved and
on resume context is restored back for normal operation.

Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/clk/tegra/clk-divider.c | 23 +++++++++++++++++++++++
 drivers/clk/tegra/clk.h         |  2 ++
 2 files changed, 25 insertions(+)

diff --git a/drivers/clk/tegra/clk-divider.c b/drivers/clk/tegra/clk-divider.c
index e76731fb7d69..ecb7ff9ce97e 100644
--- a/drivers/clk/tegra/clk-divider.c
+++ b/drivers/clk/tegra/clk-divider.c
@@ -109,10 +109,33 @@ static int clk_frac_div_set_rate(struct clk_hw *hw, unsigned long rate,
 	return 0;
 }
 
+static int clk_divider_save_context(struct clk_hw *hw)
+{
+	struct tegra_clk_frac_div *divider = to_clk_frac_div(hw);
+	struct clk_hw *parent = clk_hw_get_parent(hw);
+	unsigned long parent_rate = clk_hw_get_rate(parent);
+
+	divider->rate = clk_frac_div_recalc_rate(hw, parent_rate);
+
+	return 0;
+}
+
+static void clk_divider_restore_context(struct clk_hw *hw)
+{
+	struct tegra_clk_frac_div *divider = to_clk_frac_div(hw);
+	struct clk_hw *parent = clk_hw_get_parent(hw);
+	unsigned long parent_rate = clk_hw_get_rate(parent);
+
+	if (clk_frac_div_set_rate(hw, divider->rate, parent_rate) < 0)
+		WARN_ON(1);
+}
+
 const struct clk_ops tegra_clk_frac_div_ops = {
 	.recalc_rate = clk_frac_div_recalc_rate,
 	.set_rate = clk_frac_div_set_rate,
 	.round_rate = clk_frac_div_round_rate,
+	.save_context = clk_divider_save_context,
+	.restore_context = clk_divider_restore_context,
 };
 
 struct clk *tegra_clk_register_divider(const char *name,
diff --git a/drivers/clk/tegra/clk.h b/drivers/clk/tegra/clk.h
index 905bf1096558..83623f5f55f3 100644
--- a/drivers/clk/tegra/clk.h
+++ b/drivers/clk/tegra/clk.h
@@ -42,6 +42,7 @@ struct clk *tegra_clk_register_sync_source(const char *name,
  * @width:	width of the divider bit field
  * @frac_width:	width of the fractional bit field
  * @lock:	register lock
+ * @rate:	rate during suspend and resume
  *
  * Flags:
  * TEGRA_DIVIDER_ROUND_UP - This flags indicates to round up the divider value.
@@ -62,6 +63,7 @@ struct tegra_clk_frac_div {
 	u8		width;
 	u8		frac_width;
 	spinlock_t	*lock;
+	unsigned long	rate;
 };
 
 #define to_clk_frac_div(_hw) container_of(_hw, struct tegra_clk_frac_div, hw)
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 04/18] clk: tegra: pllout: Save and restore pllout context
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (2 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 03/18] clk: tegra: Save and restore divider rate Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 05/18] clk: tegra: pll: Save and restore pll context Sowjanya Komatineni
                   ` (13 subsequent siblings)
  17 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch implements save and restore of pllout context.

During system suspend, core power goes off and looses the settings
of the Tegra CAR controller registers.

So during suspend entry the state of pllout is saved and on resume
it is restored back to have pllout in same state as before suspend.

pllout rate is saved and restore in clock divider so it will be at
same rate as before suspend when pllout state is restored.

Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/clk/tegra/clk-pll-out.c  | 28 ++++++++++++++++++++++++++++
 drivers/clk/tegra/clk-tegra210.c |  3 ++-
 drivers/clk/tegra/clk.h          |  9 +++++++++
 3 files changed, 39 insertions(+), 1 deletion(-)

diff --git a/drivers/clk/tegra/clk-pll-out.c b/drivers/clk/tegra/clk-pll-out.c
index 35f2bf00e1e6..8f26a7e3e579 100644
--- a/drivers/clk/tegra/clk-pll-out.c
+++ b/drivers/clk/tegra/clk-pll-out.c
@@ -69,10 +69,38 @@ static void clk_pll_out_disable(struct clk_hw *hw)
 		spin_unlock_irqrestore(pll_out->lock, flags);
 }
 
+static int tegra_clk_pll_out_save_context(struct clk_hw *hw)
+{
+	struct tegra_clk_pll_out *pll_out = to_clk_pll_out(hw);
+
+	if (pll_out->flags & TEGRA_PLLRE_OUT)
+		pll_out->pllout_ctx = readl_relaxed(pll_out->reg);
+	else
+		pll_out->pllout_ctx = clk_hw_get_rate(hw);
+
+	return 0;
+}
+
+static void tegra_clk_pll_out_restore_context(struct clk_hw *hw)
+{
+	struct tegra_clk_pll_out *pll_out = to_clk_pll_out(hw);
+
+	if (pll_out->flags & TEGRA_PLLRE_OUT) {
+		writel_relaxed(pll_out->pllout_ctx, pll_out->reg);
+	} else {
+		if (!__clk_get_enable_count(hw->clk))
+			clk_pll_out_disable(hw);
+		else
+			clk_pll_out_enable(hw);
+	}
+}
+
 const struct clk_ops tegra_clk_pll_out_ops = {
 	.is_enabled = clk_pll_out_is_enabled,
 	.enable = clk_pll_out_enable,
 	.disable = clk_pll_out_disable,
+	.save_context = tegra_clk_pll_out_save_context,
+	.restore_context = tegra_clk_pll_out_restore_context,
 };
 
 struct clk *tegra_clk_register_pll_out(const char *name,
diff --git a/drivers/clk/tegra/clk-tegra210.c b/drivers/clk/tegra/clk-tegra210.c
index ac1d27a8c650..cbd77658dcf7 100644
--- a/drivers/clk/tegra/clk-tegra210.c
+++ b/drivers/clk/tegra/clk-tegra210.c
@@ -3195,7 +3195,8 @@ static void __init tegra210_pll_init(void __iomem *clk_base,
 					 8, 8, 1, NULL);
 	clk = tegra_clk_register_pll_out("pll_re_out1", "pll_re_out1_div",
 					 clk_base + PLLRE_OUT1, 1, 0,
-					 CLK_SET_RATE_PARENT, 0, NULL);
+					 CLK_SET_RATE_PARENT, TEGRA_PLLRE_OUT,
+					 NULL);
 	clks[TEGRA210_CLK_PLL_RE_OUT1] = clk;
 
 	/* PLLE */
diff --git a/drivers/clk/tegra/clk.h b/drivers/clk/tegra/clk.h
index 83623f5f55f3..fb29a8c27873 100644
--- a/drivers/clk/tegra/clk.h
+++ b/drivers/clk/tegra/clk.h
@@ -439,6 +439,12 @@ struct clk *tegra_clk_register_pllu_tegra210(const char *name,
  * @rst_bit_idx:	bit to reset PLL divider
  * @lock:		register lock
  * @flags:		hardware-specific flags
+ * @pllout_ctx:		pllout context to save and restore during suspend
+ *			and resume
+ *
+ * Flags:
+ * TEGRA_PLLRE_OUT - This flag indicates that it is PLLRE_OUT and is used to
+ *		     identify PLLRE_OUT during clk_pll_out save and restore.
  */
 struct tegra_clk_pll_out {
 	struct clk_hw	hw;
@@ -447,8 +453,11 @@ struct tegra_clk_pll_out {
 	u8		rst_bit_idx;
 	spinlock_t	*lock;
 	u8		flags;
+	unsigned int	pllout_ctx;
 };
 
+#define TEGRA_PLLRE_OUT BIT(0)
+
 #define to_clk_pll_out(_hw) container_of(_hw, struct tegra_clk_pll_out, hw)
 
 extern const struct clk_ops tegra_clk_pll_out_ops;
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 05/18] clk: tegra: pll: Save and restore pll context
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (3 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 04/18] clk: tegra: pllout: Save and restore pllout context Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 06/18] clk: tegra: Save and restore CPU and System clocks context Sowjanya Komatineni
                   ` (12 subsequent siblings)
  17 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch implements save and restore of PLL context.

During system suspend, core power goes off and looses the settings
of the Tegra CAR controller registers.

So during suspend entry pll rate is stored and on resume it is
restored back along with its state.

Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/clk/tegra/clk-pll.c      | 121 ++++++++++++++++++++++++++++-----------
 drivers/clk/tegra/clk-tegra210.c |   2 +-
 drivers/clk/tegra/clk.h          |  10 +++-
 3 files changed, 99 insertions(+), 34 deletions(-)

diff --git a/drivers/clk/tegra/clk-pll.c b/drivers/clk/tegra/clk-pll.c
index 1583f5fc992f..3686ada022aa 100644
--- a/drivers/clk/tegra/clk-pll.c
+++ b/drivers/clk/tegra/clk-pll.c
@@ -1008,6 +1008,59 @@ static unsigned long clk_plle_recalc_rate(struct clk_hw *hw,
 	return rate;
 }
 
+void tegra_clk_sync_state_pll(struct clk_hw *hw)
+{
+	if (!__clk_get_enable_count(hw->clk))
+		clk_pll_disable(hw);
+	else
+		clk_pll_enable(hw);
+}
+
+static int tegra_clk_pll_save_context(struct clk_hw *hw)
+{
+	struct tegra_clk_pll *pll = to_clk_pll(hw);
+	u32 val = 0;
+
+	pll->rate = clk_hw_get_rate(hw);
+
+	if (pll->params->flags & TEGRA_PLLMB)
+		val = pll_readl_base(pll);
+	else if (pll->params->flags & TEGRA_PLLRE)
+		val = pll_readl_base(pll) & divp_mask_shifted(pll);
+
+	pll->pllbase_ctx = val;
+
+	return 0;
+}
+
+static void tegra_clk_pll_restore_context(struct clk_hw *hw)
+{
+	struct tegra_clk_pll *pll = to_clk_pll(hw);
+	struct clk_hw *parent = clk_hw_get_parent(hw);
+	unsigned long parent_rate = clk_hw_get_rate(parent);
+	u32 val;
+
+	if (clk_pll_is_enabled(hw))
+		return;
+
+	if (pll->params->flags & TEGRA_PLLMB) {
+		pll_writel_base(pll->pllbase_ctx, pll);
+	} else if (pll->params->flags & TEGRA_PLLRE) {
+		val = pll_readl_base(pll);
+		val &= ~(divp_mask_shifted(pll));
+		pll_writel_base(pll->pllbase_ctx | val, pll);
+	}
+
+	if (pll->params->set_defaults)
+		pll->params->set_defaults(pll);
+
+	clk_pll_set_rate(hw, pll->rate, parent_rate);
+
+	/* do not sync pllx state here. pllx is sync'd after dfll resume */
+	if (!(pll->params->flags & TEGRA_PLLX))
+		tegra_clk_sync_state_pll(hw);
+}
+
 const struct clk_ops tegra_clk_pll_ops = {
 	.is_enabled = clk_pll_is_enabled,
 	.enable = clk_pll_enable,
@@ -1015,6 +1068,8 @@ const struct clk_ops tegra_clk_pll_ops = {
 	.recalc_rate = clk_pll_recalc_rate,
 	.round_rate = clk_pll_round_rate,
 	.set_rate = clk_pll_set_rate,
+	.save_context = tegra_clk_pll_save_context,
+	.restore_context = tegra_clk_pll_restore_context,
 };
 
 const struct clk_ops tegra_clk_plle_ops = {
@@ -1802,6 +1857,27 @@ static int clk_pllu_tegra114_enable(struct clk_hw *hw)
 
 	return ret;
 }
+
+static void _clk_plle_tegra_init_parent(struct tegra_clk_pll *pll)
+{
+	u32 val, val_aux;
+
+	/* ensure parent is set to pll_ref */
+	val = pll_readl_base(pll);
+	val_aux = pll_readl(pll->params->aux_reg, pll);
+
+	if (val & PLL_BASE_ENABLE) {
+		if ((val_aux & PLLE_AUX_PLLRE_SEL) ||
+		    (val_aux & PLLE_AUX_PLLP_SEL))
+			WARN(1, "pll_e enabled with unsupported parent %s\n",
+			     (val_aux & PLLE_AUX_PLLP_SEL) ? "pllp_out0" :
+			     "pll_re_vco");
+	} else {
+		val_aux &= ~(PLLE_AUX_PLLRE_SEL | PLLE_AUX_PLLP_SEL);
+		pll_writel(val_aux, pll->params->aux_reg, pll);
+		fence_udelay(1, pll->clk_base);
+	}
+}
 #endif
 
 static struct tegra_clk_pll *_tegra_init_pll(void __iomem *clk_base,
@@ -2214,27 +2290,12 @@ struct clk *tegra_clk_register_plle_tegra114(const char *name,
 {
 	struct tegra_clk_pll *pll;
 	struct clk *clk;
-	u32 val, val_aux;
 
 	pll = _tegra_init_pll(clk_base, NULL, pll_params, lock);
 	if (IS_ERR(pll))
 		return ERR_CAST(pll);
 
-	/* ensure parent is set to pll_re_vco */
-
-	val = pll_readl_base(pll);
-	val_aux = pll_readl(pll_params->aux_reg, pll);
-
-	if (val & PLL_BASE_ENABLE) {
-		if ((val_aux & PLLE_AUX_PLLRE_SEL) ||
-			(val_aux & PLLE_AUX_PLLP_SEL))
-			WARN(1, "pll_e enabled with unsupported parent %s\n",
-			  (val_aux & PLLE_AUX_PLLP_SEL) ? "pllp_out0" :
-					"pll_re_vco");
-	} else {
-		val_aux &= ~(PLLE_AUX_PLLRE_SEL | PLLE_AUX_PLLP_SEL);
-		pll_writel(val_aux, pll_params->aux_reg, pll);
-	}
+	_clk_plle_tegra_init_parent(pll);
 
 	clk = _tegra_clk_register_pll(pll, name, parent_name, flags,
 				      &tegra_clk_plle_tegra114_ops);
@@ -2276,6 +2337,8 @@ static const struct clk_ops tegra_clk_pllss_ops = {
 	.recalc_rate = clk_pll_recalc_rate,
 	.round_rate = clk_pll_ramp_round_rate,
 	.set_rate = clk_pllxc_set_rate,
+	.save_context = tegra_clk_pll_save_context,
+	.restore_context = tegra_clk_pll_restore_context,
 };
 
 struct clk *tegra_clk_register_pllss(const char *name, const char *parent_name,
@@ -2375,6 +2438,7 @@ struct clk *tegra_clk_register_pllre_tegra210(const char *name,
 		pll_params->vco_min = pll_params->adjust_vco(pll_params,
 							     parent_rate);
 
+	pll_params->flags |= TEGRA_PLLRE;
 	pll = _tegra_init_pll(clk_base, pmc, pll_params, lock);
 	if (IS_ERR(pll))
 		return ERR_CAST(pll);
@@ -2520,11 +2584,19 @@ static void clk_plle_tegra210_disable(struct clk_hw *hw)
 		spin_unlock_irqrestore(pll->lock, flags);
 }
 
+static void tegra_clk_plle_t210_restore_context(struct clk_hw *hw)
+{
+	struct tegra_clk_pll *pll = to_clk_pll(hw);
+
+	_clk_plle_tegra_init_parent(pll);
+}
+
 static const struct clk_ops tegra_clk_plle_tegra210_ops = {
 	.is_enabled =  clk_plle_tegra210_is_enabled,
 	.enable = clk_plle_tegra210_enable,
 	.disable = clk_plle_tegra210_disable,
 	.recalc_rate = clk_pll_recalc_rate,
+	.restore_context = tegra_clk_plle_t210_restore_context,
 };
 
 struct clk *tegra_clk_register_plle_tegra210(const char *name,
@@ -2535,27 +2607,12 @@ struct clk *tegra_clk_register_plle_tegra210(const char *name,
 {
 	struct tegra_clk_pll *pll;
 	struct clk *clk;
-	u32 val, val_aux;
 
 	pll = _tegra_init_pll(clk_base, NULL, pll_params, lock);
 	if (IS_ERR(pll))
 		return ERR_CAST(pll);
 
-	/* ensure parent is set to pll_re_vco */
-
-	val = pll_readl_base(pll);
-	val_aux = pll_readl(pll_params->aux_reg, pll);
-
-	if (val & PLLE_BASE_ENABLE) {
-		if ((val_aux & PLLE_AUX_PLLRE_SEL) ||
-			(val_aux & PLLE_AUX_PLLP_SEL))
-			WARN(1, "pll_e enabled with unsupported parent %s\n",
-			  (val_aux & PLLE_AUX_PLLP_SEL) ? "pllp_out0" :
-					"pll_re_vco");
-	} else {
-		val_aux &= ~(PLLE_AUX_PLLRE_SEL | PLLE_AUX_PLLP_SEL);
-		pll_writel(val_aux, pll_params->aux_reg, pll);
-	}
+	_clk_plle_tegra_init_parent(pll);
 
 	clk = _tegra_clk_register_pll(pll, name, parent_name, flags,
 				      &tegra_clk_plle_tegra210_ops);
diff --git a/drivers/clk/tegra/clk-tegra210.c b/drivers/clk/tegra/clk-tegra210.c
index cbd77658dcf7..7f90442462af 100644
--- a/drivers/clk/tegra/clk-tegra210.c
+++ b/drivers/clk/tegra/clk-tegra210.c
@@ -1597,7 +1597,7 @@ static struct tegra_clk_pll_params pll_x_params = {
 	.pdiv_tohw = pll_qlin_pdiv_to_hw,
 	.div_nmp = &pllx_nmp,
 	.freq_table = pll_x_freq_table,
-	.flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE,
+	.flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE | TEGRA_PLLX,
 	.dyn_ramp = tegra210_pllx_dyn_ramp,
 	.set_defaults = tegra210_pllx_set_defaults,
 	.calc_rate = tegra210_pll_fixed_mdiv_cfg,
diff --git a/drivers/clk/tegra/clk.h b/drivers/clk/tegra/clk.h
index fb29a8c27873..8532f5150091 100644
--- a/drivers/clk/tegra/clk.h
+++ b/drivers/clk/tegra/clk.h
@@ -235,6 +235,8 @@ struct tegra_clk_pll;
  * TEGRA_PLLMB - PLLMB has should be treated similar to PLLM. This
  *     flag indicated that it is PLLMB.
  * TEGRA_PLL_VCO_OUT - Used to indicate that the PLL has a VCO output
+ * TEGRA_PLLRE - Used to indicate that it is PLLRE.
+ * TEGRA_PLLX - Used to indicate that it is PLLX.
  */
 struct tegra_clk_pll_params {
 	unsigned long	input_min;
@@ -301,6 +303,8 @@ struct tegra_clk_pll_params {
 #define TEGRA_MDIV_NEW BIT(11)
 #define TEGRA_PLLMB BIT(12)
 #define TEGRA_PLL_VCO_OUT BIT(13)
+#define TEGRA_PLLRE BIT(14)
+#define TEGRA_PLLX BIT(15)
 
 /**
  * struct tegra_clk_pll - Tegra PLL clock
@@ -310,6 +314,8 @@ struct tegra_clk_pll_params {
  * @pmc:	address of PMC, required to read override bits
  * @lock:	register lock
  * @params:	PLL parameters
+ * @rate:	rate during system suspend and resume
+ * @pllbase_ctx: pll base register value during suspend and resume
  */
 struct tegra_clk_pll {
 	struct clk_hw	hw;
@@ -317,6 +323,8 @@ struct tegra_clk_pll {
 	void __iomem	*pmc;
 	spinlock_t	*lock;
 	struct tegra_clk_pll_params	*params;
+	unsigned long	rate;
+	u32	pllbase_ctx;
 };
 
 #define to_clk_pll(_hw) container_of(_hw, struct tegra_clk_pll, hw)
@@ -840,7 +848,7 @@ u16 tegra_pll_get_fixed_mdiv(struct clk_hw *hw, unsigned long input_rate);
 int tegra_pll_p_div_to_hw(struct tegra_clk_pll *pll, u8 p_div);
 int div_frac_get(unsigned long rate, unsigned parent_rate, u8 width,
 		 u8 frac_width, u8 flags);
-
+void tegra_clk_sync_state_pll(struct clk_hw *hw);
 
 /* Combined read fence with delay */
 #define fence_udelay(delay, reg)	\
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 06/18] clk: tegra: Save and restore CPU and System clocks context
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (4 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 05/18] clk: tegra: pll: Save and restore pll context Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-29 13:33   ` Dmitry Osipenko
  2019-06-29 15:26   ` Dmitry Osipenko
  2019-06-28  2:12 ` [PATCH V5 07/18] clk: tegra: Support for saving and restoring OSC context Sowjanya Komatineni
                   ` (11 subsequent siblings)
  17 siblings, 2 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

During system suspend state, core power goes off and looses all the
CAR controller register settings.

This patch creates APIs for saving and restoring the context of Tegra
CPUG, CPULP and SCLK.

CPU and System clock context includes
- CPUG, CPULP, and SCLK burst policy settings for clock sourcea of all
  their normal states.
- SCLK divisor and System clock rate for restoring SCLK, AHB and APB
  rates on resume.
- OSC_DIV settings which are used as reference clock input to some PLLs.
- SPARE_REG and CLK_MASK settings.

These APIs are used in Tegra210 clock driver during suspend and resume
operation.

Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/clk/tegra/clk-tegra-super-gen4.c |  4 --
 drivers/clk/tegra/clk.c                  | 80 ++++++++++++++++++++++++++++++++
 drivers/clk/tegra/clk.h                  | 14 ++++++
 3 files changed, 94 insertions(+), 4 deletions(-)

diff --git a/drivers/clk/tegra/clk-tegra-super-gen4.c b/drivers/clk/tegra/clk-tegra-super-gen4.c
index cdfe7c9697e1..ed69ec4d883e 100644
--- a/drivers/clk/tegra/clk-tegra-super-gen4.c
+++ b/drivers/clk/tegra/clk-tegra-super-gen4.c
@@ -19,10 +19,6 @@
 #define PLLX_MISC2 0x514
 #define PLLX_MISC3 0x518
 
-#define CCLKG_BURST_POLICY 0x368
-#define CCLKLP_BURST_POLICY 0x370
-#define SCLK_BURST_POLICY 0x028
-#define SYSTEM_CLK_RATE 0x030
 #define SCLK_DIVIDER 0x2c
 
 static DEFINE_SPINLOCK(sysrate_lock);
diff --git a/drivers/clk/tegra/clk.c b/drivers/clk/tegra/clk.c
index 573e3c967ae1..9e863362d2bf 100644
--- a/drivers/clk/tegra/clk.c
+++ b/drivers/clk/tegra/clk.c
@@ -70,6 +70,12 @@ static struct clk **clks;
 static int clk_num;
 static struct clk_onecell_data clk_data;
 
+static u32 cclkg_burst_policy_ctx[2];
+static u32 cclklp_burst_policy_ctx[2];
+static u32 sclk_burst_policy_ctx[2];
+static u32 sys_clk_divisor_ctx, system_rate_ctx;
+static u32 spare_ctx, misc_clk_enb_ctx, clk_arm_ctx;
+
 /* Handlers for SoC-specific reset lines */
 static int (*special_reset_assert)(unsigned long);
 static int (*special_reset_deassert)(unsigned long);
@@ -199,6 +205,80 @@ const struct tegra_clk_periph_regs *get_reg_bank(int clkid)
 	}
 }
 
+void tegra_cclkg_burst_policy_save_context(void)
+{
+	unsigned int i;
+
+	for (i = 0; i < BURST_POLICY_REG_SIZE; i++)
+		cclkg_burst_policy_ctx[i] = readl_relaxed(clk_base +
+							  CCLKG_BURST_POLICY +
+							  (i * 4));
+}
+
+void tegra_cclkg_burst_policy_restore_context(void)
+{
+	unsigned int i;
+
+	for (i = 0; i < BURST_POLICY_REG_SIZE; i++)
+		writel_relaxed(cclkg_burst_policy_ctx[i],
+			       clk_base + CCLKG_BURST_POLICY + (i * 4));
+
+	fence_udelay(2, clk_base);
+}
+
+void tegra_sclk_cclklp_burst_policy_save_context(void)
+{
+	unsigned int i;
+
+	for (i = 0; i < BURST_POLICY_REG_SIZE; i++) {
+		cclklp_burst_policy_ctx[i] = readl_relaxed(clk_base +
+							  CCLKLP_BURST_POLICY +
+							  (i * 4));
+
+		sclk_burst_policy_ctx[i] = readl_relaxed(clk_base +
+							  SCLK_BURST_POLICY +
+							  (i * 4));
+	}
+
+	sys_clk_divisor_ctx = readl_relaxed(clk_base + SYS_CLK_DIV);
+	system_rate_ctx = readl_relaxed(clk_base + SYSTEM_CLK_RATE);
+	spare_ctx = readl_relaxed(clk_base + SPARE_REG0);
+	misc_clk_enb_ctx = readl_relaxed(clk_base + MISC_CLK_ENB);
+	clk_arm_ctx = readl_relaxed(clk_base + CLK_MASK_ARM);
+}
+
+void tegra_sclk_cpulp_burst_policy_restore_context(void)
+{
+	unsigned int i;
+	u32 val;
+
+	/*
+	 * resume SCLK and CPULP clocks
+	 * for SCLk, set safe dividers values first and then restore source
+	 * and dividers
+	 */
+
+	writel_relaxed(0x1, clk_base + SYSTEM_CLK_RATE);
+	val = readl_relaxed(clk_base + SYS_CLK_DIV);
+	if (val < sys_clk_divisor_ctx)
+		writel_relaxed(sys_clk_divisor_ctx, clk_base + SYS_CLK_DIV);
+
+	fence_udelay(2, clk_base);
+
+	for (i = 0; i < BURST_POLICY_REG_SIZE; i++) {
+		writel_relaxed(cclklp_burst_policy_ctx[i],
+			       clk_base + CCLKLP_BURST_POLICY + (i * 4));
+		writel_relaxed(sclk_burst_policy_ctx[i],
+			       clk_base + SCLK_BURST_POLICY + (i * 4));
+	}
+
+	writel_relaxed(sys_clk_divisor_ctx, clk_base + SYS_CLK_DIV);
+	writel_relaxed(system_rate_ctx, clk_base + SYSTEM_CLK_RATE);
+	writel_relaxed(spare_ctx, clk_base + SPARE_REG0);
+	writel_relaxed(misc_clk_enb_ctx, clk_base + MISC_CLK_ENB);
+	writel_relaxed(clk_arm_ctx, clk_base + CLK_MASK_ARM);
+}
+
 struct clk ** __init tegra_clk_init(void __iomem *regs, int num, int banks)
 {
 	clk_base = regs;
diff --git a/drivers/clk/tegra/clk.h b/drivers/clk/tegra/clk.h
index 8532f5150091..c66b0a73bb01 100644
--- a/drivers/clk/tegra/clk.h
+++ b/drivers/clk/tegra/clk.h
@@ -10,6 +10,16 @@
 #include <linux/clkdev.h>
 #include <linux/delay.h>
 
+#define SCLK_BURST_POLICY	0x28
+#define SYSTEM_CLK_RATE		0x30
+#define CLK_MASK_ARM		0x44
+#define MISC_CLK_ENB		0x48
+#define CCLKG_BURST_POLICY	0x368
+#define CCLKLP_BURST_POLICY	0x370
+#define SYS_CLK_DIV		0x400
+#define SPARE_REG0		0x55c
+#define BURST_POLICY_REG_SIZE	2
+
 /**
  * struct tegra_clk_sync_source - external clock source from codec
  *
@@ -849,6 +859,10 @@ int tegra_pll_p_div_to_hw(struct tegra_clk_pll *pll, u8 p_div);
 int div_frac_get(unsigned long rate, unsigned parent_rate, u8 width,
 		 u8 frac_width, u8 flags);
 void tegra_clk_sync_state_pll(struct clk_hw *hw);
+void tegra_cclkg_burst_policy_save_context(void);
+void tegra_cclkg_burst_policy_restore_context(void);
+void tegra_sclk_cclklp_burst_policy_save_context(void);
+void tegra_sclk_cpulp_burst_policy_restore_context(void);
 
 /* Combined read fence with delay */
 #define fence_udelay(delay, reg)	\
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 07/18] clk: tegra: Support for saving and restoring OSC context
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (5 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 06/18] clk: tegra: Save and restore CPU and System clocks context Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 08/18] clk: tegra: Add suspend resume support for DFLL Sowjanya Komatineni
                   ` (10 subsequent siblings)
  17 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch adds support for storing OSC clock frequency and the
drive-strength during OSC clock init and creates an API to restore
OSC control register value from the saved context.

This API is invoked by Tegra210 clock driver during system resume
to restore the  OSC clock settings.

Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/clk/tegra/clk-tegra-fixed.c | 14 ++++++++++++++
 drivers/clk/tegra/clk.h             |  1 +
 2 files changed, 15 insertions(+)

diff --git a/drivers/clk/tegra/clk-tegra-fixed.c b/drivers/clk/tegra/clk-tegra-fixed.c
index 8d91b2b191cf..e8df0ccbffd0 100644
--- a/drivers/clk/tegra/clk-tegra-fixed.c
+++ b/drivers/clk/tegra/clk-tegra-fixed.c
@@ -17,7 +17,10 @@
 #define OSC_CTRL			0x50
 #define OSC_CTRL_OSC_FREQ_SHIFT		28
 #define OSC_CTRL_PLL_REF_DIV_SHIFT	26
+#define OSC_CTRL_MASK			(0x3f2 |	\
+					(0xf << OSC_CTRL_OSC_FREQ_SHIFT))
 
+static u32 osc_ctrl_ctx;
 int __init tegra_osc_clk_init(void __iomem *clk_base, struct tegra_clk *clks,
 			      unsigned long *input_freqs, unsigned int num,
 			      unsigned int clk_m_div, unsigned long *osc_freq,
@@ -29,6 +32,7 @@ int __init tegra_osc_clk_init(void __iomem *clk_base, struct tegra_clk *clks,
 	unsigned osc_idx;
 
 	val = readl_relaxed(clk_base + OSC_CTRL);
+	osc_ctrl_ctx = val & OSC_CTRL_MASK;
 	osc_idx = val >> OSC_CTRL_OSC_FREQ_SHIFT;
 
 	if (osc_idx < num)
@@ -96,3 +100,13 @@ void __init tegra_fixed_clk_init(struct tegra_clk *tegra_clks)
 		*dt_clk = clk;
 	}
 }
+
+void tegra_clk_osc_resume(void __iomem *clk_base)
+{
+	u32 val;
+
+	val = readl_relaxed(clk_base + OSC_CTRL) & ~OSC_CTRL_MASK;
+	val |= osc_ctrl_ctx;
+	writel_relaxed(val, clk_base + OSC_CTRL);
+	fence_udelay(2, clk_base);
+}
diff --git a/drivers/clk/tegra/clk.h b/drivers/clk/tegra/clk.h
index c66b0a73bb01..a687ed6127b6 100644
--- a/drivers/clk/tegra/clk.h
+++ b/drivers/clk/tegra/clk.h
@@ -863,6 +863,7 @@ void tegra_cclkg_burst_policy_save_context(void);
 void tegra_cclkg_burst_policy_restore_context(void);
 void tegra_sclk_cclklp_burst_policy_save_context(void);
 void tegra_sclk_cpulp_burst_policy_restore_context(void);
+void tegra_clk_osc_resume(void __iomem *clk_base);
 
 /* Combined read fence with delay */
 #define fence_udelay(delay, reg)	\
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 08/18] clk: tegra: Add suspend resume support for DFLL
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (6 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 07/18] clk: tegra: Support for saving and restoring OSC context Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-29 13:28   ` Dmitry Osipenko
  2019-06-28  2:12 ` [PATCH V5 09/18] clk: tegra: Add save and restore context support for peripheral clocks Sowjanya Komatineni
                   ` (9 subsequent siblings)
  17 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch creates APIs for supporting Tegra210 clock driver to
perform DFLL suspend and resume operation.

During suspend, DFLL mode is saved and on resume Tegra210 clock driver
invokes DFLL resume API to re-initialize DFLL to enable target device
clock in open loop mode or closed loop mode.

Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/clk/tegra/clk-dfll.c | 78 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/clk/tegra/clk-dfll.h |  2 ++
 2 files changed, 80 insertions(+)

diff --git a/drivers/clk/tegra/clk-dfll.c b/drivers/clk/tegra/clk-dfll.c
index f8688c2ddf1a..a1f37cf99b00 100644
--- a/drivers/clk/tegra/clk-dfll.c
+++ b/drivers/clk/tegra/clk-dfll.c
@@ -277,6 +277,7 @@ struct tegra_dfll {
 	unsigned long			dvco_rate_min;
 
 	enum dfll_ctrl_mode		mode;
+	enum dfll_ctrl_mode		resume_mode;
 	enum dfll_tune_range		tune_range;
 	struct dentry			*debugfs_dir;
 	struct clk_hw			dfll_clk_hw;
@@ -1864,6 +1865,83 @@ static int dfll_fetch_common_params(struct tegra_dfll *td)
 }
 
 /*
+ * tegra_dfll_suspend
+ * @pdev: DFLL instance
+ *
+ * dfll controls clock/voltage to other devices, including CPU. Therefore,
+ * dfll driver pm suspend callback does not stop cl-dvfs operations.
+ */
+void tegra_dfll_suspend(struct platform_device *pdev)
+{
+	struct tegra_dfll *td = dev_get_drvdata(&pdev->dev);
+
+	if (!td)
+		return;
+
+	if (td->mode <= DFLL_DISABLED)
+		return;
+
+	td->resume_mode = td->mode;
+	switch (td->mode) {
+	case DFLL_CLOSED_LOOP:
+		dfll_set_mode(td, DFLL_CLOSED_LOOP);
+		dfll_set_frequency_request(td, &td->last_req);
+
+		dfll_unlock(td);
+		break;
+	default:
+		break;
+	}
+}
+
+/**
+ * tegra_dfll_resume - reprogram the DFLL after context-loss
+ * @pdev: DFLL instance
+ *
+ * Re-initialize and enable target device clock in open loop mode. Called
+ * directly from SoC clock resume syscore operation. Closed loop will be
+ * re-entered in platform syscore ops as well after CPU clock source is
+ * switched to DFLL in open loop.
+ */
+void tegra_dfll_resume(struct platform_device *pdev, bool on_dfll)
+{
+	struct tegra_dfll *td = dev_get_drvdata(&pdev->dev);
+
+	if (!td)
+		return;
+
+	if (on_dfll) {
+		if (td->resume_mode == DFLL_CLOSED_LOOP)
+			dfll_lock(td);
+		td->resume_mode = DFLL_DISABLED;
+		return;
+	}
+
+	reset_control_deassert(td->dvco_rst);
+
+	pm_runtime_get(td->dev);
+
+	/* Re-init DFLL */
+	dfll_init_out_if(td);
+	dfll_set_default_params(td);
+	dfll_set_open_loop_config(td);
+
+	pm_runtime_put(td->dev);
+
+	/* Restore last request and mode up to open loop */
+	switch (td->resume_mode) {
+	case DFLL_CLOSED_LOOP:
+	case DFLL_OPEN_LOOP:
+		dfll_set_mode(td, DFLL_OPEN_LOOP);
+		if (td->pmu_if == TEGRA_DFLL_PMU_I2C)
+			dfll_i2c_set_output_enabled(td, false);
+		break;
+	default:
+		break;
+	}
+}
+
+/*
  * API exported to per-SoC platform drivers
  */
 
diff --git a/drivers/clk/tegra/clk-dfll.h b/drivers/clk/tegra/clk-dfll.h
index 1b14ebe7268b..c21fc2061a20 100644
--- a/drivers/clk/tegra/clk-dfll.h
+++ b/drivers/clk/tegra/clk-dfll.h
@@ -40,6 +40,8 @@ struct tegra_dfll_soc_data {
 int tegra_dfll_register(struct platform_device *pdev,
 			struct tegra_dfll_soc_data *soc);
 struct tegra_dfll_soc_data *tegra_dfll_unregister(struct platform_device *pdev);
+void tegra_dfll_suspend(struct platform_device *pdev);
+void tegra_dfll_resume(struct platform_device *pdev, bool on_dfll);
 int tegra_dfll_runtime_suspend(struct device *dev);
 int tegra_dfll_runtime_resume(struct device *dev);
 
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 09/18] clk: tegra: Add save and restore context support for peripheral clocks
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (7 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 08/18] clk: tegra: Add suspend resume support for DFLL Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-29 13:17   ` Dmitry Osipenko
  2019-06-28  2:12 ` [PATCH V5 10/18] clk: tegra210: Use fence_udelay during PLLU init Sowjanya Komatineni
                   ` (8 subsequent siblings)
  17 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch implements save and restore context for peripheral fixed
clock ops, peripheral gate clock ops, sdmmc mux clock ops, and
peripheral clock ops.

During system suspend, core power goes off and looses the settings
of the Tegra CAR controller registers.

So during suspend entry clock and reset state of peripherals is saved
and on resume they are restored to have clocks back to same rate and
state as before suspend.

Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/clk/tegra/clk-periph-fixed.c | 31 ++++++++++++++++++++++++++
 drivers/clk/tegra/clk-periph-gate.c  | 34 ++++++++++++++++++++++++++++
 drivers/clk/tegra/clk-periph.c       | 43 ++++++++++++++++++++++++++++++++++++
 drivers/clk/tegra/clk-sdmmc-mux.c    | 30 +++++++++++++++++++++++++
 drivers/clk/tegra/clk.h              |  8 +++++++
 5 files changed, 146 insertions(+)

diff --git a/drivers/clk/tegra/clk-periph-fixed.c b/drivers/clk/tegra/clk-periph-fixed.c
index c088e7a280df..981f68b0a937 100644
--- a/drivers/clk/tegra/clk-periph-fixed.c
+++ b/drivers/clk/tegra/clk-periph-fixed.c
@@ -60,11 +60,42 @@ tegra_clk_periph_fixed_recalc_rate(struct clk_hw *hw,
 	return (unsigned long)rate;
 }
 
+static int tegra_clk_periph_fixed_save_context(struct clk_hw *hw)
+{
+	struct tegra_clk_periph_fixed *fixed = to_tegra_clk_periph_fixed(hw);
+	u32 mask = 1 << (fixed->num % 32);
+
+	fixed->enb_ctx = readl(fixed->base + fixed->regs->enb_reg) & mask;
+	fixed->rst_ctx = readl(fixed->base + fixed->regs->rst_reg) & mask;
+
+	return 0;
+}
+
+static void tegra_clk_periph_fixed_restore_context(struct clk_hw *hw)
+{
+	struct tegra_clk_periph_fixed *fixed = to_tegra_clk_periph_fixed(hw);
+	u32 mask = 1 << (fixed->num % 32);
+
+	if (fixed->enb_ctx)
+		tegra_clk_periph_fixed_enable(hw);
+	else
+		tegra_clk_periph_fixed_disable(hw);
+
+	udelay(2);
+
+	if (!fixed->rst_ctx) {
+		udelay(5); /* reset propogation delay */
+		writel(mask, fixed->base + fixed->regs->rst_reg);
+	}
+}
+
 static const struct clk_ops tegra_clk_periph_fixed_ops = {
 	.is_enabled = tegra_clk_periph_fixed_is_enabled,
 	.enable = tegra_clk_periph_fixed_enable,
 	.disable = tegra_clk_periph_fixed_disable,
 	.recalc_rate = tegra_clk_periph_fixed_recalc_rate,
+	.save_context = tegra_clk_periph_fixed_save_context,
+	.restore_context = tegra_clk_periph_fixed_restore_context,
 };
 
 struct clk *tegra_clk_register_periph_fixed(const char *name,
diff --git a/drivers/clk/tegra/clk-periph-gate.c b/drivers/clk/tegra/clk-periph-gate.c
index 4b31beefc9fc..6ba5b08e0787 100644
--- a/drivers/clk/tegra/clk-periph-gate.c
+++ b/drivers/clk/tegra/clk-periph-gate.c
@@ -25,6 +25,8 @@ static DEFINE_SPINLOCK(periph_ref_lock);
 
 #define read_rst(gate) \
 	readl_relaxed(gate->clk_base + (gate->regs->rst_reg))
+#define write_rst_set(val, gate) \
+	writel_relaxed(val, gate->clk_base + (gate->regs->rst_set_reg))
 #define write_rst_clr(val, gate) \
 	writel_relaxed(val, gate->clk_base + (gate->regs->rst_clr_reg))
 
@@ -110,10 +112,42 @@ static void clk_periph_disable(struct clk_hw *hw)
 	spin_unlock_irqrestore(&periph_ref_lock, flags);
 }
 
+static int clk_periph_gate_save_context(struct clk_hw *hw)
+{
+	struct tegra_clk_periph_gate *gate = to_clk_periph_gate(hw);
+
+	gate->clk_state_ctx = read_enb(gate) & periph_clk_to_bit(gate);
+	gate->rst_state_ctx = read_rst(gate) & periph_clk_to_bit(gate);
+
+	return 0;
+}
+
+static void clk_periph_gate_restore_context(struct clk_hw *hw)
+{
+	struct tegra_clk_periph_gate *gate = to_clk_periph_gate(hw);
+
+	if (gate->clk_state_ctx)
+		write_enb_set(periph_clk_to_bit(gate), gate);
+	else
+		write_enb_clr(periph_clk_to_bit(gate), gate);
+
+	udelay(5);
+
+	if (!(gate->flags & TEGRA_PERIPH_NO_RESET) &&
+	    !(gate->flags & TEGRA_PERIPH_MANUAL_RESET)) {
+		if (gate->rst_state_ctx)
+			write_rst_set(periph_clk_to_bit(gate), gate);
+		else
+			write_rst_clr(periph_clk_to_bit(gate), gate);
+	}
+}
+
 const struct clk_ops tegra_clk_periph_gate_ops = {
 	.is_enabled = clk_periph_is_enabled,
 	.enable = clk_periph_enable,
 	.disable = clk_periph_disable,
+	.save_context = clk_periph_gate_save_context,
+	.restore_context = clk_periph_gate_restore_context,
 };
 
 struct clk *tegra_clk_register_periph_gate(const char *name,
diff --git a/drivers/clk/tegra/clk-periph.c b/drivers/clk/tegra/clk-periph.c
index 58437da25156..d07882656e66 100644
--- a/drivers/clk/tegra/clk-periph.c
+++ b/drivers/clk/tegra/clk-periph.c
@@ -5,6 +5,7 @@
 
 #include <linux/clk-provider.h>
 #include <linux/export.h>
+#include <linux/io.h>
 #include <linux/slab.h>
 #include <linux/err.h>
 
@@ -99,6 +100,42 @@ static void clk_periph_disable(struct clk_hw *hw)
 	gate_ops->disable(gate_hw);
 }
 
+static int clk_periph_save_context(struct clk_hw *hw)
+{
+	struct tegra_clk_periph *periph = to_clk_periph(hw);
+	const struct clk_ops *gate_ops = periph->gate_ops;
+	struct clk_hw *gate_hw = &periph->gate.hw;
+	struct clk_hw *parent = clk_hw_get_parent(hw);
+	unsigned long parent_rate = clk_hw_get_rate(parent);
+
+	if (!(periph->gate.flags & TEGRA_PERIPH_NO_GATE))
+		gate_ops->save_context(gate_hw);
+
+	if (!(periph->gate.flags & TEGRA_PERIPH_NO_DIV))
+		periph->rate_ctx = clk_periph_recalc_rate(hw, parent_rate);
+
+	periph->parent_ctx = clk_periph_get_parent(hw);
+
+	return 0;
+}
+
+static void clk_periph_restore_context(struct clk_hw *hw)
+{
+	struct tegra_clk_periph *periph = to_clk_periph(hw);
+	const struct clk_ops *gate_ops = periph->gate_ops;
+	struct clk_hw *gate_hw = &periph->gate.hw;
+	struct clk_hw *parent = clk_hw_get_parent(hw);
+	unsigned long parent_rate = clk_hw_get_rate(parent);
+
+	if (!(periph->gate.flags & TEGRA_PERIPH_NO_GATE))
+		gate_ops->restore_context(gate_hw);
+
+	clk_periph_set_parent(hw, periph->parent_ctx);
+
+	if (!(periph->gate.flags & TEGRA_PERIPH_NO_DIV))
+		clk_periph_set_rate(hw, periph->rate_ctx, parent_rate);
+}
+
 const struct clk_ops tegra_clk_periph_ops = {
 	.get_parent = clk_periph_get_parent,
 	.set_parent = clk_periph_set_parent,
@@ -108,6 +145,8 @@ const struct clk_ops tegra_clk_periph_ops = {
 	.is_enabled = clk_periph_is_enabled,
 	.enable = clk_periph_enable,
 	.disable = clk_periph_disable,
+	.save_context = clk_periph_save_context,
+	.restore_context = clk_periph_restore_context,
 };
 
 static const struct clk_ops tegra_clk_periph_nodiv_ops = {
@@ -116,6 +155,8 @@ static const struct clk_ops tegra_clk_periph_nodiv_ops = {
 	.is_enabled = clk_periph_is_enabled,
 	.enable = clk_periph_enable,
 	.disable = clk_periph_disable,
+	.save_context = clk_periph_save_context,
+	.restore_context = clk_periph_restore_context,
 };
 
 static const struct clk_ops tegra_clk_periph_no_gate_ops = {
@@ -124,6 +165,8 @@ static const struct clk_ops tegra_clk_periph_no_gate_ops = {
 	.recalc_rate = clk_periph_recalc_rate,
 	.round_rate = clk_periph_round_rate,
 	.set_rate = clk_periph_set_rate,
+	.save_context = clk_periph_save_context,
+	.restore_context = clk_periph_restore_context,
 };
 
 static struct clk *_tegra_clk_register_periph(const char *name,
diff --git a/drivers/clk/tegra/clk-sdmmc-mux.c b/drivers/clk/tegra/clk-sdmmc-mux.c
index a5cd3e31dbae..fffe08e02c10 100644
--- a/drivers/clk/tegra/clk-sdmmc-mux.c
+++ b/drivers/clk/tegra/clk-sdmmc-mux.c
@@ -194,6 +194,34 @@ static void clk_sdmmc_mux_disable(struct clk_hw *hw)
 	gate_ops->disable(gate_hw);
 }
 
+static int clk_sdmmc_mux_save_context(struct clk_hw *hw)
+{
+	struct tegra_sdmmc_mux *sdmmc_mux = to_clk_sdmmc_mux(hw);
+	const struct clk_ops *gate_ops = sdmmc_mux->gate_ops;
+	struct clk_hw *gate_hw = &sdmmc_mux->gate.hw;
+	struct clk_hw *parent = clk_hw_get_parent(hw);
+	unsigned long parent_rate = clk_hw_get_rate(parent);
+
+	sdmmc_mux->rate_ctx = clk_sdmmc_mux_recalc_rate(hw, parent_rate);
+	sdmmc_mux->parent_ctx = clk_sdmmc_mux_get_parent(hw);
+	gate_ops->save_context(gate_hw);
+
+	return 0;
+}
+
+static void clk_sdmmc_mux_restore_context(struct clk_hw *hw)
+{
+	struct tegra_sdmmc_mux *sdmmc_mux = to_clk_sdmmc_mux(hw);
+	const struct clk_ops *gate_ops = sdmmc_mux->gate_ops;
+	struct clk_hw *gate_hw = &sdmmc_mux->gate.hw;
+	struct clk_hw *parent = clk_hw_get_parent(hw);
+	unsigned long parent_rate = clk_hw_get_rate(parent);
+
+	clk_sdmmc_mux_set_parent(hw, sdmmc_mux->parent_ctx);
+	clk_sdmmc_mux_set_rate(hw, sdmmc_mux->rate_ctx, parent_rate);
+	gate_ops->restore_context(gate_hw);
+}
+
 static const struct clk_ops tegra_clk_sdmmc_mux_ops = {
 	.get_parent = clk_sdmmc_mux_get_parent,
 	.set_parent = clk_sdmmc_mux_set_parent,
@@ -203,6 +231,8 @@ static const struct clk_ops tegra_clk_sdmmc_mux_ops = {
 	.is_enabled = clk_sdmmc_mux_is_enabled,
 	.enable = clk_sdmmc_mux_enable,
 	.disable = clk_sdmmc_mux_disable,
+	.save_context = clk_sdmmc_mux_save_context,
+	.restore_context = clk_sdmmc_mux_restore_context,
 };
 
 struct clk *tegra_clk_register_sdmmc_mux_div(const char *name,
diff --git a/drivers/clk/tegra/clk.h b/drivers/clk/tegra/clk.h
index a687ed6127b6..13e16359ebbe 100644
--- a/drivers/clk/tegra/clk.h
+++ b/drivers/clk/tegra/clk.h
@@ -533,6 +533,8 @@ struct tegra_clk_periph_gate {
 	int			clk_num;
 	int			*enable_refcnt;
 	const struct tegra_clk_periph_regs *regs;
+	bool			clk_state_ctx;
+	bool			rst_state_ctx;
 };
 
 #define to_clk_periph_gate(_hw)					\
@@ -559,6 +561,8 @@ struct tegra_clk_periph_fixed {
 	unsigned int mul;
 	unsigned int div;
 	unsigned int num;
+	bool enb_ctx;
+	bool rst_ctx;
 };
 
 struct clk *tegra_clk_register_periph_fixed(const char *name,
@@ -591,6 +595,8 @@ struct tegra_clk_periph {
 	const struct clk_ops	*mux_ops;
 	const struct clk_ops	*div_ops;
 	const struct clk_ops	*gate_ops;
+	unsigned long		rate_ctx;
+	u8			parent_ctx;
 };
 
 #define to_clk_periph(_hw) container_of(_hw, struct tegra_clk_periph, hw)
@@ -742,6 +748,8 @@ struct tegra_sdmmc_mux {
 	const struct clk_ops	*gate_ops;
 	struct tegra_clk_periph_gate	gate;
 	u8			div_flags;
+	unsigned long		rate_ctx;
+	u8			parent_ctx;
 };
 
 #define to_clk_sdmmc_mux(_hw) container_of(_hw, struct tegra_sdmmc_mux, hw)
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 10/18] clk: tegra210: Use fence_udelay during PLLU init
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (8 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 09/18] clk: tegra: Add save and restore context support for peripheral clocks Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks Sowjanya Komatineni
                   ` (7 subsequent siblings)
  17 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch uses fence_udelay rather than udelay during PLLU
initialization to ensure writes to clock registers happens before
waiting for specified delay.

Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/clk/tegra/clk-tegra210.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/clk/tegra/clk-tegra210.c b/drivers/clk/tegra/clk-tegra210.c
index 7f90442462af..1c08c53482a5 100644
--- a/drivers/clk/tegra/clk-tegra210.c
+++ b/drivers/clk/tegra/clk-tegra210.c
@@ -2836,7 +2836,7 @@ static int tegra210_enable_pllu(void)
 	reg = readl_relaxed(clk_base + pllu.params->ext_misc_reg[0]);
 	reg &= ~BIT(pllu.params->iddq_bit_idx);
 	writel_relaxed(reg, clk_base + pllu.params->ext_misc_reg[0]);
-	udelay(5);
+	fence_udelay(5, clk_base);
 
 	reg = readl_relaxed(clk_base + PLLU_BASE);
 	reg &= ~GENMASK(20, 0);
@@ -2844,7 +2844,7 @@ static int tegra210_enable_pllu(void)
 	reg |= fentry->n << 8;
 	reg |= fentry->p << 16;
 	writel(reg, clk_base + PLLU_BASE);
-	udelay(1);
+	fence_udelay(1, clk_base);
 	reg |= PLL_ENABLE;
 	writel(reg, clk_base + PLLU_BASE);
 
@@ -2890,12 +2890,12 @@ static int tegra210_init_pllu(void)
 		reg = readl_relaxed(clk_base + XUSB_PLL_CFG0);
 		reg &= ~XUSB_PLL_CFG0_PLLU_LOCK_DLY_MASK;
 		writel_relaxed(reg, clk_base + XUSB_PLL_CFG0);
-		udelay(1);
+		fence_udelay(1, clk_base);
 
 		reg = readl_relaxed(clk_base + PLLU_HW_PWRDN_CFG0);
 		reg |= PLLU_HW_PWRDN_CFG0_SEQ_ENABLE;
 		writel_relaxed(reg, clk_base + PLLU_HW_PWRDN_CFG0);
-		udelay(1);
+		fence_udelay(1, clk_base);
 
 		reg = readl_relaxed(clk_base + PLLU_BASE);
 		reg &= ~PLLU_BASE_CLKENABLE_USB;
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (9 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 10/18] clk: tegra210: Use fence_udelay during PLLU init Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-29 13:14   ` Dmitry Osipenko
                     ` (2 more replies)
  2019-06-28  2:12 ` [PATCH V5 12/18] soc/tegra: pmc: Allow support for more tegra wake Sowjanya Komatineni
                   ` (6 subsequent siblings)
  17 siblings, 3 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch adds system suspend and resume support for Tegra210
clocks.

All the CAR controller settings are lost on suspend when core power
goes off.

This patch has implementation for saving and restoring all the PLLs
and clocks context during system suspend and resume to have the
clocks back to same state for normal operation.

Acked-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/clk/tegra/clk-tegra210.c | 115 ++++++++++++++++++++++++++++++++++++++-
 drivers/clk/tegra/clk.c          |  14 +++++
 drivers/clk/tegra/clk.h          |   1 +
 3 files changed, 127 insertions(+), 3 deletions(-)

diff --git a/drivers/clk/tegra/clk-tegra210.c b/drivers/clk/tegra/clk-tegra210.c
index 1c08c53482a5..1b839544e086 100644
--- a/drivers/clk/tegra/clk-tegra210.c
+++ b/drivers/clk/tegra/clk-tegra210.c
@@ -9,10 +9,12 @@
 #include <linux/clkdev.h>
 #include <linux/of.h>
 #include <linux/of_address.h>
+#include <linux/of_platform.h>
 #include <linux/delay.h>
 #include <linux/export.h>
 #include <linux/mutex.h>
 #include <linux/clk/tegra.h>
+#include <linux/syscore_ops.h>
 #include <dt-bindings/clock/tegra210-car.h>
 #include <dt-bindings/reset/tegra210-car.h>
 #include <linux/iopoll.h>
@@ -20,6 +22,7 @@
 #include <soc/tegra/pmc.h>
 
 #include "clk.h"
+#include "clk-dfll.h"
 #include "clk-id.h"
 
 /*
@@ -225,6 +228,7 @@
 
 #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
 #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
+#define CPU_SOFTRST_CTRL 0x380
 
 #define LVL2_CLK_GATE_OVRA 0xf8
 #define LVL2_CLK_GATE_OVRC 0x3a0
@@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
 	struct tegra_clk_pll_freq_table *fentry;
 	struct tegra_clk_pll pllu;
 	u32 reg;
+	int ret;
 
 	for (fentry = pll_u_freq_table; fentry->input_rate; fentry++) {
 		if (fentry->input_rate == pll_ref_freq)
@@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
 	fence_udelay(1, clk_base);
 	reg |= PLL_ENABLE;
 	writel(reg, clk_base + PLLU_BASE);
+	fence_udelay(1, clk_base);
 
-	readl_relaxed_poll_timeout_atomic(clk_base + PLLU_BASE, reg,
-					  reg & PLL_BASE_LOCK, 2, 1000);
-	if (!(reg & PLL_BASE_LOCK)) {
+	ret = tegra210_wait_for_mask(&pllu, PLLU_BASE, PLL_BASE_LOCK);
+	if (ret) {
 		pr_err("Timed out waiting for PLL_U to lock\n");
 		return -ETIMEDOUT;
 	}
@@ -3283,6 +3288,103 @@ static void tegra210_disable_cpu_clock(u32 cpu)
 }
 
 #ifdef CONFIG_PM_SLEEP
+static u32 cpu_softrst_ctx[3];
+static struct platform_device *dfll_pdev;
+#define car_readl(_base, _off) readl_relaxed(clk_base + (_base) + ((_off) * 4))
+#define car_writel(_val, _base, _off) \
+		writel_relaxed(_val, clk_base + (_base) + ((_off) * 4))
+
+static int tegra210_clk_suspend(void)
+{
+	unsigned int i;
+	struct device_node *node;
+
+	tegra_cclkg_burst_policy_save_context();
+
+	if (!dfll_pdev) {
+		node = of_find_compatible_node(NULL, NULL,
+					       "nvidia,tegra210-dfll");
+		if (node)
+			dfll_pdev = of_find_device_by_node(node);
+
+		of_node_put(node);
+		if (!dfll_pdev)
+			pr_err("dfll node not found. no suspend for dfll\n");
+	}
+
+	if (dfll_pdev)
+		tegra_dfll_suspend(dfll_pdev);
+
+	/* Enable PLLP_OUT_CPU after dfll suspend */
+	tegra_clk_set_pllp_out_cpu(true);
+
+	tegra_sclk_cclklp_burst_policy_save_context();
+
+	clk_save_context();
+
+	for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
+		cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
+
+	return 0;
+}
+
+static void tegra210_clk_resume(void)
+{
+	unsigned int i;
+	struct clk_hw *parent;
+	struct clk *clk;
+
+	/*
+	 * clk_restore_context restores clocks as per the clock tree.
+	 *
+	 * dfllCPU_out is first in the clock tree to get restored and it
+	 * involves programming DFLL controller along with restoring CPUG
+	 * clock burst policy.
+	 *
+	 * DFLL programming needs dfll_ref and dfll_soc peripheral clocks
+	 * to be restores which are part ofthe peripheral clocks.
+	 * So, peripheral clocks restore should happen prior to dfll clock
+	 * restore.
+	 */
+
+	tegra_clk_osc_resume(clk_base);
+	for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
+		car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
+
+	/* restore all plls and peripheral clocks */
+	tegra210_init_pllu();
+	clk_restore_context();
+
+	fence_udelay(5, clk_base);
+
+	/* resume SCLK and CPULP clocks */
+	tegra_sclk_cpulp_burst_policy_restore_context();
+
+	/*
+	 * restore CPUG clocks:
+	 * - enable DFLL in open loop mode
+	 * - switch CPUG to DFLL clock source
+	 * - close DFLL loop
+	 * - sync PLLX state
+	 */
+	if (dfll_pdev)
+		tegra_dfll_resume(dfll_pdev, false);
+
+	tegra_cclkg_burst_policy_restore_context();
+	fence_udelay(2, clk_base);
+
+	if (dfll_pdev)
+		tegra_dfll_resume(dfll_pdev, true);
+
+	parent = clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
+	clk = clks[TEGRA210_CLK_PLL_X];
+	if (parent != __clk_get_hw(clk))
+		tegra_clk_sync_state_pll(__clk_get_hw(clk));
+
+	/* Disable PLL_OUT_CPU after DFLL resume */
+	tegra_clk_set_pllp_out_cpu(false);
+}
+
 static void tegra210_cpu_clock_suspend(void)
 {
 	/* switch coresite to clk_m, save off original source */
@@ -3298,6 +3400,11 @@ static void tegra210_cpu_clock_resume(void)
 }
 #endif
 
+static struct syscore_ops tegra_clk_syscore_ops = {
+	.suspend = tegra210_clk_suspend,
+	.resume = tegra210_clk_resume,
+};
+
 static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
 	.wait_for_reset	= tegra210_wait_cpu_in_reset,
 	.disable_clock	= tegra210_disable_cpu_clock,
@@ -3583,5 +3690,7 @@ static void __init tegra210_clock_init(struct device_node *np)
 	tegra210_mbist_clk_init();
 
 	tegra_cpu_car_ops = &tegra210_cpu_car_ops;
+
+	register_syscore_ops(&tegra_clk_syscore_ops);
 }
 CLK_OF_DECLARE(tegra210, "nvidia,tegra210-car", tegra210_clock_init);
diff --git a/drivers/clk/tegra/clk.c b/drivers/clk/tegra/clk.c
index 9e863362d2bf..96cc9937ea37 100644
--- a/drivers/clk/tegra/clk.c
+++ b/drivers/clk/tegra/clk.c
@@ -23,6 +23,7 @@
 #define CLK_OUT_ENB_W			0x364
 #define CLK_OUT_ENB_X			0x280
 #define CLK_OUT_ENB_Y			0x298
+#define CLK_ENB_PLLP_OUT_CPU		BIT(31)
 #define CLK_OUT_ENB_SET_L		0x320
 #define CLK_OUT_ENB_CLR_L		0x324
 #define CLK_OUT_ENB_SET_H		0x328
@@ -205,6 +206,19 @@ const struct tegra_clk_periph_regs *get_reg_bank(int clkid)
 	}
 }
 
+void tegra_clk_set_pllp_out_cpu(bool enable)
+{
+	u32 val;
+
+	val = readl_relaxed(clk_base + CLK_OUT_ENB_Y);
+	if (enable)
+		val |= CLK_ENB_PLLP_OUT_CPU;
+	else
+		val &= ~CLK_ENB_PLLP_OUT_CPU;
+
+	writel_relaxed(val, clk_base + CLK_OUT_ENB_Y);
+}
+
 void tegra_cclkg_burst_policy_save_context(void)
 {
 	unsigned int i;
diff --git a/drivers/clk/tegra/clk.h b/drivers/clk/tegra/clk.h
index 13e16359ebbe..a0fbebc1d913 100644
--- a/drivers/clk/tegra/clk.h
+++ b/drivers/clk/tegra/clk.h
@@ -872,6 +872,7 @@ void tegra_cclkg_burst_policy_restore_context(void);
 void tegra_sclk_cclklp_burst_policy_save_context(void);
 void tegra_sclk_cpulp_burst_policy_restore_context(void);
 void tegra_clk_osc_resume(void __iomem *clk_base);
+void tegra_clk_set_pllp_out_cpu(bool enable);
 
 /* Combined read fence with delay */
 #define fence_udelay(delay, reg)	\
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 12/18] soc/tegra: pmc: Allow support for more tegra wake
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (10 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 13/18] soc/tegra: pmc: Add pmc wake support for tegra210 Sowjanya Komatineni
                   ` (5 subsequent siblings)
  17 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch allows to create separate irq_set_wake and irq_set_type
implementations for different tegra designs PMC that has different
wake models which require difference wake registers and different
programming sequence.

AOWAKE model support is available for Tegra186 and Tegra194 only
and it resides within PMC and supports tiered wake architecture.

Tegra210 and prior tegra designs uses PMC directly to receive wake
events and coordinate the wake sequence.

Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/soc/tegra/pmc.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/soc/tegra/pmc.c b/drivers/soc/tegra/pmc.c
index 9f9c1c677cf4..91c84d0e66ae 100644
--- a/drivers/soc/tegra/pmc.c
+++ b/drivers/soc/tegra/pmc.c
@@ -226,6 +226,8 @@ struct tegra_pmc_soc {
 	void (*setup_irq_polarity)(struct tegra_pmc *pmc,
 				   struct device_node *np,
 				   bool invert);
+	int (*irq_set_wake)(struct irq_data *data, unsigned int on);
+	int (*irq_set_type)(struct irq_data *data, unsigned int type);
 
 	const char * const *reset_sources;
 	unsigned int num_reset_sources;
@@ -1920,7 +1922,7 @@ static const struct irq_domain_ops tegra_pmc_irq_domain_ops = {
 	.alloc = tegra_pmc_irq_alloc,
 };
 
-static int tegra_pmc_irq_set_wake(struct irq_data *data, unsigned int on)
+static int tegra186_pmc_irq_set_wake(struct irq_data *data, unsigned int on)
 {
 	struct tegra_pmc *pmc = irq_data_get_irq_chip_data(data);
 	unsigned int offset, bit;
@@ -1952,7 +1954,7 @@ static int tegra_pmc_irq_set_wake(struct irq_data *data, unsigned int on)
 	return 0;
 }
 
-static int tegra_pmc_irq_set_type(struct irq_data *data, unsigned int type)
+static int tegra186_pmc_irq_set_type(struct irq_data *data, unsigned int type)
 {
 	struct tegra_pmc *pmc = irq_data_get_irq_chip_data(data);
 	u32 value;
@@ -2006,8 +2008,8 @@ static int tegra_pmc_irq_init(struct tegra_pmc *pmc)
 	pmc->irq.irq_unmask = irq_chip_unmask_parent;
 	pmc->irq.irq_eoi = irq_chip_eoi_parent;
 	pmc->irq.irq_set_affinity = irq_chip_set_affinity_parent;
-	pmc->irq.irq_set_type = tegra_pmc_irq_set_type;
-	pmc->irq.irq_set_wake = tegra_pmc_irq_set_wake;
+	pmc->irq.irq_set_type = pmc->soc->irq_set_type;
+	pmc->irq.irq_set_wake = pmc->soc->irq_set_wake;
 
 	pmc->domain = irq_domain_add_hierarchy(parent, 0, 96, pmc->dev->of_node,
 					       &tegra_pmc_irq_domain_ops, pmc);
@@ -2680,6 +2682,8 @@ static const struct tegra_pmc_soc tegra186_pmc_soc = {
 	.regs = &tegra186_pmc_regs,
 	.init = NULL,
 	.setup_irq_polarity = tegra186_pmc_setup_irq_polarity,
+	.irq_set_wake = tegra186_pmc_irq_set_wake,
+	.irq_set_type = tegra186_pmc_irq_set_type,
 	.reset_sources = tegra186_reset_sources,
 	.num_reset_sources = ARRAY_SIZE(tegra186_reset_sources),
 	.reset_levels = tegra186_reset_levels,
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 13/18] soc/tegra: pmc: Add pmc wake support for tegra210
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (11 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 12/18] soc/tegra: pmc: Allow support for more tegra wake Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-29 13:11   ` Dmitry Osipenko
  2019-06-28  2:12 ` [PATCH V5 14/18] arm64: tegra: Enable wake from deep sleep on RTC alarm Sowjanya Komatineni
                   ` (4 subsequent siblings)
  17 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch implements PMC wakeup sequence for Tegra210 and defines
common used RTC alarm wake event.

Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/soc/tegra/pmc.c | 111 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 111 insertions(+)

diff --git a/drivers/soc/tegra/pmc.c b/drivers/soc/tegra/pmc.c
index 91c84d0e66ae..194100d77243 100644
--- a/drivers/soc/tegra/pmc.c
+++ b/drivers/soc/tegra/pmc.c
@@ -57,6 +57,12 @@
 #define  PMC_CNTRL_SYSCLK_OE		BIT(11) /* system clock enable */
 #define  PMC_CNTRL_SYSCLK_POLARITY	BIT(10) /* sys clk polarity */
 #define  PMC_CNTRL_MAIN_RST		BIT(4)
+#define  PMC_CNTRL_LATCH_WAKEUPS	BIT(5)
+
+#define PMC_WAKE_MASK			0x0c
+#define PMC_WAKE_LEVEL			0x10
+#define PMC_WAKE_STATUS			0x14
+#define PMC_SW_WAKE_STATUS		0x18
 
 #define DPD_SAMPLE			0x020
 #define  DPD_SAMPLE_ENABLE		BIT(0)
@@ -87,6 +93,11 @@
 
 #define PMC_SCRATCH41			0x140
 
+#define PMC_WAKE2_MASK			0x160
+#define PMC_WAKE2_LEVEL			0x164
+#define PMC_WAKE2_STATUS		0x168
+#define PMC_SW_WAKE2_STATUS		0x16c
+
 #define PMC_SENSOR_CTRL			0x1b0
 #define  PMC_SENSOR_CTRL_SCRATCH_WRITE	BIT(2)
 #define  PMC_SENSOR_CTRL_ENABLE_RST	BIT(1)
@@ -1922,6 +1933,55 @@ static const struct irq_domain_ops tegra_pmc_irq_domain_ops = {
 	.alloc = tegra_pmc_irq_alloc,
 };
 
+static int tegra210_pmc_irq_set_wake(struct irq_data *data, unsigned int on)
+{
+	struct tegra_pmc *pmc = irq_data_get_irq_chip_data(data);
+	unsigned int offset, bit;
+	u32 value;
+
+	if (data->hwirq == ULONG_MAX)
+		return 0;
+
+	offset = data->hwirq / 32;
+	bit = data->hwirq % 32;
+
+	/*
+	 * latch wakeups to SW_WAKE_STATUS register to capture events
+	 * that would not make it into wakeup event register during LP0 exit.
+	 */
+	value = tegra_pmc_readl(pmc, PMC_CNTRL);
+	value |= PMC_CNTRL_LATCH_WAKEUPS;
+	tegra_pmc_writel(pmc, value, PMC_CNTRL);
+	udelay(120);
+
+	value &= ~PMC_CNTRL_LATCH_WAKEUPS;
+	tegra_pmc_writel(pmc, value, PMC_CNTRL);
+	udelay(120);
+
+	tegra_pmc_writel(pmc, 0, PMC_SW_WAKE_STATUS);
+	tegra_pmc_writel(pmc, 0, PMC_SW_WAKE2_STATUS);
+
+	tegra_pmc_writel(pmc, 0, PMC_WAKE_STATUS);
+	tegra_pmc_writel(pmc, 0, PMC_WAKE2_STATUS);
+
+	/* enable PMC wake */
+	if (data->hwirq >= 32)
+		offset = PMC_WAKE2_MASK;
+	else
+		offset = PMC_WAKE_MASK;
+
+	value = tegra_pmc_readl(pmc, offset);
+
+	if (on)
+		value |= 1 << bit;
+	else
+		value &= ~(1 << bit);
+
+	tegra_pmc_writel(pmc, value, offset);
+
+	return 0;
+}
+
 static int tegra186_pmc_irq_set_wake(struct irq_data *data, unsigned int on)
 {
 	struct tegra_pmc *pmc = irq_data_get_irq_chip_data(data);
@@ -1954,6 +2014,49 @@ static int tegra186_pmc_irq_set_wake(struct irq_data *data, unsigned int on)
 	return 0;
 }
 
+static int tegra210_pmc_irq_set_type(struct irq_data *data, unsigned int type)
+{
+	struct tegra_pmc *pmc = irq_data_get_irq_chip_data(data);
+	unsigned int offset, bit;
+	u32 value;
+
+	if (data->hwirq == ULONG_MAX)
+		return 0;
+
+	offset = data->hwirq / 32;
+	bit = data->hwirq % 32;
+
+	if (data->hwirq >= 32)
+		offset = PMC_WAKE2_LEVEL;
+	else
+		offset = PMC_WAKE_LEVEL;
+
+	value = tegra_pmc_readl(pmc, offset);
+
+	switch (type) {
+	case IRQ_TYPE_EDGE_RISING:
+	case IRQ_TYPE_LEVEL_HIGH:
+		value |= 1 << bit;
+		break;
+
+	case IRQ_TYPE_EDGE_FALLING:
+	case IRQ_TYPE_LEVEL_LOW:
+		value &= ~(1 << bit);
+		break;
+
+	case IRQ_TYPE_EDGE_RISING | IRQ_TYPE_EDGE_FALLING:
+		value ^= 1 << bit;
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	tegra_pmc_writel(pmc, value, offset);
+
+	return 0;
+}
+
 static int tegra186_pmc_irq_set_type(struct irq_data *data, unsigned int type)
 {
 	struct tegra_pmc *pmc = irq_data_get_irq_chip_data(data);
@@ -2540,6 +2643,10 @@ static const struct pinctrl_pin_desc tegra210_pin_descs[] = {
 	TEGRA210_IO_PAD_TABLE(TEGRA_IO_PIN_DESC)
 };
 
+static const struct tegra_wake_event tegra210_wake_events[] = {
+	TEGRA_WAKE_IRQ("rtc", 16, 2),
+};
+
 static const struct tegra_pmc_soc tegra210_pmc_soc = {
 	.num_powergates = ARRAY_SIZE(tegra210_powergates),
 	.powergates = tegra210_powergates,
@@ -2557,10 +2664,14 @@ static const struct tegra_pmc_soc tegra210_pmc_soc = {
 	.regs = &tegra20_pmc_regs,
 	.init = tegra20_pmc_init,
 	.setup_irq_polarity = tegra20_pmc_setup_irq_polarity,
+	.irq_set_wake = tegra210_pmc_irq_set_wake,
+	.irq_set_type = tegra210_pmc_irq_set_type,
 	.reset_sources = tegra210_reset_sources,
 	.num_reset_sources = ARRAY_SIZE(tegra210_reset_sources),
 	.reset_levels = NULL,
 	.num_reset_levels = 0,
+	.num_wake_events = ARRAY_SIZE(tegra210_wake_events),
+	.wake_events = tegra210_wake_events,
 };
 
 #define TEGRA186_IO_PAD_TABLE(_pad)					     \
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 14/18] arm64: tegra: Enable wake from deep sleep on RTC alarm.
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (12 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 13/18] soc/tegra: pmc: Add pmc wake support for tegra210 Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 15/18] soc/tegra: pmc: Configure core power request polarity Sowjanya Komatineni
                   ` (3 subsequent siblings)
  17 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch updates device tree for RTC and PMC to allow system wake
from deep sleep on RTC alarm.

Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 arch/arm64/boot/dts/nvidia/tegra210.dtsi | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/boot/dts/nvidia/tegra210.dtsi b/arch/arm64/boot/dts/nvidia/tegra210.dtsi
index 659753118e96..30a7c48385a2 100644
--- a/arch/arm64/boot/dts/nvidia/tegra210.dtsi
+++ b/arch/arm64/boot/dts/nvidia/tegra210.dtsi
@@ -768,7 +768,8 @@
 	rtc@7000e000 {
 		compatible = "nvidia,tegra210-rtc", "nvidia,tegra20-rtc";
 		reg = <0x0 0x7000e000 0x0 0x100>;
-		interrupts = <GIC_SPI 2 IRQ_TYPE_LEVEL_HIGH>;
+		interrupts = <16 IRQ_TYPE_LEVEL_HIGH>;
+		interrupt-parent = <&pmc>;
 		clocks = <&tegra_car TEGRA210_CLK_RTC>;
 		clock-names = "rtc";
 	};
@@ -778,6 +779,8 @@
 		reg = <0x0 0x7000e400 0x0 0x400>;
 		clocks = <&tegra_car TEGRA210_CLK_PCLK>, <&clk32k_in>;
 		clock-names = "pclk", "clk32k_in";
+		#interrupt-cells = <2>;
+		interrupt-controller;
 
 		powergates {
 			pd_audio: aud {
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 15/18] soc/tegra: pmc: Configure core power request polarity
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (13 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 14/18] arm64: tegra: Enable wake from deep sleep on RTC alarm Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 16/18] soc/tegra: pmc: Configure deep sleep control settings Sowjanya Komatineni
                   ` (2 subsequent siblings)
  17 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch configures polarity of the core power request signal
in PMC control register based on the device tree property.

PMC asserts and de-asserts power request signal based on it polarity
when it need to power-up and power-down the core rail during SC7.

Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/soc/tegra/pmc.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/soc/tegra/pmc.c b/drivers/soc/tegra/pmc.c
index 194100d77243..ed83c0cd09a3 100644
--- a/drivers/soc/tegra/pmc.c
+++ b/drivers/soc/tegra/pmc.c
@@ -56,6 +56,7 @@
 #define  PMC_CNTRL_SIDE_EFFECT_LP0	BIT(14) /* LP0 when CPU pwr gated */
 #define  PMC_CNTRL_SYSCLK_OE		BIT(11) /* system clock enable */
 #define  PMC_CNTRL_SYSCLK_POLARITY	BIT(10) /* sys clk polarity */
+#define  PMC_CNTRL_PWRREQ_POLARITY	BIT(8)
 #define  PMC_CNTRL_MAIN_RST		BIT(4)
 #define  PMC_CNTRL_LATCH_WAKEUPS	BIT(5)
 
@@ -2303,6 +2304,11 @@ static void tegra20_pmc_init(struct tegra_pmc *pmc)
 	else
 		value |= PMC_CNTRL_SYSCLK_POLARITY;
 
+	if (pmc->corereq_high)
+		value &= ~PMC_CNTRL_PWRREQ_POLARITY;
+	else
+		value |= PMC_CNTRL_PWRREQ_POLARITY;
+
 	/* configure the output polarity while the request is tristated */
 	tegra_pmc_writel(pmc, value, PMC_CNTRL);
 
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 16/18] soc/tegra: pmc: Configure deep sleep control settings
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (14 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 15/18] soc/tegra: pmc: Configure core power request polarity Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-29 13:00   ` Dmitry Osipenko
  2019-06-29 13:02   ` Dmitry Osipenko
  2019-06-28  2:12 ` [PATCH V5 17/18] arm64: dts: tegra210-p2180: Jetson TX1 SC7 timings Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 18/18] arm64: dts: tegra210-p3450: Jetson nano " Sowjanya Komatineni
  17 siblings, 2 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

Tegra210 and prior Tegra chips have deep sleep entry and wakeup related
timings which are platform specific that should be configured before
entering into deep sleep.

Below are the timing specific configurations for deep sleep entry and
wakeup.
- Core rail power-on stabilization timer
- OSC clock stabilization timer after SOC rail power is stabilized.
- Core power off time is the minimum wake delay to keep the system
  in deep sleep state irrespective of any quick wake event.

These values depends on the discharge time of regulators and turn OFF
time of the PMIC to allow the complete system to finish entering into
deep sleep state.

These values vary based on the platform design and are specified
through the device tree.

This patch has implementation to configure these timings which are must
to have for proper deep sleep and wakeup operations.

Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 drivers/soc/tegra/pmc.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/soc/tegra/pmc.c b/drivers/soc/tegra/pmc.c
index ed83c0cd09a3..7e4a8f04f4c4 100644
--- a/drivers/soc/tegra/pmc.c
+++ b/drivers/soc/tegra/pmc.c
@@ -89,6 +89,8 @@
 
 #define PMC_CPUPWRGOOD_TIMER		0xc8
 #define PMC_CPUPWROFF_TIMER		0xcc
+#define PMC_COREPWRGOOD_TIMER		0x3c
+#define PMC_COREPWROFF_TIMER		0xe0
 
 #define PMC_PWR_DET_VALUE		0xe4
 
@@ -2291,6 +2293,7 @@ static const struct tegra_pmc_regs tegra20_pmc_regs = {
 static void tegra20_pmc_init(struct tegra_pmc *pmc)
 {
 	u32 value;
+	unsigned long osc, pmu, off;
 
 	/* Always enable CPU power request */
 	value = tegra_pmc_readl(pmc, PMC_CNTRL);
@@ -2316,6 +2319,15 @@ static void tegra20_pmc_init(struct tegra_pmc *pmc)
 	value = tegra_pmc_readl(pmc, PMC_CNTRL);
 	value |= PMC_CNTRL_SYSCLK_OE;
 	tegra_pmc_writel(pmc, value, PMC_CNTRL);
+
+	osc = DIV_ROUND_UP_ULL(pmc->core_osc_time * 8192, 1000000);
+	pmu = DIV_ROUND_UP_ULL(pmc->core_pmu_time * 32768, 1000000);
+	off = DIV_ROUND_UP_ULL(pmc->core_off_time * 32768, 1000000);
+	if (osc && pmu)
+		tegra_pmc_writel(pmc, ((osc << 8) & 0xff00) | (pmu & 0xff),
+				 PMC_COREPWRGOOD_TIMER);
+	if (off)
+		tegra_pmc_writel(pmc, off, PMC_COREPWROFF_TIMER);
 }
 
 static void tegra20_pmc_setup_irq_polarity(struct tegra_pmc *pmc,
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 17/18] arm64: dts: tegra210-p2180: Jetson TX1 SC7 timings
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (15 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 16/18] soc/tegra: pmc: Configure deep sleep control settings Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  2019-06-28  2:12 ` [PATCH V5 18/18] arm64: dts: tegra210-p3450: Jetson nano " Sowjanya Komatineni
  17 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch has Jetson TX1 platform specific SC7 timing configuration
in device tree.

Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 arch/arm64/boot/dts/nvidia/tegra210-p2180.dtsi | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/boot/dts/nvidia/tegra210-p2180.dtsi b/arch/arm64/boot/dts/nvidia/tegra210-p2180.dtsi
index 27723829d033..cb58f79deb48 100644
--- a/arch/arm64/boot/dts/nvidia/tegra210-p2180.dtsi
+++ b/arch/arm64/boot/dts/nvidia/tegra210-p2180.dtsi
@@ -279,6 +279,13 @@
 
 	pmc@7000e400 {
 		nvidia,invert-interrupt;
+		nvidia,suspend-mode = <0>;
+		nvidia,cpu-pwr-good-time = <0>;
+		nvidia,cpu-pwr-off-time = <0>;
+		nvidia,core-pwr-good-time = <4587 3876>;
+		nvidia,core-pwr-off-time = <39065>;
+		nvidia,core-power-req-active-high;
+		nvidia,sys-clock-req-active-high;
 	};
 
 	/* eMMC */
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* [PATCH V5 18/18] arm64: dts: tegra210-p3450: Jetson nano SC7 timings
  2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
                   ` (16 preceding siblings ...)
  2019-06-28  2:12 ` [PATCH V5 17/18] arm64: dts: tegra210-p2180: Jetson TX1 SC7 timings Sowjanya Komatineni
@ 2019-06-28  2:12 ` Sowjanya Komatineni
  17 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28  2:12 UTC (permalink / raw)
  To: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, skomatineni, linux-tegra, linux-kernel,
	mperttunen, spatra, robh+dt, digetx, devicetree

This patch adds Jetson nano platform specific SC7 timing configuration
in the device tree.

Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
---
 arch/arm64/boot/dts/nvidia/tegra210-p3450-0000.dts | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/boot/dts/nvidia/tegra210-p3450-0000.dts b/arch/arm64/boot/dts/nvidia/tegra210-p3450-0000.dts
index 9d17ec707bce..b81bfdfbc115 100644
--- a/arch/arm64/boot/dts/nvidia/tegra210-p3450-0000.dts
+++ b/arch/arm64/boot/dts/nvidia/tegra210-p3450-0000.dts
@@ -382,6 +382,13 @@
 
 	pmc@7000e400 {
 		nvidia,invert-interrupt;
++		nvidia,suspend-mode = <0>;
+		nvidia,cpu-pwr-good-time = <0>;
+		nvidia,cpu-pwr-off-time = <0>;
+		nvidia,core-pwr-good-time = <4587 3876>;
+		nvidia,core-pwr-off-time = <39065>;
+		nvidia,core-power-req-active-high;
+		nvidia,sys-clock-req-active-high;
 	};
 
 	hda@70030000 {
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-06-28  2:12 ` [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support Sowjanya Komatineni
@ 2019-06-28 11:56   ` Dmitry Osipenko
  2019-06-28 12:05     ` Dmitry Osipenko
  2019-06-29 15:46   ` Dmitry Osipenko
  2019-07-04  7:26   ` Linus Walleij
  2 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-28 11:56 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

28.06.2019 5:12, Sowjanya Komatineni пишет:
> This patch adds support for Tegra pinctrl driver suspend and resume.
> 
> During suspend, context of all pinctrl registers are stored and
> on resume they are all restored to have all the pinmux and pad
> configuration for normal operation.
> 
> Acked-by: Thierry Reding <treding@nvidia.com>
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> ---
>  drivers/pinctrl/tegra/pinctrl-tegra.c    | 52 ++++++++++++++++++++++++++++++++
>  drivers/pinctrl/tegra/pinctrl-tegra.h    |  3 ++
>  drivers/pinctrl/tegra/pinctrl-tegra210.c |  1 +
>  3 files changed, 56 insertions(+)
> 
> diff --git a/drivers/pinctrl/tegra/pinctrl-tegra.c b/drivers/pinctrl/tegra/pinctrl-tegra.c
> index 34596b246578..e7c0a1011cba 100644
> --- a/drivers/pinctrl/tegra/pinctrl-tegra.c
> +++ b/drivers/pinctrl/tegra/pinctrl-tegra.c
> @@ -621,6 +621,43 @@ static void tegra_pinctrl_clear_parked_bits(struct tegra_pmx *pmx)
>  	}
>  }
>  
> +static int tegra_pinctrl_suspend(struct device *dev)
> +{
> +	struct tegra_pmx *pmx = dev_get_drvdata(dev);
> +	u32 *backup_regs = pmx->backup_regs;
> +	u32 *regs;
> +	unsigned int i, j;

In general it's better not to use "j" in conjunction with "i" because they look
similar and I seen quite a lot of bugs caused by unnoticed typos like that. So I'm
suggesting to use "i, k" for clarity.

> +
> +	for (i = 0; i < pmx->nbanks; i++) {
> +		regs = pmx->regs[i];
> +		for (j = 0; j < pmx->reg_bank_size[i] / 4; j++)
> +			*backup_regs++ = readl(regs++);

Please use readl_relaxed(), we don't need memory barriers here.

> +	}
> +
> +	return pinctrl_force_sleep(pmx->pctl);
> +}
> +
> +static int tegra_pinctrl_resume(struct device *dev)
> +{
> +	struct tegra_pmx *pmx = dev_get_drvdata(dev);
> +	u32 *backup_regs = pmx->backup_regs;
> +	u32 *regs;
> +	unsigned int i, j;
> +
> +	for (i = 0; i < pmx->nbanks; i++) {
> +		regs = pmx->regs[i];> +		for (j = 0; j < pmx->reg_bank_size[i] / 4; j++)
> +			writel(*backup_regs++, regs++);

Same for writel_relaxed(), memory barrier is inserted *before* the write to ensure
that all previous memory stores are completed. IOREMAP'ed memory is strongly-ordered,
memory barriers are not needed here.

> +	}
> +
> +	return 0;
> +}
> +
> +const struct dev_pm_ops tegra_pinctrl_pm = {
> +	.suspend = &tegra_pinctrl_suspend,
> +	.resume = &tegra_pinctrl_resume
> +};
> +
>  static bool gpio_node_has_range(const char *compatible)
>  {
>  	struct device_node *np;
> @@ -645,6 +682,7 @@ int tegra_pinctrl_probe(struct platform_device *pdev,
>  	int i;
>  	const char **group_pins;
>  	int fn, gn, gfn;
> +	unsigned long backup_regs_size = 0;
>  
>  	pmx = devm_kzalloc(&pdev->dev, sizeof(*pmx), GFP_KERNEL);
>  	if (!pmx)
> @@ -697,6 +735,7 @@ int tegra_pinctrl_probe(struct platform_device *pdev,
>  		res = platform_get_resource(pdev, IORESOURCE_MEM, i);
>  		if (!res)
>  			break;
> +		backup_regs_size += resource_size(res);
>  	}
>  	pmx->nbanks = i;
>  
> @@ -705,11 +744,24 @@ int tegra_pinctrl_probe(struct platform_device *pdev,
>  	if (!pmx->regs)
>  		return -ENOMEM;
>  
> +	pmx->reg_bank_size = devm_kcalloc(&pdev->dev, pmx->nbanks,
> +					  sizeof(*pmx->reg_bank_size),
> +					  GFP_KERNEL);
> +	if (!pmx->reg_bank_size)
> +		return -ENOMEM;

It looks to me that we don't really need to churn with this allocation because the
bank sizes are already a part of the platform driver's description.

We could add a simple helper function that retrieves the bank sizes, like this:

static unsigned int tegra_pinctrl_bank_size(struct device *dev,
					    unsigned int bank_id)
{
	struct platform_device *pdev;
	struct resource *res;

	pdev = to_platform_device(dev);
	res = platform_get_resource(pdev, IORESOURCE_MEM, bank_id);

	return resource_size(res) / 4;
}

> +	pmx->backup_regs = devm_kzalloc(&pdev->dev, backup_regs_size,
> +					GFP_KERNEL);
> +	if (!pmx->backup_regs)
> +		return -ENOMEM;
> +
>  	for (i = 0; i < pmx->nbanks; i++) {
>  		res = platform_get_resource(pdev, IORESOURCE_MEM, i);
>  		pmx->regs[i] = devm_ioremap_resource(&pdev->dev, res);
>  		if (IS_ERR(pmx->regs[i]))
>  			return PTR_ERR(pmx->regs[i]);
> +
> +		pmx->reg_bank_size[i] = resource_size(res);
>  	}
>  
>  	pmx->pctl = devm_pinctrl_register(&pdev->dev, &tegra_pinctrl_desc, pmx);
> diff --git a/drivers/pinctrl/tegra/pinctrl-tegra.h b/drivers/pinctrl/tegra/pinctrl-tegra.h
> index 287702660783..55456f8d44cf 100644
> --- a/drivers/pinctrl/tegra/pinctrl-tegra.h
> +++ b/drivers/pinctrl/tegra/pinctrl-tegra.h
> @@ -17,6 +17,8 @@ struct tegra_pmx {
>  
>  	int nbanks;
>  	void __iomem **regs;
> +	size_t *reg_bank_size;
> +	u32 *backup_regs;
>  };
>  
>  enum tegra_pinconf_param {
> @@ -193,6 +195,7 @@ struct tegra_pinctrl_soc_data {
>  	bool drvtype_in_mux;
>  };
>  
> +extern const struct dev_pm_ops tegra_pinctrl_pm;

Please add a newline here.

>  int tegra_pinctrl_probe(struct platform_device *pdev,
>  			const struct tegra_pinctrl_soc_data *soc_data);
>  #endif
> diff --git a/drivers/pinctrl/tegra/pinctrl-tegra210.c b/drivers/pinctrl/tegra/pinctrl-tegra210.c
> index 0b56ad5c9c1c..edd3f4606cdb 100644
> --- a/drivers/pinctrl/tegra/pinctrl-tegra210.c
> +++ b/drivers/pinctrl/tegra/pinctrl-tegra210.c
> @@ -1571,6 +1571,7 @@ static struct platform_driver tegra210_pinctrl_driver = {
>  	.driver = {
>  		.name = "tegra210-pinctrl",
>  		.of_match_table = tegra210_pinctrl_of_match,
> +		.pm = &tegra_pinctrl_pm,
>  	},
>  	.probe = tegra210_pinctrl_probe,
>  };
> 

Could you please address my comments in the next revision if there will be one?

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-06-28 11:56   ` Dmitry Osipenko
@ 2019-06-28 12:05     ` Dmitry Osipenko
  2019-06-28 23:00       ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-28 12:05 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

28.06.2019 14:56, Dmitry Osipenko пишет:
> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>> This patch adds support for Tegra pinctrl driver suspend and resume.
>>
>> During suspend, context of all pinctrl registers are stored and
>> on resume they are all restored to have all the pinmux and pad
>> configuration for normal operation.
>>
>> Acked-by: Thierry Reding <treding@nvidia.com>
>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>> ---

>>  int tegra_pinctrl_probe(struct platform_device *pdev,
>>  			const struct tegra_pinctrl_soc_data *soc_data);
>>  #endif
>> diff --git a/drivers/pinctrl/tegra/pinctrl-tegra210.c b/drivers/pinctrl/tegra/pinctrl-tegra210.c
>> index 0b56ad5c9c1c..edd3f4606cdb 100644
>> --- a/drivers/pinctrl/tegra/pinctrl-tegra210.c
>> +++ b/drivers/pinctrl/tegra/pinctrl-tegra210.c
>> @@ -1571,6 +1571,7 @@ static struct platform_driver tegra210_pinctrl_driver = {
>>  	.driver = {
>>  		.name = "tegra210-pinctrl",
>>  		.of_match_table = tegra210_pinctrl_of_match,
>> +		.pm = &tegra_pinctrl_pm,
>>  	},
>>  	.probe = tegra210_pinctrl_probe,
>>  };
>>
> 
> Could you please address my comments in the next revision if there will be one?
> 

Also, what about adding ".pm' for other Tegras? I'm sure Jon could test them for you.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-06-28 12:05     ` Dmitry Osipenko
@ 2019-06-28 23:00       ` Sowjanya Komatineni
  2019-06-29 12:38         ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-06-28 23:00 UTC (permalink / raw)
  To: Dmitry Osipenko, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree


On 6/28/19 5:05 AM, Dmitry Osipenko wrote:
> 28.06.2019 14:56, Dmitry Osipenko пишет:
>> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>>> This patch adds support for Tegra pinctrl driver suspend and resume.
>>>
>>> During suspend, context of all pinctrl registers are stored and
>>> on resume they are all restored to have all the pinmux and pad
>>> configuration for normal operation.
>>>
>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>>> ---
>>>   int tegra_pinctrl_probe(struct platform_device *pdev,
>>>   			const struct tegra_pinctrl_soc_data *soc_data);
>>>   #endif
>>> diff --git a/drivers/pinctrl/tegra/pinctrl-tegra210.c b/drivers/pinctrl/tegra/pinctrl-tegra210.c
>>> index 0b56ad5c9c1c..edd3f4606cdb 100644
>>> --- a/drivers/pinctrl/tegra/pinctrl-tegra210.c
>>> +++ b/drivers/pinctrl/tegra/pinctrl-tegra210.c
>>> @@ -1571,6 +1571,7 @@ static struct platform_driver tegra210_pinctrl_driver = {
>>>   	.driver = {
>>>   		.name = "tegra210-pinctrl",
>>>   		.of_match_table = tegra210_pinctrl_of_match,
>>> +		.pm = &tegra_pinctrl_pm,
>>>   	},
>>>   	.probe = tegra210_pinctrl_probe,
>>>   };
>>>
>> Could you please address my comments in the next revision if there will be one?
>>
> Also, what about adding ".pm' for other Tegras? I'm sure Jon could test them for you.

This series is for Tegra210 SC7 entry/exit along with clocks and pinctrl 
suspend resume needed for Tegra210 basic sc7 entry and exit.

This includes pinctrl, pmc changes, clock-tegra210 driver changes all 
w.r.t Tegra210 platforms specific.

Suspend/resume support for other Tegras will be in separate patch series.


thanks

Sowjanya


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-06-28 23:00       ` Sowjanya Komatineni
@ 2019-06-29 12:38         ` Dmitry Osipenko
  2019-06-29 15:40           ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 12:38 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

29.06.2019 2:00, Sowjanya Komatineni пишет:
> 
> On 6/28/19 5:05 AM, Dmitry Osipenko wrote:
>> 28.06.2019 14:56, Dmitry Osipenko пишет:
>>> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>>>> This patch adds support for Tegra pinctrl driver suspend and resume.
>>>>
>>>> During suspend, context of all pinctrl registers are stored and
>>>> on resume they are all restored to have all the pinmux and pad
>>>> configuration for normal operation.
>>>>
>>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>>>> ---
>>>>   int tegra_pinctrl_probe(struct platform_device *pdev,
>>>>               const struct tegra_pinctrl_soc_data *soc_data);
>>>>   #endif
>>>> diff --git a/drivers/pinctrl/tegra/pinctrl-tegra210.c
>>>> b/drivers/pinctrl/tegra/pinctrl-tegra210.c
>>>> index 0b56ad5c9c1c..edd3f4606cdb 100644
>>>> --- a/drivers/pinctrl/tegra/pinctrl-tegra210.c
>>>> +++ b/drivers/pinctrl/tegra/pinctrl-tegra210.c
>>>> @@ -1571,6 +1571,7 @@ static struct platform_driver tegra210_pinctrl_driver = {
>>>>       .driver = {
>>>>           .name = "tegra210-pinctrl",
>>>>           .of_match_table = tegra210_pinctrl_of_match,
>>>> +        .pm = &tegra_pinctrl_pm,
>>>>       },
>>>>       .probe = tegra210_pinctrl_probe,
>>>>   };
>>>>
>>> Could you please address my comments in the next revision if there will be one?
>>>
>> Also, what about adding ".pm' for other Tegras? I'm sure Jon could test them for you.
> 
> This series is for Tegra210 SC7 entry/exit along with clocks and pinctrl suspend
> resume needed for Tegra210 basic sc7 entry and exit.
> 
> This includes pinctrl, pmc changes, clock-tegra210 driver changes all w.r.t Tegra210
> platforms specific.
> 
> Suspend/resume support for other Tegras will be in separate patch series.

Okay, fair enough.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 16/18] soc/tegra: pmc: Configure deep sleep control settings
  2019-06-28  2:12 ` [PATCH V5 16/18] soc/tegra: pmc: Configure deep sleep control settings Sowjanya Komatineni
@ 2019-06-29 13:00   ` Dmitry Osipenko
  2019-06-29 13:02   ` Dmitry Osipenko
  1 sibling, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 13:00 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

28.06.2019 5:12, Sowjanya Komatineni пишет:
> Tegra210 and prior Tegra chips have deep sleep entry and wakeup related
> timings which are platform specific that should be configured before
> entering into deep sleep.
> 
> Below are the timing specific configurations for deep sleep entry and
> wakeup.
> - Core rail power-on stabilization timer
> - OSC clock stabilization timer after SOC rail power is stabilized.
> - Core power off time is the minimum wake delay to keep the system
>   in deep sleep state irrespective of any quick wake event.
> 
> These values depends on the discharge time of regulators and turn OFF
> time of the PMIC to allow the complete system to finish entering into
> deep sleep state.
> 
> These values vary based on the platform design and are specified
> through the device tree.
> 
> This patch has implementation to configure these timings which are must
> to have for proper deep sleep and wakeup operations.
> 
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> ---
>  drivers/soc/tegra/pmc.c | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/drivers/soc/tegra/pmc.c b/drivers/soc/tegra/pmc.c
> index ed83c0cd09a3..7e4a8f04f4c4 100644
> --- a/drivers/soc/tegra/pmc.c
> +++ b/drivers/soc/tegra/pmc.c
> @@ -89,6 +89,8 @@
>  
>  #define PMC_CPUPWRGOOD_TIMER		0xc8
>  #define PMC_CPUPWROFF_TIMER		0xcc
> +#define PMC_COREPWRGOOD_TIMER		0x3c
> +#define PMC_COREPWROFF_TIMER		0xe0
>  
>  #define PMC_PWR_DET_VALUE		0xe4
>  
> @@ -2291,6 +2293,7 @@ static const struct tegra_pmc_regs tegra20_pmc_regs = {
>  static void tegra20_pmc_init(struct tegra_pmc *pmc)
>  {
>  	u32 value;
> +	unsigned long osc, pmu, off;
>  
>  	/* Always enable CPU power request */
>  	value = tegra_pmc_readl(pmc, PMC_CNTRL);
> @@ -2316,6 +2319,15 @@ static void tegra20_pmc_init(struct tegra_pmc *pmc)
>  	value = tegra_pmc_readl(pmc, PMC_CNTRL);
>  	value |= PMC_CNTRL_SYSCLK_OE;
>  	tegra_pmc_writel(pmc, value, PMC_CNTRL);
> +
> +	osc = DIV_ROUND_UP_ULL(pmc->core_osc_time * 8192, 1000000);
> +	pmu = DIV_ROUND_UP_ULL(pmc->core_pmu_time * 32768, 1000000);
> +	off = DIV_ROUND_UP_ULL(pmc->core_off_time * 32768, 1000000);

IIUC, the first argument shall be explicitly of a type "long long", shouldn't it?
Otherwise the multiplication will overflow before division happens.

Thus:

	osc = DIV_ROUND_UP_ULL((u64)pmc->core_osc_time * 8192, 1000000);
	pmu = DIV_ROUND_UP_ULL((u64)pmc->core_pmu_time * 32768, 1000000);
	off = DIV_ROUND_UP_ULL((u64)pmc->core_off_time * 32768, 1000000);

Also, could you please tell what of the above multiplications could overflow u32 in
the first place? Maybe DIV_ROUND_UP_ULL isn't needed at all and DIV_ROUND_UP could be
use instead?

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 16/18] soc/tegra: pmc: Configure deep sleep control settings
  2019-06-28  2:12 ` [PATCH V5 16/18] soc/tegra: pmc: Configure deep sleep control settings Sowjanya Komatineni
  2019-06-29 13:00   ` Dmitry Osipenko
@ 2019-06-29 13:02   ` Dmitry Osipenko
  1 sibling, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 13:02 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

28.06.2019 5:12, Sowjanya Komatineni пишет:
> Tegra210 and prior Tegra chips have deep sleep entry and wakeup related
> timings which are platform specific that should be configured before
> entering into deep sleep.
> 
> Below are the timing specific configurations for deep sleep entry and
> wakeup.
> - Core rail power-on stabilization timer
> - OSC clock stabilization timer after SOC rail power is stabilized.
> - Core power off time is the minimum wake delay to keep the system
>   in deep sleep state irrespective of any quick wake event.
> 
> These values depends on the discharge time of regulators and turn OFF
> time of the PMIC to allow the complete system to finish entering into
> deep sleep state.
> 
> These values vary based on the platform design and are specified
> through the device tree.
> 
> This patch has implementation to configure these timings which are must
> to have for proper deep sleep and wakeup operations.
> 
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> ---
>  drivers/soc/tegra/pmc.c | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/drivers/soc/tegra/pmc.c b/drivers/soc/tegra/pmc.c
> index ed83c0cd09a3..7e4a8f04f4c4 100644
> --- a/drivers/soc/tegra/pmc.c
> +++ b/drivers/soc/tegra/pmc.c
> @@ -89,6 +89,8 @@
>  
>  #define PMC_CPUPWRGOOD_TIMER		0xc8
>  #define PMC_CPUPWROFF_TIMER		0xcc
> +#define PMC_COREPWRGOOD_TIMER		0x3c
> +#define PMC_COREPWROFF_TIMER		0xe0
>  
>  #define PMC_PWR_DET_VALUE		0xe4
>  
> @@ -2291,6 +2293,7 @@ static const struct tegra_pmc_regs tegra20_pmc_regs = {
>  static void tegra20_pmc_init(struct tegra_pmc *pmc)
>  {
>  	u32 value;
> +	unsigned long osc, pmu, off;

I'd write this as:

 	u32 value, osc, pmu, off;

Because "unsigned long" has the same size as u32 in this case.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 13/18] soc/tegra: pmc: Add pmc wake support for tegra210
  2019-06-28  2:12 ` [PATCH V5 13/18] soc/tegra: pmc: Add pmc wake support for tegra210 Sowjanya Komatineni
@ 2019-06-29 13:11   ` Dmitry Osipenko
  0 siblings, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 13:11 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

28.06.2019 5:12, Sowjanya Komatineni пишет:
> This patch implements PMC wakeup sequence for Tegra210 and defines
> common used RTC alarm wake event.
> 
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> ---
>  drivers/soc/tegra/pmc.c | 111 ++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 111 insertions(+)
> 
> diff --git a/drivers/soc/tegra/pmc.c b/drivers/soc/tegra/pmc.c
> index 91c84d0e66ae..194100d77243 100644
> --- a/drivers/soc/tegra/pmc.c
> +++ b/drivers/soc/tegra/pmc.c
> @@ -57,6 +57,12 @@
>  #define  PMC_CNTRL_SYSCLK_OE		BIT(11) /* system clock enable */
>  #define  PMC_CNTRL_SYSCLK_POLARITY	BIT(10) /* sys clk polarity */
>  #define  PMC_CNTRL_MAIN_RST		BIT(4)
> +#define  PMC_CNTRL_LATCH_WAKEUPS	BIT(5)
> +
> +#define PMC_WAKE_MASK			0x0c
> +#define PMC_WAKE_LEVEL			0x10
> +#define PMC_WAKE_STATUS			0x14
> +#define PMC_SW_WAKE_STATUS		0x18
>  
>  #define DPD_SAMPLE			0x020
>  #define  DPD_SAMPLE_ENABLE		BIT(0)
> @@ -87,6 +93,11 @@
>  
>  #define PMC_SCRATCH41			0x140
>  
> +#define PMC_WAKE2_MASK			0x160
> +#define PMC_WAKE2_LEVEL			0x164
> +#define PMC_WAKE2_STATUS		0x168
> +#define PMC_SW_WAKE2_STATUS		0x16c
> +
>  #define PMC_SENSOR_CTRL			0x1b0
>  #define  PMC_SENSOR_CTRL_SCRATCH_WRITE	BIT(2)
>  #define  PMC_SENSOR_CTRL_ENABLE_RST	BIT(1)
> @@ -1922,6 +1933,55 @@ static const struct irq_domain_ops tegra_pmc_irq_domain_ops = {
>  	.alloc = tegra_pmc_irq_alloc,
>  };
>  
> +static int tegra210_pmc_irq_set_wake(struct irq_data *data, unsigned int on)
> +{
> +	struct tegra_pmc *pmc = irq_data_get_irq_chip_data(data);
> +	unsigned int offset, bit;
> +	u32 value;
> +
> +	if (data->hwirq == ULONG_MAX)
> +		return 0;
> +
> +	offset = data->hwirq / 32;
> +	bit = data->hwirq % 32;
> +
> +	/*
> +	 * latch wakeups to SW_WAKE_STATUS register to capture events
> +	 * that would not make it into wakeup event register during LP0 exit.
> +	 */

Minor nit: start multi-line comments with a capital letter.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-06-28  2:12 ` [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks Sowjanya Komatineni
@ 2019-06-29 13:14   ` Dmitry Osipenko
  2019-06-29 15:10   ` Dmitry Osipenko
  2019-06-29 15:13   ` Dmitry Osipenko
  2 siblings, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 13:14 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland, pdeschrijver
  Cc: pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, josephl, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree

28.06.2019 5:12, Sowjanya Komatineni пишет:
> This patch adds system suspend and resume support for Tegra210
> clocks.
> 
> All the CAR controller settings are lost on suspend when core power
> goes off.
> 
> This patch has implementation for saving and restoring all the PLLs
> and clocks context during system suspend and resume to have the
> clocks back to same state for normal operation.
> 
> Acked-by: Thierry Reding <treding@nvidia.com>
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> ---
>  drivers/clk/tegra/clk-tegra210.c | 115 ++++++++++++++++++++++++++++++++++++++-
>  drivers/clk/tegra/clk.c          |  14 +++++
>  drivers/clk/tegra/clk.h          |   1 +
>  3 files changed, 127 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/clk/tegra/clk-tegra210.c b/drivers/clk/tegra/clk-tegra210.c
> index 1c08c53482a5..1b839544e086 100644
> --- a/drivers/clk/tegra/clk-tegra210.c
> +++ b/drivers/clk/tegra/clk-tegra210.c
> @@ -9,10 +9,12 @@
>  #include <linux/clkdev.h>
>  #include <linux/of.h>
>  #include <linux/of_address.h>
> +#include <linux/of_platform.h>
>  #include <linux/delay.h>
>  #include <linux/export.h>
>  #include <linux/mutex.h>
>  #include <linux/clk/tegra.h>
> +#include <linux/syscore_ops.h>
>  #include <dt-bindings/clock/tegra210-car.h>
>  #include <dt-bindings/reset/tegra210-car.h>
>  #include <linux/iopoll.h>
> @@ -20,6 +22,7 @@
>  #include <soc/tegra/pmc.h>
>  
>  #include "clk.h"
> +#include "clk-dfll.h"
>  #include "clk-id.h"
>  
>  /*
> @@ -225,6 +228,7 @@
>  
>  #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
>  #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
> +#define CPU_SOFTRST_CTRL 0x380
>  
>  #define LVL2_CLK_GATE_OVRA 0xf8
>  #define LVL2_CLK_GATE_OVRC 0x3a0
> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
>  	struct tegra_clk_pll_freq_table *fentry;
>  	struct tegra_clk_pll pllu;
>  	u32 reg;
> +	int ret;
>  
>  	for (fentry = pll_u_freq_table; fentry->input_rate; fentry++) {
>  		if (fentry->input_rate == pll_ref_freq)
> @@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
>  	fence_udelay(1, clk_base);
>  	reg |= PLL_ENABLE;
>  	writel(reg, clk_base + PLLU_BASE);
> +	fence_udelay(1, clk_base);
>  
> -	readl_relaxed_poll_timeout_atomic(clk_base + PLLU_BASE, reg,
> -					  reg & PLL_BASE_LOCK, 2, 1000);
> -	if (!(reg & PLL_BASE_LOCK)) {
> +	ret = tegra210_wait_for_mask(&pllu, PLLU_BASE, PLL_BASE_LOCK);
> +	if (ret) {
>  		pr_err("Timed out waiting for PLL_U to lock\n");
>  		return -ETIMEDOUT;
>  	}
> @@ -3283,6 +3288,103 @@ static void tegra210_disable_cpu_clock(u32 cpu)
>  }
>  
>  #ifdef CONFIG_PM_SLEEP
> +static u32 cpu_softrst_ctx[3];
> +static struct platform_device *dfll_pdev;
> +#define car_readl(_base, _off) readl_relaxed(clk_base + (_base) + ((_off) * 4))
> +#define car_writel(_val, _base, _off) \
> +		writel_relaxed(_val, clk_base + (_base) + ((_off) * 4))
> +
> +static int tegra210_clk_suspend(void)
> +{
> +	unsigned int i;
> +	struct device_node *node;
> +
> +	tegra_cclkg_burst_policy_save_context();
> +
> +	if (!dfll_pdev) {
> +		node = of_find_compatible_node(NULL, NULL,
> +					       "nvidia,tegra210-dfll");
> +		if (node)
> +			dfll_pdev = of_find_device_by_node(node);
> +
> +		of_node_put(node);
> +		if (!dfll_pdev)
> +			pr_err("dfll node not found. no suspend for dfll\n");
> +	}
> +
> +	if (dfll_pdev)
> +		tegra_dfll_suspend(dfll_pdev);
> +
> +	/* Enable PLLP_OUT_CPU after dfll suspend */
> +	tegra_clk_set_pllp_out_cpu(true);
> +
> +	tegra_sclk_cclklp_burst_policy_save_context();
> +
> +	clk_save_context();
> +
> +	for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
> +		cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
> +
> +	return 0;
> +}
> +
> +static void tegra210_clk_resume(void)
> +{
> +	unsigned int i;
> +	struct clk_hw *parent;
> +	struct clk *clk;
> +
> +	/*
> +	 * clk_restore_context restores clocks as per the clock tree.
> +	 *
> +	 * dfllCPU_out is first in the clock tree to get restored and it
> +	 * involves programming DFLL controller along with restoring CPUG
> +	 * clock burst policy.
> +	 *
> +	 * DFLL programming needs dfll_ref and dfll_soc peripheral clocks
> +	 * to be restores which are part ofthe peripheral clocks.
> +	 * So, peripheral clocks restore should happen prior to dfll clock
> +	 * restore.
> +	 */
> +
> +	tegra_clk_osc_resume(clk_base);
> +	for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
> +		car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
> +
> +	/* restore all plls and peripheral clocks */
> +	tegra210_init_pllu();
> +	clk_restore_context();
> +
> +	fence_udelay(5, clk_base);
> +
> +	/* resume SCLK and CPULP clocks */
> +	tegra_sclk_cpulp_burst_policy_restore_context();
> +
> +	/*
> +	 * restore CPUG clocks:
> +	 * - enable DFLL in open loop mode
> +	 * - switch CPUG to DFLL clock source
> +	 * - close DFLL loop
> +	 * - sync PLLX state
> +	 */
> +	if (dfll_pdev)
> +		tegra_dfll_resume(dfll_pdev, false);
> +
> +	tegra_cclkg_burst_policy_restore_context();
> +	fence_udelay(2, clk_base);
> +
> +	if (dfll_pdev)
> +		tegra_dfll_resume(dfll_pdev, true);
> +
> +	parent = clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
> +	clk = clks[TEGRA210_CLK_PLL_X];
> +	if (parent != __clk_get_hw(clk))
> +		tegra_clk_sync_state_pll(__clk_get_hw(clk));
> +
> +	/* Disable PLL_OUT_CPU after DFLL resume */
> +	tegra_clk_set_pllp_out_cpu(false);
> +}
> +
>  static void tegra210_cpu_clock_suspend(void)
>  {
>  	/* switch coresite to clk_m, save off original source */
> @@ -3298,6 +3400,11 @@ static void tegra210_cpu_clock_resume(void)
>  }
>  #endif
>  
> +static struct syscore_ops tegra_clk_syscore_ops = {
> +	.suspend = tegra210_clk_suspend,
> +	.resume = tegra210_clk_resume,
> +};
> +
>  static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
>  	.wait_for_reset	= tegra210_wait_cpu_in_reset,
>  	.disable_clock	= tegra210_disable_cpu_clock,
> @@ -3583,5 +3690,7 @@ static void __init tegra210_clock_init(struct device_node *np)
>  	tegra210_mbist_clk_init();
>  
>  	tegra_cpu_car_ops = &tegra210_cpu_car_ops;
> +
> +	register_syscore_ops(&tegra_clk_syscore_ops);
>  }
>  CLK_OF_DECLARE(tegra210, "nvidia,tegra210-car", tegra210_clock_init);
> diff --git a/drivers/clk/tegra/clk.c b/drivers/clk/tegra/clk.c
> index 9e863362d2bf..96cc9937ea37 100644
> --- a/drivers/clk/tegra/clk.c
> +++ b/drivers/clk/tegra/clk.c
> @@ -23,6 +23,7 @@
>  #define CLK_OUT_ENB_W			0x364
>  #define CLK_OUT_ENB_X			0x280
>  #define CLK_OUT_ENB_Y			0x298
> +#define CLK_ENB_PLLP_OUT_CPU		BIT(31)
>  #define CLK_OUT_ENB_SET_L		0x320
>  #define CLK_OUT_ENB_CLR_L		0x324
>  #define CLK_OUT_ENB_SET_H		0x328
> @@ -205,6 +206,19 @@ const struct tegra_clk_periph_regs *get_reg_bank(int clkid)
>  	}
>  }
>  
> +void tegra_clk_set_pllp_out_cpu(bool enable)
> +{
> +	u32 val;
> +
> +	val = readl_relaxed(clk_base + CLK_OUT_ENB_Y);
> +	if (enable)
> +		val |= CLK_ENB_PLLP_OUT_CPU;
> +	else
> +		val &= ~CLK_ENB_PLLP_OUT_CPU;
> +
> +	writel_relaxed(val, clk_base + CLK_OUT_ENB_Y);
> +}

Do we needed fence_udelay() here? I'm a bit confused about when the fence should be
used for the CLK hardware.. Maybe Peter could clarify?

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 09/18] clk: tegra: Add save and restore context support for peripheral clocks
  2019-06-28  2:12 ` [PATCH V5 09/18] clk: tegra: Add save and restore context support for peripheral clocks Sowjanya Komatineni
@ 2019-06-29 13:17   ` Dmitry Osipenko
  0 siblings, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 13:17 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

28.06.2019 5:12, Sowjanya Komatineni пишет:
> This patch implements save and restore context for peripheral fixed
> clock ops, peripheral gate clock ops, sdmmc mux clock ops, and
> peripheral clock ops.
> 
> During system suspend, core power goes off and looses the settings
> of the Tegra CAR controller registers.
> 
> So during suspend entry clock and reset state of peripherals is saved
> and on resume they are restored to have clocks back to same rate and
> state as before suspend.
> 
> Acked-by: Thierry Reding <treding@nvidia.com>
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> ---
>  drivers/clk/tegra/clk-periph-fixed.c | 31 ++++++++++++++++++++++++++
>  drivers/clk/tegra/clk-periph-gate.c  | 34 ++++++++++++++++++++++++++++
>  drivers/clk/tegra/clk-periph.c       | 43 ++++++++++++++++++++++++++++++++++++
>  drivers/clk/tegra/clk-sdmmc-mux.c    | 30 +++++++++++++++++++++++++
>  drivers/clk/tegra/clk.h              |  8 +++++++
>  5 files changed, 146 insertions(+)
> 
> diff --git a/drivers/clk/tegra/clk-periph-fixed.c b/drivers/clk/tegra/clk-periph-fixed.c
> index c088e7a280df..981f68b0a937 100644
> --- a/drivers/clk/tegra/clk-periph-fixed.c
> +++ b/drivers/clk/tegra/clk-periph-fixed.c
> @@ -60,11 +60,42 @@ tegra_clk_periph_fixed_recalc_rate(struct clk_hw *hw,
>  	return (unsigned long)rate;
>  }
>  
> +static int tegra_clk_periph_fixed_save_context(struct clk_hw *hw)
> +{
> +	struct tegra_clk_periph_fixed *fixed = to_tegra_clk_periph_fixed(hw);
> +	u32 mask = 1 << (fixed->num % 32);
> +
> +	fixed->enb_ctx = readl(fixed->base + fixed->regs->enb_reg) & mask;
> +	fixed->rst_ctx = readl(fixed->base + fixed->regs->rst_reg) & mask;
> +

readl_relaxed() ?


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 08/18] clk: tegra: Add suspend resume support for DFLL
  2019-06-28  2:12 ` [PATCH V5 08/18] clk: tegra: Add suspend resume support for DFLL Sowjanya Komatineni
@ 2019-06-29 13:28   ` Dmitry Osipenko
  2019-06-29 21:45     ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 13:28 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

28.06.2019 5:12, Sowjanya Komatineni пишет:
> This patch creates APIs for supporting Tegra210 clock driver to
> perform DFLL suspend and resume operation.
> 
> During suspend, DFLL mode is saved and on resume Tegra210 clock driver
> invokes DFLL resume API to re-initialize DFLL to enable target device
> clock in open loop mode or closed loop mode.
> 
> Acked-by: Thierry Reding <treding@nvidia.com>
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> ---
>  drivers/clk/tegra/clk-dfll.c | 78 ++++++++++++++++++++++++++++++++++++++++++++
>  drivers/clk/tegra/clk-dfll.h |  2 ++
>  2 files changed, 80 insertions(+)
> 
> diff --git a/drivers/clk/tegra/clk-dfll.c b/drivers/clk/tegra/clk-dfll.c
> index f8688c2ddf1a..a1f37cf99b00 100644
> --- a/drivers/clk/tegra/clk-dfll.c
> +++ b/drivers/clk/tegra/clk-dfll.c
> @@ -277,6 +277,7 @@ struct tegra_dfll {
>  	unsigned long			dvco_rate_min;
>  
>  	enum dfll_ctrl_mode		mode;
> +	enum dfll_ctrl_mode		resume_mode;
>  	enum dfll_tune_range		tune_range;
>  	struct dentry			*debugfs_dir;
>  	struct clk_hw			dfll_clk_hw;
> @@ -1864,6 +1865,83 @@ static int dfll_fetch_common_params(struct tegra_dfll *td)
>  }
>  
>  /*
> + * tegra_dfll_suspend
> + * @pdev: DFLL instance
> + *
> + * dfll controls clock/voltage to other devices, including CPU. Therefore,
> + * dfll driver pm suspend callback does not stop cl-dvfs operations.
> + */
> +void tegra_dfll_suspend(struct platform_device *pdev)
> +{
> +	struct tegra_dfll *td = dev_get_drvdata(&pdev->dev);
> +
> +	if (!td)
> +		return;
> +
> +	if (td->mode <= DFLL_DISABLED)
> +		return;
> +
> +	td->resume_mode = td->mode;
> +	switch (td->mode) {
> +	case DFLL_CLOSED_LOOP:
> +		dfll_set_mode(td, DFLL_CLOSED_LOOP);
> +		dfll_set_frequency_request(td, &td->last_req);
> +
> +		dfll_unlock(td);
> +		break;
> +	default:
> +		break;
> +	}
> +}
> +
> +/**
> + * tegra_dfll_resume - reprogram the DFLL after context-loss
> + * @pdev: DFLL instance
> + *
> + * Re-initialize and enable target device clock in open loop mode. Called
> + * directly from SoC clock resume syscore operation. Closed loop will be
> + * re-entered in platform syscore ops as well after CPU clock source is
> + * switched to DFLL in open loop.
> + */
> +void tegra_dfll_resume(struct platform_device *pdev, bool on_dfll)
> +{
> +	struct tegra_dfll *td = dev_get_drvdata(&pdev->dev);
> +
> +	if (!td)
> +		return;
> +
> +	if (on_dfll) {
> +		if (td->resume_mode == DFLL_CLOSED_LOOP)
> +			dfll_lock(td);
> +		td->resume_mode = DFLL_DISABLED;
> +		return;
> +	}
> +
> +	reset_control_deassert(td->dvco_rst);
> +
> +	pm_runtime_get(td->dev);

pm_runtime_get_sync()?

Otherwise looks like you're risking a lot here because pm_runtime_get() is an
asynchronous request.

> +	/* Re-init DFLL */
> +	dfll_init_out_if(td);
> +	dfll_set_default_params(td);
> +	dfll_set_open_loop_config(td);
> +
> +	pm_runtime_put(td->dev);
> +
> +	/* Restore last request and mode up to open loop */
> +	switch (td->resume_mode) {
> +	case DFLL_CLOSED_LOOP:
> +	case DFLL_OPEN_LOOP:
> +		dfll_set_mode(td, DFLL_OPEN_LOOP);
> +		if (td->pmu_if == TEGRA_DFLL_PMU_I2C)
> +			dfll_i2c_set_output_enabled(td, false);
> +		break;
> +	default:
> +		break;
> +	}
> +}



^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 06/18] clk: tegra: Save and restore CPU and System clocks context
  2019-06-28  2:12 ` [PATCH V5 06/18] clk: tegra: Save and restore CPU and System clocks context Sowjanya Komatineni
@ 2019-06-29 13:33   ` Dmitry Osipenko
  2019-06-29 15:26   ` Dmitry Osipenko
  1 sibling, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 13:33 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

28.06.2019 5:12, Sowjanya Komatineni пишет:
> During system suspend state, core power goes off and looses all the
> CAR controller register settings.
> 
> This patch creates APIs for saving and restoring the context of Tegra
> CPUG, CPULP and SCLK.
> 
> CPU and System clock context includes
> - CPUG, CPULP, and SCLK burst policy settings for clock sourcea of all
>   their normal states.
> - SCLK divisor and System clock rate for restoring SCLK, AHB and APB
>   rates on resume.
> - OSC_DIV settings which are used as reference clock input to some PLLs.
> - SPARE_REG and CLK_MASK settings.
> 
> These APIs are used in Tegra210 clock driver during suspend and resume
> operation.
> 
> Acked-by: Thierry Reding <treding@nvidia.com>
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> ---
>  drivers/clk/tegra/clk-tegra-super-gen4.c |  4 --
>  drivers/clk/tegra/clk.c                  | 80 ++++++++++++++++++++++++++++++++
>  drivers/clk/tegra/clk.h                  | 14 ++++++
>  3 files changed, 94 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/clk/tegra/clk-tegra-super-gen4.c b/drivers/clk/tegra/clk-tegra-super-gen4.c
> index cdfe7c9697e1..ed69ec4d883e 100644
> --- a/drivers/clk/tegra/clk-tegra-super-gen4.c
> +++ b/drivers/clk/tegra/clk-tegra-super-gen4.c
> @@ -19,10 +19,6 @@
>  #define PLLX_MISC2 0x514
>  #define PLLX_MISC3 0x518
>  
> -#define CCLKG_BURST_POLICY 0x368
> -#define CCLKLP_BURST_POLICY 0x370
> -#define SCLK_BURST_POLICY 0x028
> -#define SYSTEM_CLK_RATE 0x030
>  #define SCLK_DIVIDER 0x2c
>  
>  static DEFINE_SPINLOCK(sysrate_lock);
> diff --git a/drivers/clk/tegra/clk.c b/drivers/clk/tegra/clk.c
> index 573e3c967ae1..9e863362d2bf 100644
> --- a/drivers/clk/tegra/clk.c
> +++ b/drivers/clk/tegra/clk.c
> @@ -70,6 +70,12 @@ static struct clk **clks;
>  static int clk_num;
>  static struct clk_onecell_data clk_data;
>  
> +static u32 cclkg_burst_policy_ctx[2];
> +static u32 cclklp_burst_policy_ctx[2];
> +static u32 sclk_burst_policy_ctx[2];
> +static u32 sys_clk_divisor_ctx, system_rate_ctx;
> +static u32 spare_ctx, misc_clk_enb_ctx, clk_arm_ctx;
> +
>  /* Handlers for SoC-specific reset lines */
>  static int (*special_reset_assert)(unsigned long);
>  static int (*special_reset_deassert)(unsigned long);
> @@ -199,6 +205,80 @@ const struct tegra_clk_periph_regs *get_reg_bank(int clkid)
>  	}
>  }
>  
> +void tegra_cclkg_burst_policy_save_context(void)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < BURST_POLICY_REG_SIZE; i++)
> +		cclkg_burst_policy_ctx[i] = readl_relaxed(clk_base +
> +							  CCLKG_BURST_POLICY +
> +							  (i * 4));
> +}
> +
> +void tegra_cclkg_burst_policy_restore_context(void)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < BURST_POLICY_REG_SIZE; i++)
> +		writel_relaxed(cclkg_burst_policy_ctx[i],
> +			       clk_base + CCLKG_BURST_POLICY + (i * 4));
> +
> +	fence_udelay(2, clk_base);
> +}
> +
> +void tegra_sclk_cclklp_burst_policy_save_context(void)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < BURST_POLICY_REG_SIZE; i++) {
> +		cclklp_burst_policy_ctx[i] = readl_relaxed(clk_base +
> +							  CCLKLP_BURST_POLICY +
> +							  (i * 4));
> +
> +		sclk_burst_policy_ctx[i] = readl_relaxed(clk_base +
> +							  SCLK_BURST_POLICY +
> +							  (i * 4));
> +	}
> +
> +	sys_clk_divisor_ctx = readl_relaxed(clk_base + SYS_CLK_DIV);
> +	system_rate_ctx = readl_relaxed(clk_base + SYSTEM_CLK_RATE);
> +	spare_ctx = readl_relaxed(clk_base + SPARE_REG0);
> +	misc_clk_enb_ctx = readl_relaxed(clk_base + MISC_CLK_ENB);
> +	clk_arm_ctx = readl_relaxed(clk_base + CLK_MASK_ARM);
> +}
> +
> +void tegra_sclk_cpulp_burst_policy_restore_context(void)
> +{
> +	unsigned int i;
> +	u32 val;
> +
> +	/*
> +	 * resume SCLK and CPULP clocks
> +	 * for SCLk, set safe dividers values first and then restore source
> +	 * and dividers
> +	 */
> +
> +	writel_relaxed(0x1, clk_base + SYSTEM_CLK_RATE);
> +	val = readl_relaxed(clk_base + SYS_CLK_DIV);
> +	if (val < sys_clk_divisor_ctx)
> +		writel_relaxed(sys_clk_divisor_ctx, clk_base + SYS_CLK_DIV);
> +
> +	fence_udelay(2, clk_base);
> +
> +	for (i = 0; i < BURST_POLICY_REG_SIZE; i++) {
> +		writel_relaxed(cclklp_burst_policy_ctx[i],
> +			       clk_base + CCLKLP_BURST_POLICY + (i * 4));
> +		writel_relaxed(sclk_burst_policy_ctx[i],
> +			       clk_base + SCLK_BURST_POLICY + (i * 4));
> +	}
> +
> +	writel_relaxed(sys_clk_divisor_ctx, clk_base + SYS_CLK_DIV);
> +	writel_relaxed(system_rate_ctx, clk_base + SYSTEM_CLK_RATE);
> +	writel_relaxed(spare_ctx, clk_base + SPARE_REG0);
> +	writel_relaxed(misc_clk_enb_ctx, clk_base + MISC_CLK_ENB);
> +	writel_relaxed(clk_arm_ctx, clk_base + CLK_MASK_ARM);

Why fence_udelay was needed above and not needed here?

> +}



^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-06-28  2:12 ` [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks Sowjanya Komatineni
  2019-06-29 13:14   ` Dmitry Osipenko
@ 2019-06-29 15:10   ` Dmitry Osipenko
  2019-07-13  5:54     ` Sowjanya Komatineni
  2019-06-29 15:13   ` Dmitry Osipenko
  2 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 15:10 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

28.06.2019 5:12, Sowjanya Komatineni пишет:
> This patch adds system suspend and resume support for Tegra210
> clocks.
> 
> All the CAR controller settings are lost on suspend when core power
> goes off.
> 
> This patch has implementation for saving and restoring all the PLLs
> and clocks context during system suspend and resume to have the
> clocks back to same state for normal operation.
> 
> Acked-by: Thierry Reding <treding@nvidia.com>
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> ---
>  drivers/clk/tegra/clk-tegra210.c | 115 ++++++++++++++++++++++++++++++++++++++-
>  drivers/clk/tegra/clk.c          |  14 +++++
>  drivers/clk/tegra/clk.h          |   1 +
>  3 files changed, 127 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/clk/tegra/clk-tegra210.c b/drivers/clk/tegra/clk-tegra210.c
> index 1c08c53482a5..1b839544e086 100644
> --- a/drivers/clk/tegra/clk-tegra210.c
> +++ b/drivers/clk/tegra/clk-tegra210.c
> @@ -9,10 +9,12 @@
>  #include <linux/clkdev.h>
>  #include <linux/of.h>
>  #include <linux/of_address.h>
> +#include <linux/of_platform.h>
>  #include <linux/delay.h>
>  #include <linux/export.h>
>  #include <linux/mutex.h>
>  #include <linux/clk/tegra.h>
> +#include <linux/syscore_ops.h>
>  #include <dt-bindings/clock/tegra210-car.h>
>  #include <dt-bindings/reset/tegra210-car.h>
>  #include <linux/iopoll.h>
> @@ -20,6 +22,7 @@
>  #include <soc/tegra/pmc.h>
>  
>  #include "clk.h"
> +#include "clk-dfll.h"
>  #include "clk-id.h"
>  
>  /*
> @@ -225,6 +228,7 @@
>  
>  #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
>  #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
> +#define CPU_SOFTRST_CTRL 0x380
>  
>  #define LVL2_CLK_GATE_OVRA 0xf8
>  #define LVL2_CLK_GATE_OVRC 0x3a0
> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
>  	struct tegra_clk_pll_freq_table *fentry;
>  	struct tegra_clk_pll pllu;
>  	u32 reg;
> +	int ret;
>  
>  	for (fentry = pll_u_freq_table; fentry->input_rate; fentry++) {
>  		if (fentry->input_rate == pll_ref_freq)
> @@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
>  	fence_udelay(1, clk_base);
>  	reg |= PLL_ENABLE;
>  	writel(reg, clk_base + PLLU_BASE);
> +	fence_udelay(1, clk_base);
>  
> -	readl_relaxed_poll_timeout_atomic(clk_base + PLLU_BASE, reg,
> -					  reg & PLL_BASE_LOCK, 2, 1000);
> -	if (!(reg & PLL_BASE_LOCK)) {
> +	ret = tegra210_wait_for_mask(&pllu, PLLU_BASE, PLL_BASE_LOCK);
> +	if (ret) {
>  		pr_err("Timed out waiting for PLL_U to lock\n");
>  		return -ETIMEDOUT;
>  	}
> @@ -3283,6 +3288,103 @@ static void tegra210_disable_cpu_clock(u32 cpu)
>  }
>  
>  #ifdef CONFIG_PM_SLEEP
> +static u32 cpu_softrst_ctx[3];
> +static struct platform_device *dfll_pdev;
> +#define car_readl(_base, _off) readl_relaxed(clk_base + (_base) + ((_off) * 4))
> +#define car_writel(_val, _base, _off) \
> +		writel_relaxed(_val, clk_base + (_base) + ((_off) * 4))
> +
> +static int tegra210_clk_suspend(void)
> +{
> +	unsigned int i;
> +	struct device_node *node;
> +
> +	tegra_cclkg_burst_policy_save_context();
> +
> +	if (!dfll_pdev) {
> +		node = of_find_compatible_node(NULL, NULL,
> +					       "nvidia,tegra210-dfll");
> +		if (node)
> +			dfll_pdev = of_find_device_by_node(node);
> +
> +		of_node_put(node);
> +		if (!dfll_pdev)
> +			pr_err("dfll node not found. no suspend for dfll\n");
> +	}
> +
> +	if (dfll_pdev)
> +		tegra_dfll_suspend(dfll_pdev);
> +
> +	/* Enable PLLP_OUT_CPU after dfll suspend */
> +	tegra_clk_set_pllp_out_cpu(true);
> +
> +	tegra_sclk_cclklp_burst_policy_save_context();
> +
> +	clk_save_context();
> +
> +	for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
> +		cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
> +
> +	return 0;
> +}
> +
> +static void tegra210_clk_resume(void)
> +{
> +	unsigned int i;
> +	struct clk_hw *parent;
> +	struct clk *clk;
> +
> +	/*
> +	 * clk_restore_context restores clocks as per the clock tree.
> +	 *
> +	 * dfllCPU_out is first in the clock tree to get restored and it
> +	 * involves programming DFLL controller along with restoring CPUG
> +	 * clock burst policy.
> +	 *
> +	 * DFLL programming needs dfll_ref and dfll_soc peripheral clocks
> +	 * to be restores which are part ofthe peripheral clocks.
> +	 * So, peripheral clocks restore should happen prior to dfll clock
> +	 * restore.
> +	 */
> +
> +	tegra_clk_osc_resume(clk_base);
> +	for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
> +		car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
> +
> +	/* restore all plls and peripheral clocks */
> +	tegra210_init_pllu();
> +	clk_restore_context();
> +
> +	fence_udelay(5, clk_base);
> +
> +	/* resume SCLK and CPULP clocks */
> +	tegra_sclk_cpulp_burst_policy_restore_context();
> +
> +	/*
> +	 * restore CPUG clocks:
> +	 * - enable DFLL in open loop mode
> +	 * - switch CPUG to DFLL clock source
> +	 * - close DFLL loop
> +	 * - sync PLLX state
> +	 */
> +	if (dfll_pdev)
> +		tegra_dfll_resume(dfll_pdev, false);
> +
> +	tegra_cclkg_burst_policy_restore_context();
> +	fence_udelay(2, clk_base);
> +
> +	if (dfll_pdev)
> +		tegra_dfll_resume(dfll_pdev, true);
> +
> +	parent = clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
> +	clk = clks[TEGRA210_CLK_PLL_X];
> +	if (parent != __clk_get_hw(clk))
> +		tegra_clk_sync_state_pll(__clk_get_hw(clk));
> +
> +	/* Disable PLL_OUT_CPU after DFLL resume */
> +	tegra_clk_set_pllp_out_cpu(false);
> +}
> +
>  static void tegra210_cpu_clock_suspend(void)
>  {
>  	/* switch coresite to clk_m, save off original source */
> @@ -3298,6 +3400,11 @@ static void tegra210_cpu_clock_resume(void)
>  }
>  #endif
>  
> +static struct syscore_ops tegra_clk_syscore_ops = {
> +	.suspend = tegra210_clk_suspend,
> +	.resume = tegra210_clk_resume,
> +};
> +
>  static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
>  	.wait_for_reset	= tegra210_wait_cpu_in_reset,
>  	.disable_clock	= tegra210_disable_cpu_clock,
> @@ -3583,5 +3690,7 @@ static void __init tegra210_clock_init(struct device_node *np)
>  	tegra210_mbist_clk_init();
>  
>  	tegra_cpu_car_ops = &tegra210_cpu_car_ops;
> +
> +	register_syscore_ops(&tegra_clk_syscore_ops);
>  }

Is it really worthwhile to use syscore_ops for suspend/resume given that drivers for
won't resume before the CLK driver anyway? Are there any other options for CLK
suspend/resume?

I'm also not sure whether PM runtime API could be used at all in the context of
syscore_ops ..

Secondly, what about to use generic clk_save_context() / clk_restore_context()
helpers for the suspend-resume? It looks to me that some other essential (and proper)
platform driver (soc/tegra/? PMC?) should suspend-resume the clocks using the generic
CLK Framework API.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-06-28  2:12 ` [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks Sowjanya Komatineni
  2019-06-29 13:14   ` Dmitry Osipenko
  2019-06-29 15:10   ` Dmitry Osipenko
@ 2019-06-29 15:13   ` Dmitry Osipenko
  2 siblings, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 15:13 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

28.06.2019 5:12, Sowjanya Komatineni пишет:
> This patch adds system suspend and resume support for Tegra210
> clocks.
> 
> All the CAR controller settings are lost on suspend when core power
> goes off.
> 
> This patch has implementation for saving and restoring all the PLLs
> and clocks context during system suspend and resume to have the
> clocks back to same state for normal operation.
> 
> Acked-by: Thierry Reding <treding@nvidia.com>
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> ---

I'd also change the commit's title to something more brief and explicit, like "clk:
tegra210: Support suspend-resume".

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 06/18] clk: tegra: Save and restore CPU and System clocks context
  2019-06-28  2:12 ` [PATCH V5 06/18] clk: tegra: Save and restore CPU and System clocks context Sowjanya Komatineni
  2019-06-29 13:33   ` Dmitry Osipenko
@ 2019-06-29 15:26   ` Dmitry Osipenko
  1 sibling, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 15:26 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

28.06.2019 5:12, Sowjanya Komatineni пишет:
> During system suspend state, core power goes off and looses all the
> CAR controller register settings.
> 
> This patch creates APIs for saving and restoring the context of Tegra
> CPUG, CPULP and SCLK.
> 
> CPU and System clock context includes
> - CPUG, CPULP, and SCLK burst policy settings for clock sourcea of all
>   their normal states.
> - SCLK divisor and System clock rate for restoring SCLK, AHB and APB
>   rates on resume.
> - OSC_DIV settings which are used as reference clock input to some PLLs.
> - SPARE_REG and CLK_MASK settings.
> 
> These APIs are used in Tegra210 clock driver during suspend and resume
> operation.
> 
> Acked-by: Thierry Reding <treding@nvidia.com>
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> ---
>  drivers/clk/tegra/clk-tegra-super-gen4.c |  4 --
>  drivers/clk/tegra/clk.c                  | 80 ++++++++++++++++++++++++++++++++
>  drivers/clk/tegra/clk.h                  | 14 ++++++
>  3 files changed, 94 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/clk/tegra/clk-tegra-super-gen4.c b/drivers/clk/tegra/clk-tegra-super-gen4.c
> index cdfe7c9697e1..ed69ec4d883e 100644
> --- a/drivers/clk/tegra/clk-tegra-super-gen4.c
> +++ b/drivers/clk/tegra/clk-tegra-super-gen4.c
> @@ -19,10 +19,6 @@
>  #define PLLX_MISC2 0x514
>  #define PLLX_MISC3 0x518
>  
> -#define CCLKG_BURST_POLICY 0x368
> -#define CCLKLP_BURST_POLICY 0x370
> -#define SCLK_BURST_POLICY 0x028
> -#define SYSTEM_CLK_RATE 0x030
>  #define SCLK_DIVIDER 0x2c
>  
>  static DEFINE_SPINLOCK(sysrate_lock);
> diff --git a/drivers/clk/tegra/clk.c b/drivers/clk/tegra/clk.c
> index 573e3c967ae1..9e863362d2bf 100644
> --- a/drivers/clk/tegra/clk.c
> +++ b/drivers/clk/tegra/clk.c
> @@ -70,6 +70,12 @@ static struct clk **clks;
>  static int clk_num;
>  static struct clk_onecell_data clk_data;
>  
> +static u32 cclkg_burst_policy_ctx[2];
> +static u32 cclklp_burst_policy_ctx[2];
> +static u32 sclk_burst_policy_ctx[2];
> +static u32 sys_clk_divisor_ctx, system_rate_ctx;
> +static u32 spare_ctx, misc_clk_enb_ctx, clk_arm_ctx;
> +
>  /* Handlers for SoC-specific reset lines */
>  static int (*special_reset_assert)(unsigned long);
>  static int (*special_reset_deassert)(unsigned long);
> @@ -199,6 +205,80 @@ const struct tegra_clk_periph_regs *get_reg_bank(int clkid)
>  	}
>  }
>  
> +void tegra_cclkg_burst_policy_save_context(void)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < BURST_POLICY_REG_SIZE; i++)
> +		cclkg_burst_policy_ctx[i] = readl_relaxed(clk_base +
> +							  CCLKG_BURST_POLICY +
> +							  (i * 4));
> +}
> +
> +void tegra_cclkg_burst_policy_restore_context(void)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < BURST_POLICY_REG_SIZE; i++)
> +		writel_relaxed(cclkg_burst_policy_ctx[i],
> +			       clk_base + CCLKG_BURST_POLICY + (i * 4));
> +
> +	fence_udelay(2, clk_base);
> +}
> +
> +void tegra_sclk_cclklp_burst_policy_save_context(void)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < BURST_POLICY_REG_SIZE; i++) {
> +		cclklp_burst_policy_ctx[i] = readl_relaxed(clk_base +
> +							  CCLKLP_BURST_POLICY +
> +							  (i * 4));
> +
> +		sclk_burst_policy_ctx[i] = readl_relaxed(clk_base +
> +							  SCLK_BURST_POLICY +
> +							  (i * 4));
> +	}
> +
> +	sys_clk_divisor_ctx = readl_relaxed(clk_base + SYS_CLK_DIV);
> +	system_rate_ctx = readl_relaxed(clk_base + SYSTEM_CLK_RATE);
> +	spare_ctx = readl_relaxed(clk_base + SPARE_REG0);
> +	misc_clk_enb_ctx = readl_relaxed(clk_base + MISC_CLK_ENB);
> +	clk_arm_ctx = readl_relaxed(clk_base + CLK_MASK_ARM);
> +}
> +
> +void tegra_sclk_cpulp_burst_policy_restore_context(void)
> +{
> +	unsigned int i;
> +	u32 val;
> +
> +	/*
> +	 * resume SCLK and CPULP clocks
> +	 * for SCLk, set safe dividers values first and then restore source
> +	 * and dividers
> +	 */
> +
> +	writel_relaxed(0x1, clk_base + SYSTEM_CLK_RATE);
> +	val = readl_relaxed(clk_base + SYS_CLK_DIV);
> +	if (val < sys_clk_divisor_ctx)
> +		writel_relaxed(sys_clk_divisor_ctx, clk_base + SYS_CLK_DIV);
> +
> +	fence_udelay(2, clk_base);
> +
> +	for (i = 0; i < BURST_POLICY_REG_SIZE; i++) {
> +		writel_relaxed(cclklp_burst_policy_ctx[i],
> +			       clk_base + CCLKLP_BURST_POLICY + (i * 4));
> +		writel_relaxed(sclk_burst_policy_ctx[i],
> +			       clk_base + SCLK_BURST_POLICY + (i * 4));
> +	}
> +
> +	writel_relaxed(sys_clk_divisor_ctx, clk_base + SYS_CLK_DIV);
> +	writel_relaxed(system_rate_ctx, clk_base + SYSTEM_CLK_RATE);
> +	writel_relaxed(spare_ctx, clk_base + SPARE_REG0);
> +	writel_relaxed(misc_clk_enb_ctx, clk_base + MISC_CLK_ENB);
> +	writel_relaxed(clk_arm_ctx, clk_base + CLK_MASK_ARM);
> +}
> +
>  struct clk ** __init tegra_clk_init(void __iomem *regs, int num, int banks)
>  {
>  	clk_base = regs;
> diff --git a/drivers/clk/tegra/clk.h b/drivers/clk/tegra/clk.h
> index 8532f5150091..c66b0a73bb01 100644
> --- a/drivers/clk/tegra/clk.h
> +++ b/drivers/clk/tegra/clk.h
> @@ -10,6 +10,16 @@
>  #include <linux/clkdev.h>
>  #include <linux/delay.h>
>  
> +#define SCLK_BURST_POLICY	0x28
> +#define SYSTEM_CLK_RATE		0x30
> +#define CLK_MASK_ARM		0x44
> +#define MISC_CLK_ENB		0x48
> +#define CCLKG_BURST_POLICY	0x368
> +#define CCLKLP_BURST_POLICY	0x370
> +#define SYS_CLK_DIV		0x400
> +#define SPARE_REG0		0x55c
> +#define BURST_POLICY_REG_SIZE	2

"clk-tegra30.c",  "clk-tegra114.c" and "clk-tegra124.c" also define the
CCLKG_BURST_POLICY .. apparently you haven't tried to compile ARM32 kernel because I
assume that compile should bark at the re-definitions.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-06-29 12:38         ` Dmitry Osipenko
@ 2019-06-29 15:40           ` Dmitry Osipenko
  0 siblings, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 15:40 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

29.06.2019 15:38, Dmitry Osipenko пишет:
> 29.06.2019 2:00, Sowjanya Komatineni пишет:
>>
>> On 6/28/19 5:05 AM, Dmitry Osipenko wrote:
>>> 28.06.2019 14:56, Dmitry Osipenko пишет:
>>>> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>>>>> This patch adds support for Tegra pinctrl driver suspend and resume.
>>>>>
>>>>> During suspend, context of all pinctrl registers are stored and
>>>>> on resume they are all restored to have all the pinmux and pad
>>>>> configuration for normal operation.
>>>>>
>>>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>>>>> ---
>>>>>   int tegra_pinctrl_probe(struct platform_device *pdev,
>>>>>               const struct tegra_pinctrl_soc_data *soc_data);
>>>>>   #endif
>>>>> diff --git a/drivers/pinctrl/tegra/pinctrl-tegra210.c
>>>>> b/drivers/pinctrl/tegra/pinctrl-tegra210.c
>>>>> index 0b56ad5c9c1c..edd3f4606cdb 100644
>>>>> --- a/drivers/pinctrl/tegra/pinctrl-tegra210.c
>>>>> +++ b/drivers/pinctrl/tegra/pinctrl-tegra210.c
>>>>> @@ -1571,6 +1571,7 @@ static struct platform_driver tegra210_pinctrl_driver = {
>>>>>       .driver = {
>>>>>           .name = "tegra210-pinctrl",
>>>>>           .of_match_table = tegra210_pinctrl_of_match,
>>>>> +        .pm = &tegra_pinctrl_pm,
>>>>>       },
>>>>>       .probe = tegra210_pinctrl_probe,
>>>>>   };
>>>>>
>>>> Could you please address my comments in the next revision if there will be one?
>>>>
>>> Also, what about adding ".pm' for other Tegras? I'm sure Jon could test them for you.
>>
>> This series is for Tegra210 SC7 entry/exit along with clocks and pinctrl suspend
>> resume needed for Tegra210 basic sc7 entry and exit.
>>
>> This includes pinctrl, pmc changes, clock-tegra210 driver changes all w.r.t Tegra210
>> platforms specific.
>>
>> Suspend/resume support for other Tegras will be in separate patch series.
> 
> Okay, fair enough.
> 

It may also make some sense to split this patch into two:

 1) add generic tegra-pinctrl suspend-resume support
 2) add suspend-resume OPS to the pinctrl-tegra210

For consistency.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-06-28  2:12 ` [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support Sowjanya Komatineni
  2019-06-28 11:56   ` Dmitry Osipenko
@ 2019-06-29 15:46   ` Dmitry Osipenko
  2019-06-29 15:58     ` Dmitry Osipenko
  2019-07-13  5:48     ` Sowjanya Komatineni
  2019-07-04  7:26   ` Linus Walleij
  2 siblings, 2 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 15:46 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

28.06.2019 5:12, Sowjanya Komatineni пишет:
> This patch adds support for Tegra pinctrl driver suspend and resume.
> 
> During suspend, context of all pinctrl registers are stored and
> on resume they are all restored to have all the pinmux and pad
> configuration for normal operation.
> 
> Acked-by: Thierry Reding <treding@nvidia.com>
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> ---
>  drivers/pinctrl/tegra/pinctrl-tegra.c    | 52 ++++++++++++++++++++++++++++++++
>  drivers/pinctrl/tegra/pinctrl-tegra.h    |  3 ++
>  drivers/pinctrl/tegra/pinctrl-tegra210.c |  1 +
>  3 files changed, 56 insertions(+)
> 
> diff --git a/drivers/pinctrl/tegra/pinctrl-tegra.c b/drivers/pinctrl/tegra/pinctrl-tegra.c
> index 34596b246578..e7c0a1011cba 100644
> --- a/drivers/pinctrl/tegra/pinctrl-tegra.c
> +++ b/drivers/pinctrl/tegra/pinctrl-tegra.c
> @@ -621,6 +621,43 @@ static void tegra_pinctrl_clear_parked_bits(struct tegra_pmx *pmx)
>  	}
>  }
>  
> +static int tegra_pinctrl_suspend(struct device *dev)
> +{
> +	struct tegra_pmx *pmx = dev_get_drvdata(dev);
> +	u32 *backup_regs = pmx->backup_regs;
> +	u32 *regs;
> +	unsigned int i, j;
> +
> +	for (i = 0; i < pmx->nbanks; i++) {
> +		regs = pmx->regs[i];
> +		for (j = 0; j < pmx->reg_bank_size[i] / 4; j++)
> +			*backup_regs++ = readl(regs++);
> +	}
> +
> +	return pinctrl_force_sleep(pmx->pctl);
> +}
> +
> +static int tegra_pinctrl_resume(struct device *dev)
> +{
> +	struct tegra_pmx *pmx = dev_get_drvdata(dev);
> +	u32 *backup_regs = pmx->backup_regs;
> +	u32 *regs;
> +	unsigned int i, j;
> +
> +	for (i = 0; i < pmx->nbanks; i++) {
> +		regs = pmx->regs[i];
> +		for (j = 0; j < pmx->reg_bank_size[i] / 4; j++)
> +			writel(*backup_regs++, regs++);
> +	}
> +
> +	return 0;
> +}
> +
> +const struct dev_pm_ops tegra_pinctrl_pm = {
> +	.suspend = &tegra_pinctrl_suspend,
> +	.resume = &tegra_pinctrl_resume
> +};

Hm, so this are the generic platform-driver suspend-resume OPS here, which is very
nice! But.. shouldn't pinctrl be resumed before the CLK driver (which is syscore_ops
in this version of the series)? .. Given that "clock" function may need to be
selected for some of the pins.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-06-29 15:46   ` Dmitry Osipenko
@ 2019-06-29 15:58     ` Dmitry Osipenko
  2019-06-29 16:28       ` Dmitry Osipenko
  2019-07-04  7:31       ` Linus Walleij
  2019-07-13  5:48     ` Sowjanya Komatineni
  1 sibling, 2 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 15:58 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

29.06.2019 18:46, Dmitry Osipenko пишет:
> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>> This patch adds support for Tegra pinctrl driver suspend and resume.
>>
>> During suspend, context of all pinctrl registers are stored and
>> on resume they are all restored to have all the pinmux and pad
>> configuration for normal operation.
>>
>> Acked-by: Thierry Reding <treding@nvidia.com>
>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>> ---
>>  drivers/pinctrl/tegra/pinctrl-tegra.c    | 52 ++++++++++++++++++++++++++++++++
>>  drivers/pinctrl/tegra/pinctrl-tegra.h    |  3 ++
>>  drivers/pinctrl/tegra/pinctrl-tegra210.c |  1 +
>>  3 files changed, 56 insertions(+)
>>
>> diff --git a/drivers/pinctrl/tegra/pinctrl-tegra.c b/drivers/pinctrl/tegra/pinctrl-tegra.c
>> index 34596b246578..e7c0a1011cba 100644
>> --- a/drivers/pinctrl/tegra/pinctrl-tegra.c
>> +++ b/drivers/pinctrl/tegra/pinctrl-tegra.c
>> @@ -621,6 +621,43 @@ static void tegra_pinctrl_clear_parked_bits(struct tegra_pmx *pmx)
>>  	}
>>  }
>>  
>> +static int tegra_pinctrl_suspend(struct device *dev)
>> +{
>> +	struct tegra_pmx *pmx = dev_get_drvdata(dev);
>> +	u32 *backup_regs = pmx->backup_regs;
>> +	u32 *regs;
>> +	unsigned int i, j;
>> +
>> +	for (i = 0; i < pmx->nbanks; i++) {
>> +		regs = pmx->regs[i];
>> +		for (j = 0; j < pmx->reg_bank_size[i] / 4; j++)
>> +			*backup_regs++ = readl(regs++);
>> +	}
>> +
>> +	return pinctrl_force_sleep(pmx->pctl);
>> +}
>> +
>> +static int tegra_pinctrl_resume(struct device *dev)
>> +{
>> +	struct tegra_pmx *pmx = dev_get_drvdata(dev);
>> +	u32 *backup_regs = pmx->backup_regs;
>> +	u32 *regs;
>> +	unsigned int i, j;
>> +
>> +	for (i = 0; i < pmx->nbanks; i++) {
>> +		regs = pmx->regs[i];
>> +		for (j = 0; j < pmx->reg_bank_size[i] / 4; j++)
>> +			writel(*backup_regs++, regs++);
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +const struct dev_pm_ops tegra_pinctrl_pm = {
>> +	.suspend = &tegra_pinctrl_suspend,
>> +	.resume = &tegra_pinctrl_resume
>> +};
> 
> Hm, so this are the generic platform-driver suspend-resume OPS here, which is very
> nice! But.. shouldn't pinctrl be resumed before the CLK driver (which is syscore_ops
> in this version of the series)? .. Given that "clock" function may need to be
> selected for some of the pins.
> 

Oh, also what about GPIO-pinctrl suspend resume ordering .. is it okay that pinctrl
will be resumed after GPIO? Shouldn't a proper pin-muxing be selected at first?

This also looks to me very unsafe in a context of older Tegras which are initializing
the static muxing very early during of the boot, otherwise things won't work well for
the drivers.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-06-29 15:58     ` Dmitry Osipenko
@ 2019-06-29 16:28       ` Dmitry Osipenko
  2019-07-04  7:31       ` Linus Walleij
  1 sibling, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 16:28 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

29.06.2019 18:58, Dmitry Osipenko пишет:
> 29.06.2019 18:46, Dmitry Osipenko пишет:
>> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>>> This patch adds support for Tegra pinctrl driver suspend and resume.
>>>
>>> During suspend, context of all pinctrl registers are stored and
>>> on resume they are all restored to have all the pinmux and pad
>>> configuration for normal operation.
>>>
>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>>> ---
>>>  drivers/pinctrl/tegra/pinctrl-tegra.c    | 52 ++++++++++++++++++++++++++++++++
>>>  drivers/pinctrl/tegra/pinctrl-tegra.h    |  3 ++
>>>  drivers/pinctrl/tegra/pinctrl-tegra210.c |  1 +
>>>  3 files changed, 56 insertions(+)
>>>
>>> diff --git a/drivers/pinctrl/tegra/pinctrl-tegra.c b/drivers/pinctrl/tegra/pinctrl-tegra.c
>>> index 34596b246578..e7c0a1011cba 100644
>>> --- a/drivers/pinctrl/tegra/pinctrl-tegra.c
>>> +++ b/drivers/pinctrl/tegra/pinctrl-tegra.c
>>> @@ -621,6 +621,43 @@ static void tegra_pinctrl_clear_parked_bits(struct tegra_pmx *pmx)
>>>  	}
>>>  }
>>>  
>>> +static int tegra_pinctrl_suspend(struct device *dev)
>>> +{
>>> +	struct tegra_pmx *pmx = dev_get_drvdata(dev);
>>> +	u32 *backup_regs = pmx->backup_regs;
>>> +	u32 *regs;
>>> +	unsigned int i, j;
>>> +
>>> +	for (i = 0; i < pmx->nbanks; i++) {
>>> +		regs = pmx->regs[i];
>>> +		for (j = 0; j < pmx->reg_bank_size[i] / 4; j++)
>>> +			*backup_regs++ = readl(regs++);
>>> +	}
>>> +
>>> +	return pinctrl_force_sleep(pmx->pctl);
>>> +}
>>> +
>>> +static int tegra_pinctrl_resume(struct device *dev)
>>> +{
>>> +	struct tegra_pmx *pmx = dev_get_drvdata(dev);
>>> +	u32 *backup_regs = pmx->backup_regs;
>>> +	u32 *regs;
>>> +	unsigned int i, j;
>>> +
>>> +	for (i = 0; i < pmx->nbanks; i++) {
>>> +		regs = pmx->regs[i];
>>> +		for (j = 0; j < pmx->reg_bank_size[i] / 4; j++)
>>> +			writel(*backup_regs++, regs++);
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +const struct dev_pm_ops tegra_pinctrl_pm = {
>>> +	.suspend = &tegra_pinctrl_suspend,
>>> +	.resume = &tegra_pinctrl_resume
>>> +};
>>
>> Hm, so this are the generic platform-driver suspend-resume OPS here, which is very
>> nice! But.. shouldn't pinctrl be resumed before the CLK driver (which is syscore_ops
>> in this version of the series)? .. Given that "clock" function may need to be
>> selected for some of the pins.
>>
> 
> Oh, also what about GPIO-pinctrl suspend resume ordering .. is it okay that pinctrl
> will be resumed after GPIO? Shouldn't a proper pin-muxing be selected at first?
> 
> This also looks to me very unsafe in a context of older Tegras which are initializing
> the static muxing very early during of the boot, otherwise things won't work well for
> the drivers.
> 

Although, scratch what I wrote about older Tegras. We are probing pinctl driver very
early, hence it should suspend last and resume first. Should be okay.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 08/18] clk: tegra: Add suspend resume support for DFLL
  2019-06-29 13:28   ` Dmitry Osipenko
@ 2019-06-29 21:45     ` Dmitry Osipenko
  0 siblings, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-06-29 21:45 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

29.06.2019 16:28, Dmitry Osipenko пишет:
> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>> This patch creates APIs for supporting Tegra210 clock driver to
>> perform DFLL suspend and resume operation.
>>
>> During suspend, DFLL mode is saved and on resume Tegra210 clock driver
>> invokes DFLL resume API to re-initialize DFLL to enable target device
>> clock in open loop mode or closed loop mode.
>>
>> Acked-by: Thierry Reding <treding@nvidia.com>
>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>> ---
>>  drivers/clk/tegra/clk-dfll.c | 78 ++++++++++++++++++++++++++++++++++++++++++++
>>  drivers/clk/tegra/clk-dfll.h |  2 ++
>>  2 files changed, 80 insertions(+)
>>
>> diff --git a/drivers/clk/tegra/clk-dfll.c b/drivers/clk/tegra/clk-dfll.c
>> index f8688c2ddf1a..a1f37cf99b00 100644
>> --- a/drivers/clk/tegra/clk-dfll.c
>> +++ b/drivers/clk/tegra/clk-dfll.c
>> @@ -277,6 +277,7 @@ struct tegra_dfll {
>>  	unsigned long			dvco_rate_min;
>>  
>>  	enum dfll_ctrl_mode		mode;
>> +	enum dfll_ctrl_mode		resume_mode;
>>  	enum dfll_tune_range		tune_range;
>>  	struct dentry			*debugfs_dir;
>>  	struct clk_hw			dfll_clk_hw;
>> @@ -1864,6 +1865,83 @@ static int dfll_fetch_common_params(struct tegra_dfll *td)
>>  }
>>  
>>  /*
>> + * tegra_dfll_suspend
>> + * @pdev: DFLL instance
>> + *
>> + * dfll controls clock/voltage to other devices, including CPU. Therefore,
>> + * dfll driver pm suspend callback does not stop cl-dvfs operations.
>> + */
>> +void tegra_dfll_suspend(struct platform_device *pdev)
>> +{
>> +	struct tegra_dfll *td = dev_get_drvdata(&pdev->dev);
>> +
>> +	if (!td)
>> +		return;
>> +
>> +	if (td->mode <= DFLL_DISABLED)
>> +		return;
>> +
>> +	td->resume_mode = td->mode;
>> +	switch (td->mode) {
>> +	case DFLL_CLOSED_LOOP:
>> +		dfll_set_mode(td, DFLL_CLOSED_LOOP);
>> +		dfll_set_frequency_request(td, &td->last_req);
>> +
>> +		dfll_unlock(td);
>> +		break;
>> +	default:
>> +		break;
>> +	}
>> +}
>> +
>> +/**
>> + * tegra_dfll_resume - reprogram the DFLL after context-loss
>> + * @pdev: DFLL instance
>> + *
>> + * Re-initialize and enable target device clock in open loop mode. Called
>> + * directly from SoC clock resume syscore operation. Closed loop will be
>> + * re-entered in platform syscore ops as well after CPU clock source is
>> + * switched to DFLL in open loop.
>> + */
>> +void tegra_dfll_resume(struct platform_device *pdev, bool on_dfll)
>> +{
>> +	struct tegra_dfll *td = dev_get_drvdata(&pdev->dev);
>> +
>> +	if (!td)
>> +		return;
>> +
>> +	if (on_dfll) {
>> +		if (td->resume_mode == DFLL_CLOSED_LOOP)
>> +			dfll_lock(td);
>> +		td->resume_mode = DFLL_DISABLED;
>> +		return;
>> +	}
>> +
>> +	reset_control_deassert(td->dvco_rst);
>> +
>> +	pm_runtime_get(td->dev);
> 
> pm_runtime_get_sync()?
> 
> Otherwise looks like you're risking a lot here because pm_runtime_get() is an
> asynchronous request.

It looks like DFLL driver should be masked as IRQ-safe using pm_runtime_irq_safe()
and then the synchronous resume could be used..

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-06-28  2:12 ` [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support Sowjanya Komatineni
  2019-06-28 11:56   ` Dmitry Osipenko
  2019-06-29 15:46   ` Dmitry Osipenko
@ 2019-07-04  7:26   ` Linus Walleij
  2 siblings, 0 replies; 111+ messages in thread
From: Linus Walleij @ 2019-07-04  7:26 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: thierry.reding, Jon Hunter, Thomas Gleixner, Jason Cooper,
	Marc Zyngier, Stefan Agner, Mark Rutland, Peter De Schrijver,
	Prashant Gaikwad, Stephen Boyd, linux-clk,
	open list:GPIO SUBSYSTEM, jckuo, Joseph Lo, talho, linux-tegra,
	linux-kernel, Mikko Perttunen, spatra, Rob Herring,
	Dmitry Osipenko,
	open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS

On Fri, Jun 28, 2019 at 4:13 AM Sowjanya Komatineni
<skomatineni@nvidia.com> wrote:

> This patch adds support for Tegra pinctrl driver suspend and resume.
>
> During suspend, context of all pinctrl registers are stored and
> on resume they are all restored to have all the pinmux and pad
> configuration for normal operation.
>
> Acked-by: Thierry Reding <treding@nvidia.com>
> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>

Looks good.

Can I just apply this patch or does it need to go in with
the other (clk) changes?

Yours,
Linus Walleij

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-06-29 15:58     ` Dmitry Osipenko
  2019-06-29 16:28       ` Dmitry Osipenko
@ 2019-07-04  7:31       ` Linus Walleij
  2019-07-04 10:40         ` Dmitry Osipenko
  1 sibling, 1 reply; 111+ messages in thread
From: Linus Walleij @ 2019-07-04  7:31 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Sowjanya Komatineni, thierry.reding, Jon Hunter, Thomas Gleixner,
	Jason Cooper, Marc Zyngier, Stefan Agner, Mark Rutland,
	Peter De Schrijver, Prashant Gaikwad, Stephen Boyd, linux-clk,
	open list:GPIO SUBSYSTEM, jckuo, Joseph Lo, talho, linux-tegra,
	linux-kernel, Mikko Perttunen, spatra, Rob Herring,
	open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS

On Sat, Jun 29, 2019 at 5:58 PM Dmitry Osipenko <digetx@gmail.com> wrote:

> Oh, also what about GPIO-pinctrl suspend resume ordering .. is it okay that pinctrl
> will be resumed after GPIO? Shouldn't a proper pin-muxing be selected at first?

Thierry sent some initial patches about this I think. We need to use
device links for this to work properly so he adds support for
linking the pinctrl and GPIO devices through the ranges.

For links between pin control handles and their consumers, see also:
036f394dd77f pinctrl: Enable device link creation for pin control
c6045b4e3cad pinctrl: stmfx: enable links creations
489b64d66325 pinctrl: stm32: Add links to consumers

I am using STM32 as guinea pig for this, consider adding links also
from the Tegra pinctrl. I might simply make these pinctrl consumer
to producer links default because I think it makes a lot sense.

Yours,
Linus Walleij

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-07-04  7:31       ` Linus Walleij
@ 2019-07-04 10:40         ` Dmitry Osipenko
  2019-07-13  5:31           ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-04 10:40 UTC (permalink / raw)
  To: Linus Walleij
  Cc: Sowjanya Komatineni, thierry.reding, Jon Hunter, Thomas Gleixner,
	Jason Cooper, Marc Zyngier, Stefan Agner, Mark Rutland,
	Peter De Schrijver, Prashant Gaikwad, Stephen Boyd, linux-clk,
	open list:GPIO SUBSYSTEM, jckuo, Joseph Lo, talho, linux-tegra,
	linux-kernel, Mikko Perttunen, spatra, Rob Herring,
	open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS

04.07.2019 10:31, Linus Walleij пишет:
> On Sat, Jun 29, 2019 at 5:58 PM Dmitry Osipenko <digetx@gmail.com> wrote:
> 
>> Oh, also what about GPIO-pinctrl suspend resume ordering .. is it okay that pinctrl
>> will be resumed after GPIO? Shouldn't a proper pin-muxing be selected at first?
> 
> Thierry sent some initial patches about this I think. We need to use
> device links for this to work properly so he adds support for
> linking the pinctrl and GPIO devices through the ranges.
> 
> For links between pin control handles and their consumers, see also:
> 036f394dd77f pinctrl: Enable device link creation for pin control
> c6045b4e3cad pinctrl: stmfx: enable links creations
> 489b64d66325 pinctrl: stm32: Add links to consumers
> 
> I am using STM32 as guinea pig for this, consider adding links also
> from the Tegra pinctrl. I might simply make these pinctrl consumer
> to producer links default because I think it makes a lot sense.

IIUC, currently the plan is to resume pinctrl *after* GPIO for Tegra210 [1]. But this
contradicts to what was traditionally done for older Tegras where pinctrl was always
resumed first and apparently it won't work well for the GPIO ranges as well. I think this
and the other patchsets related to suspend-resume still need some more thought.

[1] https://patchwork.kernel.org/patch/11012077/

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-07-04 10:40         ` Dmitry Osipenko
@ 2019-07-13  5:31           ` Sowjanya Komatineni
  2019-07-14 21:41             ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-13  5:31 UTC (permalink / raw)
  To: Dmitry Osipenko, Linus Walleij
  Cc: thierry.reding, Jon Hunter, Thomas Gleixner, Jason Cooper,
	Marc Zyngier, Stefan Agner, Mark Rutland, Peter De Schrijver,
	Prashant Gaikwad, Stephen Boyd, linux-clk,
	open list:GPIO SUBSYSTEM, jckuo, Joseph Lo, talho, linux-tegra,
	linux-kernel, Mikko Perttunen, spatra, Rob Herring,
	open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS


On 7/4/19 3:40 AM, Dmitry Osipenko wrote:
> 04.07.2019 10:31, Linus Walleij пишет:
>> On Sat, Jun 29, 2019 at 5:58 PM Dmitry Osipenko <digetx@gmail.com> wrote:
>>
>>> Oh, also what about GPIO-pinctrl suspend resume ordering .. is it okay that pinctrl
>>> will be resumed after GPIO? Shouldn't a proper pin-muxing be selected at first?
>> Thierry sent some initial patches about this I think. We need to use
>> device links for this to work properly so he adds support for
>> linking the pinctrl and GPIO devices through the ranges.
>>
>> For links between pin control handles and their consumers, see also:
>> 036f394dd77f pinctrl: Enable device link creation for pin control
>> c6045b4e3cad pinctrl: stmfx: enable links creations
>> 489b64d66325 pinctrl: stm32: Add links to consumers
>>
>> I am using STM32 as guinea pig for this, consider adding links also
>> from the Tegra pinctrl. I might simply make these pinctrl consumer
>> to producer links default because I think it makes a lot sense.
> IIUC, currently the plan is to resume pinctrl *after* GPIO for Tegra210 [1]. But this
> contradicts to what was traditionally done for older Tegras where pinctrl was always
> resumed first and apparently it won't work well for the GPIO ranges as well. I think this
> and the other patchsets related to suspend-resume still need some more thought.
>
> [1] https://patchwork.kernel.org/patch/11012077/

Park bit was introduced from Tegra210 onwards and during suspend/resume, 
requirement of gpio restore prior to pinctrl restore is not required for 
prior Tegra210.

Also currently pinctrl suspend/resume implementation for prior Tegra210 
is not yet upstreamed but having gpio restore prior to pinmux during 
suspend/resume should not cause any issue for prior tegra's as well as 
gpio resume restores pins back to same gpio config as they were during 
suspend entry.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-06-29 15:46   ` Dmitry Osipenko
  2019-06-29 15:58     ` Dmitry Osipenko
@ 2019-07-13  5:48     ` Sowjanya Komatineni
  1 sibling, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-13  5:48 UTC (permalink / raw)
  To: Dmitry Osipenko, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree


On 6/29/19 8:46 AM, Dmitry Osipenko wrote:
> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>> This patch adds support for Tegra pinctrl driver suspend and resume.
>>
>> During suspend, context of all pinctrl registers are stored and
>> on resume they are all restored to have all the pinmux and pad
>> configuration for normal operation.
>>
>> Acked-by: Thierry Reding <treding@nvidia.com>
>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>> ---
>>   drivers/pinctrl/tegra/pinctrl-tegra.c    | 52 ++++++++++++++++++++++++++++++++
>>   drivers/pinctrl/tegra/pinctrl-tegra.h    |  3 ++
>>   drivers/pinctrl/tegra/pinctrl-tegra210.c |  1 +
>>   3 files changed, 56 insertions(+)
>>
>> diff --git a/drivers/pinctrl/tegra/pinctrl-tegra.c b/drivers/pinctrl/tegra/pinctrl-tegra.c
>> index 34596b246578..e7c0a1011cba 100644
>> --- a/drivers/pinctrl/tegra/pinctrl-tegra.c
>> +++ b/drivers/pinctrl/tegra/pinctrl-tegra.c
>> @@ -621,6 +621,43 @@ static void tegra_pinctrl_clear_parked_bits(struct tegra_pmx *pmx)
>>   	}
>>   }
>>   
>> +static int tegra_pinctrl_suspend(struct device *dev)
>> +{
>> +	struct tegra_pmx *pmx = dev_get_drvdata(dev);
>> +	u32 *backup_regs = pmx->backup_regs;
>> +	u32 *regs;
>> +	unsigned int i, j;
>> +
>> +	for (i = 0; i < pmx->nbanks; i++) {
>> +		regs = pmx->regs[i];
>> +		for (j = 0; j < pmx->reg_bank_size[i] / 4; j++)
>> +			*backup_regs++ = readl(regs++);
>> +	}
>> +
>> +	return pinctrl_force_sleep(pmx->pctl);
>> +}
>> +
>> +static int tegra_pinctrl_resume(struct device *dev)
>> +{
>> +	struct tegra_pmx *pmx = dev_get_drvdata(dev);
>> +	u32 *backup_regs = pmx->backup_regs;
>> +	u32 *regs;
>> +	unsigned int i, j;
>> +
>> +	for (i = 0; i < pmx->nbanks; i++) {
>> +		regs = pmx->regs[i];
>> +		for (j = 0; j < pmx->reg_bank_size[i] / 4; j++)
>> +			writel(*backup_regs++, regs++);
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +const struct dev_pm_ops tegra_pinctrl_pm = {
>> +	.suspend = &tegra_pinctrl_suspend,
>> +	.resume = &tegra_pinctrl_resume
>> +};
> Hm, so this are the generic platform-driver suspend-resume OPS here, which is very
> nice! But.. shouldn't pinctrl be resumed before the CLK driver (which is syscore_ops
> in this version of the series)? .. Given that "clock" function may need to be
> selected for some of the pins.

selection of clock functions on some Tegra pins through corresponding 
pinmux (like extperiph clks) can happen after clock driver resume as 
well where clock source is restored to state during suspend before 
selecting clock function on that pin.



^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-06-29 15:10   ` Dmitry Osipenko
@ 2019-07-13  5:54     ` Sowjanya Komatineni
  2019-07-14 21:41       ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-13  5:54 UTC (permalink / raw)
  To: Dmitry Osipenko, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree


On 6/29/19 8:10 AM, Dmitry Osipenko wrote:
> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>> This patch adds system suspend and resume support for Tegra210
>> clocks.
>>
>> All the CAR controller settings are lost on suspend when core power
>> goes off.
>>
>> This patch has implementation for saving and restoring all the PLLs
>> and clocks context during system suspend and resume to have the
>> clocks back to same state for normal operation.
>>
>> Acked-by: Thierry Reding <treding@nvidia.com>
>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>> ---
>>   drivers/clk/tegra/clk-tegra210.c | 115 ++++++++++++++++++++++++++++++++++++++-
>>   drivers/clk/tegra/clk.c          |  14 +++++
>>   drivers/clk/tegra/clk.h          |   1 +
>>   3 files changed, 127 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/clk/tegra/clk-tegra210.c b/drivers/clk/tegra/clk-tegra210.c
>> index 1c08c53482a5..1b839544e086 100644
>> --- a/drivers/clk/tegra/clk-tegra210.c
>> +++ b/drivers/clk/tegra/clk-tegra210.c
>> @@ -9,10 +9,12 @@
>>   #include <linux/clkdev.h>
>>   #include <linux/of.h>
>>   #include <linux/of_address.h>
>> +#include <linux/of_platform.h>
>>   #include <linux/delay.h>
>>   #include <linux/export.h>
>>   #include <linux/mutex.h>
>>   #include <linux/clk/tegra.h>
>> +#include <linux/syscore_ops.h>
>>   #include <dt-bindings/clock/tegra210-car.h>
>>   #include <dt-bindings/reset/tegra210-car.h>
>>   #include <linux/iopoll.h>
>> @@ -20,6 +22,7 @@
>>   #include <soc/tegra/pmc.h>
>>   
>>   #include "clk.h"
>> +#include "clk-dfll.h"
>>   #include "clk-id.h"
>>   
>>   /*
>> @@ -225,6 +228,7 @@
>>   
>>   #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
>>   #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
>> +#define CPU_SOFTRST_CTRL 0x380
>>   
>>   #define LVL2_CLK_GATE_OVRA 0xf8
>>   #define LVL2_CLK_GATE_OVRC 0x3a0
>> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
>>   	struct tegra_clk_pll_freq_table *fentry;
>>   	struct tegra_clk_pll pllu;
>>   	u32 reg;
>> +	int ret;
>>   
>>   	for (fentry = pll_u_freq_table; fentry->input_rate; fentry++) {
>>   		if (fentry->input_rate == pll_ref_freq)
>> @@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
>>   	fence_udelay(1, clk_base);
>>   	reg |= PLL_ENABLE;
>>   	writel(reg, clk_base + PLLU_BASE);
>> +	fence_udelay(1, clk_base);
>>   
>> -	readl_relaxed_poll_timeout_atomic(clk_base + PLLU_BASE, reg,
>> -					  reg & PLL_BASE_LOCK, 2, 1000);
>> -	if (!(reg & PLL_BASE_LOCK)) {
>> +	ret = tegra210_wait_for_mask(&pllu, PLLU_BASE, PLL_BASE_LOCK);
>> +	if (ret) {
>>   		pr_err("Timed out waiting for PLL_U to lock\n");
>>   		return -ETIMEDOUT;
>>   	}
>> @@ -3283,6 +3288,103 @@ static void tegra210_disable_cpu_clock(u32 cpu)
>>   }
>>   
>>   #ifdef CONFIG_PM_SLEEP
>> +static u32 cpu_softrst_ctx[3];
>> +static struct platform_device *dfll_pdev;
>> +#define car_readl(_base, _off) readl_relaxed(clk_base + (_base) + ((_off) * 4))
>> +#define car_writel(_val, _base, _off) \
>> +		writel_relaxed(_val, clk_base + (_base) + ((_off) * 4))
>> +
>> +static int tegra210_clk_suspend(void)
>> +{
>> +	unsigned int i;
>> +	struct device_node *node;
>> +
>> +	tegra_cclkg_burst_policy_save_context();
>> +
>> +	if (!dfll_pdev) {
>> +		node = of_find_compatible_node(NULL, NULL,
>> +					       "nvidia,tegra210-dfll");
>> +		if (node)
>> +			dfll_pdev = of_find_device_by_node(node);
>> +
>> +		of_node_put(node);
>> +		if (!dfll_pdev)
>> +			pr_err("dfll node not found. no suspend for dfll\n");
>> +	}
>> +
>> +	if (dfll_pdev)
>> +		tegra_dfll_suspend(dfll_pdev);
>> +
>> +	/* Enable PLLP_OUT_CPU after dfll suspend */
>> +	tegra_clk_set_pllp_out_cpu(true);
>> +
>> +	tegra_sclk_cclklp_burst_policy_save_context();
>> +
>> +	clk_save_context();
>> +
>> +	for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>> +		cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
>> +
>> +	return 0;
>> +}
>> +
>> +static void tegra210_clk_resume(void)
>> +{
>> +	unsigned int i;
>> +	struct clk_hw *parent;
>> +	struct clk *clk;
>> +
>> +	/*
>> +	 * clk_restore_context restores clocks as per the clock tree.
>> +	 *
>> +	 * dfllCPU_out is first in the clock tree to get restored and it
>> +	 * involves programming DFLL controller along with restoring CPUG
>> +	 * clock burst policy.
>> +	 *
>> +	 * DFLL programming needs dfll_ref and dfll_soc peripheral clocks
>> +	 * to be restores which are part ofthe peripheral clocks.
>> +	 * So, peripheral clocks restore should happen prior to dfll clock
>> +	 * restore.
>> +	 */
>> +
>> +	tegra_clk_osc_resume(clk_base);
>> +	for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>> +		car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
>> +
>> +	/* restore all plls and peripheral clocks */
>> +	tegra210_init_pllu();
>> +	clk_restore_context();
>> +
>> +	fence_udelay(5, clk_base);
>> +
>> +	/* resume SCLK and CPULP clocks */
>> +	tegra_sclk_cpulp_burst_policy_restore_context();
>> +
>> +	/*
>> +	 * restore CPUG clocks:
>> +	 * - enable DFLL in open loop mode
>> +	 * - switch CPUG to DFLL clock source
>> +	 * - close DFLL loop
>> +	 * - sync PLLX state
>> +	 */
>> +	if (dfll_pdev)
>> +		tegra_dfll_resume(dfll_pdev, false);
>> +
>> +	tegra_cclkg_burst_policy_restore_context();
>> +	fence_udelay(2, clk_base);
>> +
>> +	if (dfll_pdev)
>> +		tegra_dfll_resume(dfll_pdev, true);
>> +
>> +	parent = clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
>> +	clk = clks[TEGRA210_CLK_PLL_X];
>> +	if (parent != __clk_get_hw(clk))
>> +		tegra_clk_sync_state_pll(__clk_get_hw(clk));
>> +
>> +	/* Disable PLL_OUT_CPU after DFLL resume */
>> +	tegra_clk_set_pllp_out_cpu(false);
>> +}
>> +
>>   static void tegra210_cpu_clock_suspend(void)
>>   {
>>   	/* switch coresite to clk_m, save off original source */
>> @@ -3298,6 +3400,11 @@ static void tegra210_cpu_clock_resume(void)
>>   }
>>   #endif
>>   
>> +static struct syscore_ops tegra_clk_syscore_ops = {
>> +	.suspend = tegra210_clk_suspend,
>> +	.resume = tegra210_clk_resume,
>> +};
>> +
>>   static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
>>   	.wait_for_reset	= tegra210_wait_cpu_in_reset,
>>   	.disable_clock	= tegra210_disable_cpu_clock,
>> @@ -3583,5 +3690,7 @@ static void __init tegra210_clock_init(struct device_node *np)
>>   	tegra210_mbist_clk_init();
>>   
>>   	tegra_cpu_car_ops = &tegra210_cpu_car_ops;
>> +
>> +	register_syscore_ops(&tegra_clk_syscore_ops);
>>   }
> Is it really worthwhile to use syscore_ops for suspend/resume given that drivers for
> won't resume before the CLK driver anyway? Are there any other options for CLK
> suspend/resume?
>
> I'm also not sure whether PM runtime API could be used at all in the context of
> syscore_ops ..
>
> Secondly, what about to use generic clk_save_context() / clk_restore_context()
> helpers for the suspend-resume? It looks to me that some other essential (and proper)
> platform driver (soc/tegra/? PMC?) should suspend-resume the clocks using the generic
> CLK Framework API.

Clock resume should happen very early to restore peripheral and cpu 
clocks very early than peripheral drivers resume happens.

this patch series uses clk_save_context and clk_restore_context for 
corresponding divider, pll, pllout.. save and restore context.

But as there is dependency on dfll resume and cpu and pllx clocks 
restore, couldnt use clk_save_context and clk_restore_context for dfll.

So implemented recommended dfll resume sequence in main Tegra210 clock 
driver along with invoking clk_save_context/clk_restore_context where 
all other clocks save/restore happens as per clock tree traversal.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-13  5:54     ` Sowjanya Komatineni
@ 2019-07-14 21:41       ` Dmitry Osipenko
  2019-07-16  0:35         ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-14 21:41 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

13.07.2019 8:54, Sowjanya Komatineni пишет:
> 
> On 6/29/19 8:10 AM, Dmitry Osipenko wrote:
>> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>>> This patch adds system suspend and resume support for Tegra210
>>> clocks.
>>>
>>> All the CAR controller settings are lost on suspend when core power
>>> goes off.
>>>
>>> This patch has implementation for saving and restoring all the PLLs
>>> and clocks context during system suspend and resume to have the
>>> clocks back to same state for normal operation.
>>>
>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>>> ---
>>>   drivers/clk/tegra/clk-tegra210.c | 115
>>> ++++++++++++++++++++++++++++++++++++++-
>>>   drivers/clk/tegra/clk.c          |  14 +++++
>>>   drivers/clk/tegra/clk.h          |   1 +
>>>   3 files changed, 127 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/drivers/clk/tegra/clk-tegra210.c
>>> b/drivers/clk/tegra/clk-tegra210.c
>>> index 1c08c53482a5..1b839544e086 100644
>>> --- a/drivers/clk/tegra/clk-tegra210.c
>>> +++ b/drivers/clk/tegra/clk-tegra210.c
>>> @@ -9,10 +9,12 @@
>>>   #include <linux/clkdev.h>
>>>   #include <linux/of.h>
>>>   #include <linux/of_address.h>
>>> +#include <linux/of_platform.h>
>>>   #include <linux/delay.h>
>>>   #include <linux/export.h>
>>>   #include <linux/mutex.h>
>>>   #include <linux/clk/tegra.h>
>>> +#include <linux/syscore_ops.h>
>>>   #include <dt-bindings/clock/tegra210-car.h>
>>>   #include <dt-bindings/reset/tegra210-car.h>
>>>   #include <linux/iopoll.h>
>>> @@ -20,6 +22,7 @@
>>>   #include <soc/tegra/pmc.h>
>>>     #include "clk.h"
>>> +#include "clk-dfll.h"
>>>   #include "clk-id.h"
>>>     /*
>>> @@ -225,6 +228,7 @@
>>>     #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
>>>   #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
>>> +#define CPU_SOFTRST_CTRL 0x380
>>>     #define LVL2_CLK_GATE_OVRA 0xf8
>>>   #define LVL2_CLK_GATE_OVRC 0x3a0
>>> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
>>>       struct tegra_clk_pll_freq_table *fentry;
>>>       struct tegra_clk_pll pllu;
>>>       u32 reg;
>>> +    int ret;
>>>         for (fentry = pll_u_freq_table; fentry->input_rate; fentry++) {
>>>           if (fentry->input_rate == pll_ref_freq)
>>> @@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
>>>       fence_udelay(1, clk_base);
>>>       reg |= PLL_ENABLE;
>>>       writel(reg, clk_base + PLLU_BASE);
>>> +    fence_udelay(1, clk_base);
>>>   -    readl_relaxed_poll_timeout_atomic(clk_base + PLLU_BASE, reg,
>>> -                      reg & PLL_BASE_LOCK, 2, 1000);
>>> -    if (!(reg & PLL_BASE_LOCK)) {
>>> +    ret = tegra210_wait_for_mask(&pllu, PLLU_BASE, PLL_BASE_LOCK);
>>> +    if (ret) {
>>>           pr_err("Timed out waiting for PLL_U to lock\n");
>>>           return -ETIMEDOUT;
>>>       }
>>> @@ -3283,6 +3288,103 @@ static void tegra210_disable_cpu_clock(u32 cpu)
>>>   }
>>>     #ifdef CONFIG_PM_SLEEP
>>> +static u32 cpu_softrst_ctx[3];
>>> +static struct platform_device *dfll_pdev;
>>> +#define car_readl(_base, _off) readl_relaxed(clk_base + (_base) +
>>> ((_off) * 4))
>>> +#define car_writel(_val, _base, _off) \
>>> +        writel_relaxed(_val, clk_base + (_base) + ((_off) * 4))
>>> +
>>> +static int tegra210_clk_suspend(void)
>>> +{
>>> +    unsigned int i;
>>> +    struct device_node *node;
>>> +
>>> +    tegra_cclkg_burst_policy_save_context();
>>> +
>>> +    if (!dfll_pdev) {
>>> +        node = of_find_compatible_node(NULL, NULL,
>>> +                           "nvidia,tegra210-dfll");
>>> +        if (node)
>>> +            dfll_pdev = of_find_device_by_node(node);
>>> +
>>> +        of_node_put(node);
>>> +        if (!dfll_pdev)
>>> +            pr_err("dfll node not found. no suspend for dfll\n");
>>> +    }
>>> +
>>> +    if (dfll_pdev)
>>> +        tegra_dfll_suspend(dfll_pdev);
>>> +
>>> +    /* Enable PLLP_OUT_CPU after dfll suspend */
>>> +    tegra_clk_set_pllp_out_cpu(true);
>>> +
>>> +    tegra_sclk_cclklp_burst_policy_save_context();
>>> +
>>> +    clk_save_context();
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>> +        cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static void tegra210_clk_resume(void)
>>> +{
>>> +    unsigned int i;
>>> +    struct clk_hw *parent;
>>> +    struct clk *clk;
>>> +
>>> +    /*
>>> +     * clk_restore_context restores clocks as per the clock tree.
>>> +     *
>>> +     * dfllCPU_out is first in the clock tree to get restored and it
>>> +     * involves programming DFLL controller along with restoring CPUG
>>> +     * clock burst policy.
>>> +     *
>>> +     * DFLL programming needs dfll_ref and dfll_soc peripheral clocks
>>> +     * to be restores which are part ofthe peripheral clocks.

                                            ^ white-space

Please use spellchecker to avoid typos.

>>> +     * So, peripheral clocks restore should happen prior to dfll clock
>>> +     * restore.
>>> +     */
>>> +
>>> +    tegra_clk_osc_resume(clk_base);
>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>> +        car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
>>> +
>>> +    /* restore all plls and peripheral clocks */
>>> +    tegra210_init_pllu();
>>> +    clk_restore_context();
>>> +
>>> +    fence_udelay(5, clk_base);
>>> +
>>> +    /* resume SCLK and CPULP clocks */
>>> +    tegra_sclk_cpulp_burst_policy_restore_context();
>>> +
>>> +    /*
>>> +     * restore CPUG clocks:
>>> +     * - enable DFLL in open loop mode
>>> +     * - switch CPUG to DFLL clock source
>>> +     * - close DFLL loop
>>> +     * - sync PLLX state
>>> +     */
>>> +    if (dfll_pdev)
>>> +        tegra_dfll_resume(dfll_pdev, false);
>>> +
>>> +    tegra_cclkg_burst_policy_restore_context();
>>> +    fence_udelay(2, clk_base);
>>> +
>>> +    if (dfll_pdev)
>>> +        tegra_dfll_resume(dfll_pdev, true);
>>> +
>>> +    parent =
>>> clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
>>> +    clk = clks[TEGRA210_CLK_PLL_X];
>>> +    if (parent != __clk_get_hw(clk))
>>> +        tegra_clk_sync_state_pll(__clk_get_hw(clk));
>>> +
>>> +    /* Disable PLL_OUT_CPU after DFLL resume */
>>> +    tegra_clk_set_pllp_out_cpu(false);
>>> +}
>>> +
>>>   static void tegra210_cpu_clock_suspend(void)
>>>   {
>>>       /* switch coresite to clk_m, save off original source */
>>> @@ -3298,6 +3400,11 @@ static void tegra210_cpu_clock_resume(void)
>>>   }
>>>   #endif
>>>   +static struct syscore_ops tegra_clk_syscore_ops = {
>>> +    .suspend = tegra210_clk_suspend,
>>> +    .resume = tegra210_clk_resume,
>>> +};
>>> +
>>>   static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
>>>       .wait_for_reset    = tegra210_wait_cpu_in_reset,
>>>       .disable_clock    = tegra210_disable_cpu_clock,
>>> @@ -3583,5 +3690,7 @@ static void __init tegra210_clock_init(struct
>>> device_node *np)
>>>       tegra210_mbist_clk_init();
>>>         tegra_cpu_car_ops = &tegra210_cpu_car_ops;
>>> +
>>> +    register_syscore_ops(&tegra_clk_syscore_ops);
>>>   }
>> Is it really worthwhile to use syscore_ops for suspend/resume given
>> that drivers for
>> won't resume before the CLK driver anyway? Are there any other options
>> for CLK
>> suspend/resume?
>>
>> I'm also not sure whether PM runtime API could be used at all in the
>> context of
>> syscore_ops ..
>>
>> Secondly, what about to use generic clk_save_context() /
>> clk_restore_context()
>> helpers for the suspend-resume? It looks to me that some other
>> essential (and proper)
>> platform driver (soc/tegra/? PMC?) should suspend-resume the clocks
>> using the generic
>> CLK Framework API.
> 
> Clock resume should happen very early to restore peripheral and cpu
> clocks very early than peripheral drivers resume happens.

If all peripheral drivers properly requested all of the necessary clocks
and CLK driver was a platform driver, then I guess the probe should have
been naturally ordered. But that's not very achievable with the
currently available infrastructure in the kernel, so I'm not arguing
that the clocks should be explicitly resumed before the users.

> this patch series uses clk_save_context and clk_restore_context for
> corresponding divider, pll, pllout.. save and restore context.

Now I see that indeed this API is utilized in this patch, thank you for
the clarification.

> But as there is dependency on dfll resume and cpu and pllx clocks
> restore, couldnt use clk_save_context and clk_restore_context for dfll.
> 
> So implemented recommended dfll resume sequence in main Tegra210 clock
> driver along with invoking clk_save_context/clk_restore_context where
> all other clocks save/restore happens as per clock tree traversal.

Could you please clarify what part of peripherals clocks is required for
DFLL's restore? Couldn't DFLL driver be changed to avoid that quirkness
and thus to make DFLL driver suspend/resume the clock?

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support
  2019-07-13  5:31           ` Sowjanya Komatineni
@ 2019-07-14 21:41             ` Dmitry Osipenko
  0 siblings, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-14 21:41 UTC (permalink / raw)
  To: Sowjanya Komatineni, Linus Walleij
  Cc: thierry.reding, Jon Hunter, Thomas Gleixner, Jason Cooper,
	Marc Zyngier, Stefan Agner, Mark Rutland, Peter De Schrijver,
	Prashant Gaikwad, Stephen Boyd, linux-clk,
	open list:GPIO SUBSYSTEM, jckuo, Joseph Lo, talho, linux-tegra,
	linux-kernel, Mikko Perttunen, spatra, Rob Herring,
	open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS

13.07.2019 8:31, Sowjanya Komatineni пишет:
> 
> On 7/4/19 3:40 AM, Dmitry Osipenko wrote:
>> 04.07.2019 10:31, Linus Walleij пишет:
>>> On Sat, Jun 29, 2019 at 5:58 PM Dmitry Osipenko <digetx@gmail.com>
>>> wrote:
>>>
>>>> Oh, also what about GPIO-pinctrl suspend resume ordering .. is it
>>>> okay that pinctrl
>>>> will be resumed after GPIO? Shouldn't a proper pin-muxing be
>>>> selected at first?
>>> Thierry sent some initial patches about this I think. We need to use
>>> device links for this to work properly so he adds support for
>>> linking the pinctrl and GPIO devices through the ranges.
>>>
>>> For links between pin control handles and their consumers, see also:
>>> 036f394dd77f pinctrl: Enable device link creation for pin control
>>> c6045b4e3cad pinctrl: stmfx: enable links creations
>>> 489b64d66325 pinctrl: stm32: Add links to consumers
>>>
>>> I am using STM32 as guinea pig for this, consider adding links also
>>> from the Tegra pinctrl. I might simply make these pinctrl consumer
>>> to producer links default because I think it makes a lot sense.
>> IIUC, currently the plan is to resume pinctrl *after* GPIO for
>> Tegra210 [1]. But this
>> contradicts to what was traditionally done for older Tegras where
>> pinctrl was always
>> resumed first and apparently it won't work well for the GPIO ranges as
>> well. I think this
>> and the other patchsets related to suspend-resume still need some more
>> thought.
>>
>> [1] https://patchwork.kernel.org/patch/11012077/
> 
> Park bit was introduced from Tegra210 onwards and during suspend/resume,
> requirement of gpio restore prior to pinctrl restore is not required for
> prior Tegra210.
> 
> Also currently pinctrl suspend/resume implementation for prior Tegra210
> is not yet upstreamed but having gpio restore prior to pinmux during
> suspend/resume should not cause any issue for prior tegra's as well as
> gpio resume restores pins back to same gpio config as they were during
> suspend entry.
> 

Okay!

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-14 21:41       ` Dmitry Osipenko
@ 2019-07-16  0:35         ` Sowjanya Komatineni
  2019-07-16  3:00           ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-16  0:35 UTC (permalink / raw)
  To: Dmitry Osipenko, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree


On 7/14/19 2:41 PM, Dmitry Osipenko wrote:
> 13.07.2019 8:54, Sowjanya Komatineni пишет:
>> On 6/29/19 8:10 AM, Dmitry Osipenko wrote:
>>> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>>>> This patch adds system suspend and resume support for Tegra210
>>>> clocks.
>>>>
>>>> All the CAR controller settings are lost on suspend when core power
>>>> goes off.
>>>>
>>>> This patch has implementation for saving and restoring all the PLLs
>>>> and clocks context during system suspend and resume to have the
>>>> clocks back to same state for normal operation.
>>>>
>>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>>>> ---
>>>>    drivers/clk/tegra/clk-tegra210.c | 115
>>>> ++++++++++++++++++++++++++++++++++++++-
>>>>    drivers/clk/tegra/clk.c          |  14 +++++
>>>>    drivers/clk/tegra/clk.h          |   1 +
>>>>    3 files changed, 127 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/drivers/clk/tegra/clk-tegra210.c
>>>> b/drivers/clk/tegra/clk-tegra210.c
>>>> index 1c08c53482a5..1b839544e086 100644
>>>> --- a/drivers/clk/tegra/clk-tegra210.c
>>>> +++ b/drivers/clk/tegra/clk-tegra210.c
>>>> @@ -9,10 +9,12 @@
>>>>    #include <linux/clkdev.h>
>>>>    #include <linux/of.h>
>>>>    #include <linux/of_address.h>
>>>> +#include <linux/of_platform.h>
>>>>    #include <linux/delay.h>
>>>>    #include <linux/export.h>
>>>>    #include <linux/mutex.h>
>>>>    #include <linux/clk/tegra.h>
>>>> +#include <linux/syscore_ops.h>
>>>>    #include <dt-bindings/clock/tegra210-car.h>
>>>>    #include <dt-bindings/reset/tegra210-car.h>
>>>>    #include <linux/iopoll.h>
>>>> @@ -20,6 +22,7 @@
>>>>    #include <soc/tegra/pmc.h>
>>>>      #include "clk.h"
>>>> +#include "clk-dfll.h"
>>>>    #include "clk-id.h"
>>>>      /*
>>>> @@ -225,6 +228,7 @@
>>>>      #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
>>>>    #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
>>>> +#define CPU_SOFTRST_CTRL 0x380
>>>>      #define LVL2_CLK_GATE_OVRA 0xf8
>>>>    #define LVL2_CLK_GATE_OVRC 0x3a0
>>>> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
>>>>        struct tegra_clk_pll_freq_table *fentry;
>>>>        struct tegra_clk_pll pllu;
>>>>        u32 reg;
>>>> +    int ret;
>>>>          for (fentry = pll_u_freq_table; fentry->input_rate; fentry++) {
>>>>            if (fentry->input_rate == pll_ref_freq)
>>>> @@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
>>>>        fence_udelay(1, clk_base);
>>>>        reg |= PLL_ENABLE;
>>>>        writel(reg, clk_base + PLLU_BASE);
>>>> +    fence_udelay(1, clk_base);
>>>>    -    readl_relaxed_poll_timeout_atomic(clk_base + PLLU_BASE, reg,
>>>> -                      reg & PLL_BASE_LOCK, 2, 1000);
>>>> -    if (!(reg & PLL_BASE_LOCK)) {
>>>> +    ret = tegra210_wait_for_mask(&pllu, PLLU_BASE, PLL_BASE_LOCK);
>>>> +    if (ret) {
>>>>            pr_err("Timed out waiting for PLL_U to lock\n");
>>>>            return -ETIMEDOUT;
>>>>        }
>>>> @@ -3283,6 +3288,103 @@ static void tegra210_disable_cpu_clock(u32 cpu)
>>>>    }
>>>>      #ifdef CONFIG_PM_SLEEP
>>>> +static u32 cpu_softrst_ctx[3];
>>>> +static struct platform_device *dfll_pdev;
>>>> +#define car_readl(_base, _off) readl_relaxed(clk_base + (_base) +
>>>> ((_off) * 4))
>>>> +#define car_writel(_val, _base, _off) \
>>>> +        writel_relaxed(_val, clk_base + (_base) + ((_off) * 4))
>>>> +
>>>> +static int tegra210_clk_suspend(void)
>>>> +{
>>>> +    unsigned int i;
>>>> +    struct device_node *node;
>>>> +
>>>> +    tegra_cclkg_burst_policy_save_context();
>>>> +
>>>> +    if (!dfll_pdev) {
>>>> +        node = of_find_compatible_node(NULL, NULL,
>>>> +                           "nvidia,tegra210-dfll");
>>>> +        if (node)
>>>> +            dfll_pdev = of_find_device_by_node(node);
>>>> +
>>>> +        of_node_put(node);
>>>> +        if (!dfll_pdev)
>>>> +            pr_err("dfll node not found. no suspend for dfll\n");
>>>> +    }
>>>> +
>>>> +    if (dfll_pdev)
>>>> +        tegra_dfll_suspend(dfll_pdev);
>>>> +
>>>> +    /* Enable PLLP_OUT_CPU after dfll suspend */
>>>> +    tegra_clk_set_pllp_out_cpu(true);
>>>> +
>>>> +    tegra_sclk_cclklp_burst_policy_save_context();
>>>> +
>>>> +    clk_save_context();
>>>> +
>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>> +        cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static void tegra210_clk_resume(void)
>>>> +{
>>>> +    unsigned int i;
>>>> +    struct clk_hw *parent;
>>>> +    struct clk *clk;
>>>> +
>>>> +    /*
>>>> +     * clk_restore_context restores clocks as per the clock tree.
>>>> +     *
>>>> +     * dfllCPU_out is first in the clock tree to get restored and it
>>>> +     * involves programming DFLL controller along with restoring CPUG
>>>> +     * clock burst policy.
>>>> +     *
>>>> +     * DFLL programming needs dfll_ref and dfll_soc peripheral clocks
>>>> +     * to be restores which are part ofthe peripheral clocks.
>                                              ^ white-space
>
> Please use spellchecker to avoid typos.
>
>>>> +     * So, peripheral clocks restore should happen prior to dfll clock
>>>> +     * restore.
>>>> +     */
>>>> +
>>>> +    tegra_clk_osc_resume(clk_base);
>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>> +        car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
>>>> +
>>>> +    /* restore all plls and peripheral clocks */
>>>> +    tegra210_init_pllu();
>>>> +    clk_restore_context();
>>>> +
>>>> +    fence_udelay(5, clk_base);
>>>> +
>>>> +    /* resume SCLK and CPULP clocks */
>>>> +    tegra_sclk_cpulp_burst_policy_restore_context();
>>>> +
>>>> +    /*
>>>> +     * restore CPUG clocks:
>>>> +     * - enable DFLL in open loop mode
>>>> +     * - switch CPUG to DFLL clock source
>>>> +     * - close DFLL loop
>>>> +     * - sync PLLX state
>>>> +     */
>>>> +    if (dfll_pdev)
>>>> +        tegra_dfll_resume(dfll_pdev, false);
>>>> +
>>>> +    tegra_cclkg_burst_policy_restore_context();
>>>> +    fence_udelay(2, clk_base);
>>>> +
>>>> +    if (dfll_pdev)
>>>> +        tegra_dfll_resume(dfll_pdev, true);
>>>> +
>>>> +    parent =
>>>> clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
>>>> +    clk = clks[TEGRA210_CLK_PLL_X];
>>>> +    if (parent != __clk_get_hw(clk))
>>>> +        tegra_clk_sync_state_pll(__clk_get_hw(clk));
>>>> +
>>>> +    /* Disable PLL_OUT_CPU after DFLL resume */
>>>> +    tegra_clk_set_pllp_out_cpu(false);
>>>> +}
>>>> +
>>>>    static void tegra210_cpu_clock_suspend(void)
>>>>    {
>>>>        /* switch coresite to clk_m, save off original source */
>>>> @@ -3298,6 +3400,11 @@ static void tegra210_cpu_clock_resume(void)
>>>>    }
>>>>    #endif
>>>>    +static struct syscore_ops tegra_clk_syscore_ops = {
>>>> +    .suspend = tegra210_clk_suspend,
>>>> +    .resume = tegra210_clk_resume,
>>>> +};
>>>> +
>>>>    static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
>>>>        .wait_for_reset    = tegra210_wait_cpu_in_reset,
>>>>        .disable_clock    = tegra210_disable_cpu_clock,
>>>> @@ -3583,5 +3690,7 @@ static void __init tegra210_clock_init(struct
>>>> device_node *np)
>>>>        tegra210_mbist_clk_init();
>>>>          tegra_cpu_car_ops = &tegra210_cpu_car_ops;
>>>> +
>>>> +    register_syscore_ops(&tegra_clk_syscore_ops);
>>>>    }
>>> Is it really worthwhile to use syscore_ops for suspend/resume given
>>> that drivers for
>>> won't resume before the CLK driver anyway? Are there any other options
>>> for CLK
>>> suspend/resume?
>>>
>>> I'm also not sure whether PM runtime API could be used at all in the
>>> context of
>>> syscore_ops ..
>>>
>>> Secondly, what about to use generic clk_save_context() /
>>> clk_restore_context()
>>> helpers for the suspend-resume? It looks to me that some other
>>> essential (and proper)
>>> platform driver (soc/tegra/? PMC?) should suspend-resume the clocks
>>> using the generic
>>> CLK Framework API.
>> Clock resume should happen very early to restore peripheral and cpu
>> clocks very early than peripheral drivers resume happens.
> If all peripheral drivers properly requested all of the necessary clocks
> and CLK driver was a platform driver, then I guess the probe should have
> been naturally ordered. But that's not very achievable with the
> currently available infrastructure in the kernel, so I'm not arguing
> that the clocks should be explicitly resumed before the users.
>
>> this patch series uses clk_save_context and clk_restore_context for
>> corresponding divider, pll, pllout.. save and restore context.
> Now I see that indeed this API is utilized in this patch, thank you for
> the clarification.
>
>> But as there is dependency on dfll resume and cpu and pllx clocks
>> restore, couldnt use clk_save_context and clk_restore_context for dfll.
>>
>> So implemented recommended dfll resume sequence in main Tegra210 clock
>> driver along with invoking clk_save_context/clk_restore_context where
>> all other clocks save/restore happens as per clock tree traversal.
> Could you please clarify what part of peripherals clocks is required for
> DFLL's restore? Couldn't DFLL driver be changed to avoid that quirkness
> and thus to make DFLL driver suspend/resume the clock?

DFLL source ref_clk and soc_clk need to be restored prior to dfll.

I see dfllCPU_out parent to CCLK_G first in the clock tree and dfll_ref 
and dfll_soc peripheral clocks are not resumed by the time dfll resume 
happens first.

ref_clk and soc_clk source is from pll_p and clock tree has these 
registered under pll_p which happens later.

tegra210_clock_init registers in order plls, peripheral clocks, 
super_clk init for cclk_g during clock driver probe and dfll probe and 
register happens later.



^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16  0:35         ` Sowjanya Komatineni
@ 2019-07-16  3:00           ` Sowjanya Komatineni
  2019-07-16  3:41             ` Sowjanya Komatineni
  2019-07-16  3:50             ` Dmitry Osipenko
  0 siblings, 2 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-16  3:00 UTC (permalink / raw)
  To: Dmitry Osipenko, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree


On 7/15/19 5:35 PM, Sowjanya Komatineni wrote:
>
> On 7/14/19 2:41 PM, Dmitry Osipenko wrote:
>> 13.07.2019 8:54, Sowjanya Komatineni пишет:
>>> On 6/29/19 8:10 AM, Dmitry Osipenko wrote:
>>>> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>>>>> This patch adds system suspend and resume support for Tegra210
>>>>> clocks.
>>>>>
>>>>> All the CAR controller settings are lost on suspend when core power
>>>>> goes off.
>>>>>
>>>>> This patch has implementation for saving and restoring all the PLLs
>>>>> and clocks context during system suspend and resume to have the
>>>>> clocks back to same state for normal operation.
>>>>>
>>>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>>>>> ---
>>>>>    drivers/clk/tegra/clk-tegra210.c | 115
>>>>> ++++++++++++++++++++++++++++++++++++++-
>>>>>    drivers/clk/tegra/clk.c          |  14 +++++
>>>>>    drivers/clk/tegra/clk.h          |   1 +
>>>>>    3 files changed, 127 insertions(+), 3 deletions(-)
>>>>>
>>>>> diff --git a/drivers/clk/tegra/clk-tegra210.c
>>>>> b/drivers/clk/tegra/clk-tegra210.c
>>>>> index 1c08c53482a5..1b839544e086 100644
>>>>> --- a/drivers/clk/tegra/clk-tegra210.c
>>>>> +++ b/drivers/clk/tegra/clk-tegra210.c
>>>>> @@ -9,10 +9,12 @@
>>>>>    #include <linux/clkdev.h>
>>>>>    #include <linux/of.h>
>>>>>    #include <linux/of_address.h>
>>>>> +#include <linux/of_platform.h>
>>>>>    #include <linux/delay.h>
>>>>>    #include <linux/export.h>
>>>>>    #include <linux/mutex.h>
>>>>>    #include <linux/clk/tegra.h>
>>>>> +#include <linux/syscore_ops.h>
>>>>>    #include <dt-bindings/clock/tegra210-car.h>
>>>>>    #include <dt-bindings/reset/tegra210-car.h>
>>>>>    #include <linux/iopoll.h>
>>>>> @@ -20,6 +22,7 @@
>>>>>    #include <soc/tegra/pmc.h>
>>>>>      #include "clk.h"
>>>>> +#include "clk-dfll.h"
>>>>>    #include "clk-id.h"
>>>>>      /*
>>>>> @@ -225,6 +228,7 @@
>>>>>      #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
>>>>>    #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
>>>>> +#define CPU_SOFTRST_CTRL 0x380
>>>>>      #define LVL2_CLK_GATE_OVRA 0xf8
>>>>>    #define LVL2_CLK_GATE_OVRC 0x3a0
>>>>> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
>>>>>        struct tegra_clk_pll_freq_table *fentry;
>>>>>        struct tegra_clk_pll pllu;
>>>>>        u32 reg;
>>>>> +    int ret;
>>>>>          for (fentry = pll_u_freq_table; fentry->input_rate; 
>>>>> fentry++) {
>>>>>            if (fentry->input_rate == pll_ref_freq)
>>>>> @@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
>>>>>        fence_udelay(1, clk_base);
>>>>>        reg |= PLL_ENABLE;
>>>>>        writel(reg, clk_base + PLLU_BASE);
>>>>> +    fence_udelay(1, clk_base);
>>>>>    -    readl_relaxed_poll_timeout_atomic(clk_base + PLLU_BASE, reg,
>>>>> -                      reg & PLL_BASE_LOCK, 2, 1000);
>>>>> -    if (!(reg & PLL_BASE_LOCK)) {
>>>>> +    ret = tegra210_wait_for_mask(&pllu, PLLU_BASE, PLL_BASE_LOCK);
>>>>> +    if (ret) {
>>>>>            pr_err("Timed out waiting for PLL_U to lock\n");
>>>>>            return -ETIMEDOUT;
>>>>>        }
>>>>> @@ -3283,6 +3288,103 @@ static void tegra210_disable_cpu_clock(u32 
>>>>> cpu)
>>>>>    }
>>>>>      #ifdef CONFIG_PM_SLEEP
>>>>> +static u32 cpu_softrst_ctx[3];
>>>>> +static struct platform_device *dfll_pdev;
>>>>> +#define car_readl(_base, _off) readl_relaxed(clk_base + (_base) +
>>>>> ((_off) * 4))
>>>>> +#define car_writel(_val, _base, _off) \
>>>>> +        writel_relaxed(_val, clk_base + (_base) + ((_off) * 4))
>>>>> +
>>>>> +static int tegra210_clk_suspend(void)
>>>>> +{
>>>>> +    unsigned int i;
>>>>> +    struct device_node *node;
>>>>> +
>>>>> +    tegra_cclkg_burst_policy_save_context();
>>>>> +
>>>>> +    if (!dfll_pdev) {
>>>>> +        node = of_find_compatible_node(NULL, NULL,
>>>>> +                           "nvidia,tegra210-dfll");
>>>>> +        if (node)
>>>>> +            dfll_pdev = of_find_device_by_node(node);
>>>>> +
>>>>> +        of_node_put(node);
>>>>> +        if (!dfll_pdev)
>>>>> +            pr_err("dfll node not found. no suspend for dfll\n");
>>>>> +    }
>>>>> +
>>>>> +    if (dfll_pdev)
>>>>> +        tegra_dfll_suspend(dfll_pdev);
>>>>> +
>>>>> +    /* Enable PLLP_OUT_CPU after dfll suspend */
>>>>> +    tegra_clk_set_pllp_out_cpu(true);
>>>>> +
>>>>> +    tegra_sclk_cclklp_burst_policy_save_context();
>>>>> +
>>>>> +    clk_save_context();
>>>>> +
>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>>> +        cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
>>>>> +
>>>>> +    return 0;
>>>>> +}
>>>>> +
>>>>> +static void tegra210_clk_resume(void)
>>>>> +{
>>>>> +    unsigned int i;
>>>>> +    struct clk_hw *parent;
>>>>> +    struct clk *clk;
>>>>> +
>>>>> +    /*
>>>>> +     * clk_restore_context restores clocks as per the clock tree.
>>>>> +     *
>>>>> +     * dfllCPU_out is first in the clock tree to get restored and it
>>>>> +     * involves programming DFLL controller along with restoring 
>>>>> CPUG
>>>>> +     * clock burst policy.
>>>>> +     *
>>>>> +     * DFLL programming needs dfll_ref and dfll_soc peripheral 
>>>>> clocks
>>>>> +     * to be restores which are part ofthe peripheral clocks.
>>                                              ^ white-space
>>
>> Please use spellchecker to avoid typos.
>>
>>>>> +     * So, peripheral clocks restore should happen prior to dfll 
>>>>> clock
>>>>> +     * restore.
>>>>> +     */
>>>>> +
>>>>> +    tegra_clk_osc_resume(clk_base);
>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>>> +        car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
>>>>> +
>>>>> +    /* restore all plls and peripheral clocks */
>>>>> +    tegra210_init_pllu();
>>>>> +    clk_restore_context();
>>>>> +
>>>>> +    fence_udelay(5, clk_base);
>>>>> +
>>>>> +    /* resume SCLK and CPULP clocks */
>>>>> +    tegra_sclk_cpulp_burst_policy_restore_context();
>>>>> +
>>>>> +    /*
>>>>> +     * restore CPUG clocks:
>>>>> +     * - enable DFLL in open loop mode
>>>>> +     * - switch CPUG to DFLL clock source
>>>>> +     * - close DFLL loop
>>>>> +     * - sync PLLX state
>>>>> +     */
>>>>> +    if (dfll_pdev)
>>>>> +        tegra_dfll_resume(dfll_pdev, false);
>>>>> +
>>>>> +    tegra_cclkg_burst_policy_restore_context();
>>>>> +    fence_udelay(2, clk_base);
>>>>> +
>>>>> +    if (dfll_pdev)
>>>>> +        tegra_dfll_resume(dfll_pdev, true);
>>>>> +
>>>>> +    parent =
>>>>> clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
>>>>> +    clk = clks[TEGRA210_CLK_PLL_X];
>>>>> +    if (parent != __clk_get_hw(clk))
>>>>> +        tegra_clk_sync_state_pll(__clk_get_hw(clk));
>>>>> +
>>>>> +    /* Disable PLL_OUT_CPU after DFLL resume */
>>>>> +    tegra_clk_set_pllp_out_cpu(false);
>>>>> +}
>>>>> +
>>>>>    static void tegra210_cpu_clock_suspend(void)
>>>>>    {
>>>>>        /* switch coresite to clk_m, save off original source */
>>>>> @@ -3298,6 +3400,11 @@ static void tegra210_cpu_clock_resume(void)
>>>>>    }
>>>>>    #endif
>>>>>    +static struct syscore_ops tegra_clk_syscore_ops = {
>>>>> +    .suspend = tegra210_clk_suspend,
>>>>> +    .resume = tegra210_clk_resume,
>>>>> +};
>>>>> +
>>>>>    static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
>>>>>        .wait_for_reset    = tegra210_wait_cpu_in_reset,
>>>>>        .disable_clock    = tegra210_disable_cpu_clock,
>>>>> @@ -3583,5 +3690,7 @@ static void __init tegra210_clock_init(struct
>>>>> device_node *np)
>>>>>        tegra210_mbist_clk_init();
>>>>>          tegra_cpu_car_ops = &tegra210_cpu_car_ops;
>>>>> +
>>>>> +    register_syscore_ops(&tegra_clk_syscore_ops);
>>>>>    }
>>>> Is it really worthwhile to use syscore_ops for suspend/resume given
>>>> that drivers for
>>>> won't resume before the CLK driver anyway? Are there any other options
>>>> for CLK
>>>> suspend/resume?
>>>>
>>>> I'm also not sure whether PM runtime API could be used at all in the
>>>> context of
>>>> syscore_ops ..
>>>>
>>>> Secondly, what about to use generic clk_save_context() /
>>>> clk_restore_context()
>>>> helpers for the suspend-resume? It looks to me that some other
>>>> essential (and proper)
>>>> platform driver (soc/tegra/? PMC?) should suspend-resume the clocks
>>>> using the generic
>>>> CLK Framework API.
>>> Clock resume should happen very early to restore peripheral and cpu
>>> clocks very early than peripheral drivers resume happens.
>> If all peripheral drivers properly requested all of the necessary clocks
>> and CLK driver was a platform driver, then I guess the probe should have
>> been naturally ordered. But that's not very achievable with the
>> currently available infrastructure in the kernel, so I'm not arguing
>> that the clocks should be explicitly resumed before the users.
>>
>>> this patch series uses clk_save_context and clk_restore_context for
>>> corresponding divider, pll, pllout.. save and restore context.
>> Now I see that indeed this API is utilized in this patch, thank you for
>> the clarification.
>>
>>> But as there is dependency on dfll resume and cpu and pllx clocks
>>> restore, couldnt use clk_save_context and clk_restore_context for dfll.
>>>
>>> So implemented recommended dfll resume sequence in main Tegra210 clock
>>> driver along with invoking clk_save_context/clk_restore_context where
>>> all other clocks save/restore happens as per clock tree traversal.
>> Could you please clarify what part of peripherals clocks is required for
>> DFLL's restore? Couldn't DFLL driver be changed to avoid that quirkness
>> and thus to make DFLL driver suspend/resume the clock?
>
> DFLL source ref_clk and soc_clk need to be restored prior to dfll.
>
> I see dfllCPU_out parent to CCLK_G first in the clock tree and 
> dfll_ref and dfll_soc peripheral clocks are not resumed by the time 
> dfll resume happens first.
>
> ref_clk and soc_clk source is from pll_p and clock tree has these 
> registered under pll_p which happens later.
>
> tegra210_clock_init registers in order plls, peripheral clocks, 
> super_clk init for cclk_g during clock driver probe and dfll probe and 
> register happens later.
>
One more thing, CLDVFS peripheral clock enable is also needed to be 
enabled to program DFLL Controller and all peripheral clock context is 
restored only after their PLL sources are restored.

DFLL restore involves dfll source clock resume along with CLDVFS 
periheral clock enable and reset


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16  3:00           ` Sowjanya Komatineni
@ 2019-07-16  3:41             ` Sowjanya Komatineni
  2019-07-16  3:50             ` Dmitry Osipenko
  1 sibling, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-16  3:41 UTC (permalink / raw)
  To: Dmitry Osipenko, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree


On 7/15/19 8:00 PM, Sowjanya Komatineni wrote:
>
> On 7/15/19 5:35 PM, Sowjanya Komatineni wrote:
>>
>> On 7/14/19 2:41 PM, Dmitry Osipenko wrote:
>>> 13.07.2019 8:54, Sowjanya Komatineni пишет:
>>>> On 6/29/19 8:10 AM, Dmitry Osipenko wrote:
>>>>> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>>>>>> This patch adds system suspend and resume support for Tegra210
>>>>>> clocks.
>>>>>>
>>>>>> All the CAR controller settings are lost on suspend when core power
>>>>>> goes off.
>>>>>>
>>>>>> This patch has implementation for saving and restoring all the PLLs
>>>>>> and clocks context during system suspend and resume to have the
>>>>>> clocks back to same state for normal operation.
>>>>>>
>>>>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>>>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>>>>>> ---
>>>>>>    drivers/clk/tegra/clk-tegra210.c | 115
>>>>>> ++++++++++++++++++++++++++++++++++++++-
>>>>>>    drivers/clk/tegra/clk.c          |  14 +++++
>>>>>>    drivers/clk/tegra/clk.h          |   1 +
>>>>>>    3 files changed, 127 insertions(+), 3 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/clk/tegra/clk-tegra210.c
>>>>>> b/drivers/clk/tegra/clk-tegra210.c
>>>>>> index 1c08c53482a5..1b839544e086 100644
>>>>>> --- a/drivers/clk/tegra/clk-tegra210.c
>>>>>> +++ b/drivers/clk/tegra/clk-tegra210.c
>>>>>> @@ -9,10 +9,12 @@
>>>>>>    #include <linux/clkdev.h>
>>>>>>    #include <linux/of.h>
>>>>>>    #include <linux/of_address.h>
>>>>>> +#include <linux/of_platform.h>
>>>>>>    #include <linux/delay.h>
>>>>>>    #include <linux/export.h>
>>>>>>    #include <linux/mutex.h>
>>>>>>    #include <linux/clk/tegra.h>
>>>>>> +#include <linux/syscore_ops.h>
>>>>>>    #include <dt-bindings/clock/tegra210-car.h>
>>>>>>    #include <dt-bindings/reset/tegra210-car.h>
>>>>>>    #include <linux/iopoll.h>
>>>>>> @@ -20,6 +22,7 @@
>>>>>>    #include <soc/tegra/pmc.h>
>>>>>>      #include "clk.h"
>>>>>> +#include "clk-dfll.h"
>>>>>>    #include "clk-id.h"
>>>>>>      /*
>>>>>> @@ -225,6 +228,7 @@
>>>>>>      #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
>>>>>>    #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
>>>>>> +#define CPU_SOFTRST_CTRL 0x380
>>>>>>      #define LVL2_CLK_GATE_OVRA 0xf8
>>>>>>    #define LVL2_CLK_GATE_OVRC 0x3a0
>>>>>> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
>>>>>>        struct tegra_clk_pll_freq_table *fentry;
>>>>>>        struct tegra_clk_pll pllu;
>>>>>>        u32 reg;
>>>>>> +    int ret;
>>>>>>          for (fentry = pll_u_freq_table; fentry->input_rate; 
>>>>>> fentry++) {
>>>>>>            if (fentry->input_rate == pll_ref_freq)
>>>>>> @@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
>>>>>>        fence_udelay(1, clk_base);
>>>>>>        reg |= PLL_ENABLE;
>>>>>>        writel(reg, clk_base + PLLU_BASE);
>>>>>> +    fence_udelay(1, clk_base);
>>>>>>    -    readl_relaxed_poll_timeout_atomic(clk_base + PLLU_BASE, reg,
>>>>>> -                      reg & PLL_BASE_LOCK, 2, 1000);
>>>>>> -    if (!(reg & PLL_BASE_LOCK)) {
>>>>>> +    ret = tegra210_wait_for_mask(&pllu, PLLU_BASE, PLL_BASE_LOCK);
>>>>>> +    if (ret) {
>>>>>>            pr_err("Timed out waiting for PLL_U to lock\n");
>>>>>>            return -ETIMEDOUT;
>>>>>>        }
>>>>>> @@ -3283,6 +3288,103 @@ static void 
>>>>>> tegra210_disable_cpu_clock(u32 cpu)
>>>>>>    }
>>>>>>      #ifdef CONFIG_PM_SLEEP
>>>>>> +static u32 cpu_softrst_ctx[3];
>>>>>> +static struct platform_device *dfll_pdev;
>>>>>> +#define car_readl(_base, _off) readl_relaxed(clk_base + (_base) +
>>>>>> ((_off) * 4))
>>>>>> +#define car_writel(_val, _base, _off) \
>>>>>> +        writel_relaxed(_val, clk_base + (_base) + ((_off) * 4))
>>>>>> +
>>>>>> +static int tegra210_clk_suspend(void)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +    struct device_node *node;
>>>>>> +
>>>>>> +    tegra_cclkg_burst_policy_save_context();
>>>>>> +
>>>>>> +    if (!dfll_pdev) {
>>>>>> +        node = of_find_compatible_node(NULL, NULL,
>>>>>> +                           "nvidia,tegra210-dfll");
>>>>>> +        if (node)
>>>>>> +            dfll_pdev = of_find_device_by_node(node);
>>>>>> +
>>>>>> +        of_node_put(node);
>>>>>> +        if (!dfll_pdev)
>>>>>> +            pr_err("dfll node not found. no suspend for dfll\n");
>>>>>> +    }
>>>>>> +
>>>>>> +    if (dfll_pdev)
>>>>>> +        tegra_dfll_suspend(dfll_pdev);
>>>>>> +
>>>>>> +    /* Enable PLLP_OUT_CPU after dfll suspend */
>>>>>> +    tegra_clk_set_pllp_out_cpu(true);
>>>>>> +
>>>>>> +    tegra_sclk_cclklp_burst_policy_save_context();
>>>>>> +
>>>>>> +    clk_save_context();
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>>>> +        cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static void tegra210_clk_resume(void)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +    struct clk_hw *parent;
>>>>>> +    struct clk *clk;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * clk_restore_context restores clocks as per the clock tree.
>>>>>> +     *
>>>>>> +     * dfllCPU_out is first in the clock tree to get restored 
>>>>>> and it
>>>>>> +     * involves programming DFLL controller along with restoring 
>>>>>> CPUG
>>>>>> +     * clock burst policy.
>>>>>> +     *
>>>>>> +     * DFLL programming needs dfll_ref and dfll_soc peripheral 
>>>>>> clocks
>>>>>> +     * to be restores which are part ofthe peripheral clocks.
>>>                                              ^ white-space
>>>
>>> Please use spellchecker to avoid typos.
>>>
>>>>>> +     * So, peripheral clocks restore should happen prior to dfll 
>>>>>> clock
>>>>>> +     * restore.
>>>>>> +     */
>>>>>> +
>>>>>> +    tegra_clk_osc_resume(clk_base);
>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>>>> +        car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
>>>>>> +
>>>>>> +    /* restore all plls and peripheral clocks */
>>>>>> +    tegra210_init_pllu();
>>>>>> +    clk_restore_context();
>>>>>> +
>>>>>> +    fence_udelay(5, clk_base);
>>>>>> +
>>>>>> +    /* resume SCLK and CPULP clocks */
>>>>>> +    tegra_sclk_cpulp_burst_policy_restore_context();
>>>>>> +
>>>>>> +    /*
>>>>>> +     * restore CPUG clocks:
>>>>>> +     * - enable DFLL in open loop mode
>>>>>> +     * - switch CPUG to DFLL clock source
>>>>>> +     * - close DFLL loop
>>>>>> +     * - sync PLLX state
>>>>>> +     */
>>>>>> +    if (dfll_pdev)
>>>>>> +        tegra_dfll_resume(dfll_pdev, false);
>>>>>> +
>>>>>> +    tegra_cclkg_burst_policy_restore_context();
>>>>>> +    fence_udelay(2, clk_base);
>>>>>> +
>>>>>> +    if (dfll_pdev)
>>>>>> +        tegra_dfll_resume(dfll_pdev, true);
>>>>>> +
>>>>>> +    parent =
>>>>>> clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
>>>>>> +    clk = clks[TEGRA210_CLK_PLL_X];
>>>>>> +    if (parent != __clk_get_hw(clk))
>>>>>> +        tegra_clk_sync_state_pll(__clk_get_hw(clk));
>>>>>> +
>>>>>> +    /* Disable PLL_OUT_CPU after DFLL resume */
>>>>>> +    tegra_clk_set_pllp_out_cpu(false);
>>>>>> +}
>>>>>> +
>>>>>>    static void tegra210_cpu_clock_suspend(void)
>>>>>>    {
>>>>>>        /* switch coresite to clk_m, save off original source */
>>>>>> @@ -3298,6 +3400,11 @@ static void tegra210_cpu_clock_resume(void)
>>>>>>    }
>>>>>>    #endif
>>>>>>    +static struct syscore_ops tegra_clk_syscore_ops = {
>>>>>> +    .suspend = tegra210_clk_suspend,
>>>>>> +    .resume = tegra210_clk_resume,
>>>>>> +};
>>>>>> +
>>>>>>    static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
>>>>>>        .wait_for_reset    = tegra210_wait_cpu_in_reset,
>>>>>>        .disable_clock    = tegra210_disable_cpu_clock,
>>>>>> @@ -3583,5 +3690,7 @@ static void __init tegra210_clock_init(struct
>>>>>> device_node *np)
>>>>>>        tegra210_mbist_clk_init();
>>>>>>          tegra_cpu_car_ops = &tegra210_cpu_car_ops;
>>>>>> +
>>>>>> +    register_syscore_ops(&tegra_clk_syscore_ops);
>>>>>>    }
>>>>> Is it really worthwhile to use syscore_ops for suspend/resume given
>>>>> that drivers for
>>>>> won't resume before the CLK driver anyway? Are there any other 
>>>>> options
>>>>> for CLK
>>>>> suspend/resume?
>>>>>
>>>>> I'm also not sure whether PM runtime API could be used at all in the
>>>>> context of
>>>>> syscore_ops ..
>>>>>
>>>>> Secondly, what about to use generic clk_save_context() /
>>>>> clk_restore_context()
>>>>> helpers for the suspend-resume? It looks to me that some other
>>>>> essential (and proper)
>>>>> platform driver (soc/tegra/? PMC?) should suspend-resume the clocks
>>>>> using the generic
>>>>> CLK Framework API.
>>>> Clock resume should happen very early to restore peripheral and cpu
>>>> clocks very early than peripheral drivers resume happens.
>>> If all peripheral drivers properly requested all of the necessary 
>>> clocks
>>> and CLK driver was a platform driver, then I guess the probe should 
>>> have
>>> been naturally ordered. But that's not very achievable with the
>>> currently available infrastructure in the kernel, so I'm not arguing
>>> that the clocks should be explicitly resumed before the users.
>>>
>>>> this patch series uses clk_save_context and clk_restore_context for
>>>> corresponding divider, pll, pllout.. save and restore context.
>>> Now I see that indeed this API is utilized in this patch, thank you for
>>> the clarification.
>>>
>>>> But as there is dependency on dfll resume and cpu and pllx clocks
>>>> restore, couldnt use clk_save_context and clk_restore_context for 
>>>> dfll.
>>>>
>>>> So implemented recommended dfll resume sequence in main Tegra210 clock
>>>> driver along with invoking clk_save_context/clk_restore_context where
>>>> all other clocks save/restore happens as per clock tree traversal.
>>> Could you please clarify what part of peripherals clocks is required 
>>> for
>>> DFLL's restore? Couldn't DFLL driver be changed to avoid that quirkness
>>> and thus to make DFLL driver suspend/resume the clock?
>>
>> DFLL source ref_clk and soc_clk need to be restored prior to dfll.
>>
>> I see dfllCPU_out parent to CCLK_G first in the clock tree and 
>> dfll_ref and dfll_soc peripheral clocks are not resumed by the time 
>> dfll resume happens first.
>>
>> ref_clk and soc_clk source is from pll_p and clock tree has these 
>> registered under pll_p which happens later.
>>
>> tegra210_clock_init registers in order plls, peripheral clocks, 
>> super_clk init for cclk_g during clock driver probe and dfll probe 
>> and register happens later.
>>
> One more thing, CLDVFS peripheral clock enable is also needed to be 
> enabled to program DFLL Controller and all peripheral clock context is 
> restored only after their PLL sources are restored.
>
> DFLL restore involves dfll source clock resume along with CLDVFS 
> periheral clock enable and reset
>
Will try with dfll_pm_ops instead of dfll suspend/resume in tegra210 
clock driver...

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16  3:00           ` Sowjanya Komatineni
  2019-07-16  3:41             ` Sowjanya Komatineni
@ 2019-07-16  3:50             ` Dmitry Osipenko
  2019-07-16  4:37               ` Sowjanya Komatineni
  1 sibling, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-16  3:50 UTC (permalink / raw)
  To: Sowjanya Komatineni, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree

16.07.2019 6:00, Sowjanya Komatineni пишет:
> 
> On 7/15/19 5:35 PM, Sowjanya Komatineni wrote:
>>
>> On 7/14/19 2:41 PM, Dmitry Osipenko wrote:
>>> 13.07.2019 8:54, Sowjanya Komatineni пишет:
>>>> On 6/29/19 8:10 AM, Dmitry Osipenko wrote:
>>>>> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>>>>>> This patch adds system suspend and resume support for Tegra210
>>>>>> clocks.
>>>>>>
>>>>>> All the CAR controller settings are lost on suspend when core power
>>>>>> goes off.
>>>>>>
>>>>>> This patch has implementation for saving and restoring all the PLLs
>>>>>> and clocks context during system suspend and resume to have the
>>>>>> clocks back to same state for normal operation.
>>>>>>
>>>>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>>>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>>>>>> ---
>>>>>>    drivers/clk/tegra/clk-tegra210.c | 115
>>>>>> ++++++++++++++++++++++++++++++++++++++-
>>>>>>    drivers/clk/tegra/clk.c          |  14 +++++
>>>>>>    drivers/clk/tegra/clk.h          |   1 +
>>>>>>    3 files changed, 127 insertions(+), 3 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/clk/tegra/clk-tegra210.c
>>>>>> b/drivers/clk/tegra/clk-tegra210.c
>>>>>> index 1c08c53482a5..1b839544e086 100644
>>>>>> --- a/drivers/clk/tegra/clk-tegra210.c
>>>>>> +++ b/drivers/clk/tegra/clk-tegra210.c
>>>>>> @@ -9,10 +9,12 @@
>>>>>>    #include <linux/clkdev.h>
>>>>>>    #include <linux/of.h>
>>>>>>    #include <linux/of_address.h>
>>>>>> +#include <linux/of_platform.h>
>>>>>>    #include <linux/delay.h>
>>>>>>    #include <linux/export.h>
>>>>>>    #include <linux/mutex.h>
>>>>>>    #include <linux/clk/tegra.h>
>>>>>> +#include <linux/syscore_ops.h>
>>>>>>    #include <dt-bindings/clock/tegra210-car.h>
>>>>>>    #include <dt-bindings/reset/tegra210-car.h>
>>>>>>    #include <linux/iopoll.h>
>>>>>> @@ -20,6 +22,7 @@
>>>>>>    #include <soc/tegra/pmc.h>
>>>>>>      #include "clk.h"
>>>>>> +#include "clk-dfll.h"
>>>>>>    #include "clk-id.h"
>>>>>>      /*
>>>>>> @@ -225,6 +228,7 @@
>>>>>>      #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
>>>>>>    #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
>>>>>> +#define CPU_SOFTRST_CTRL 0x380
>>>>>>      #define LVL2_CLK_GATE_OVRA 0xf8
>>>>>>    #define LVL2_CLK_GATE_OVRC 0x3a0
>>>>>> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
>>>>>>        struct tegra_clk_pll_freq_table *fentry;
>>>>>>        struct tegra_clk_pll pllu;
>>>>>>        u32 reg;
>>>>>> +    int ret;
>>>>>>          for (fentry = pll_u_freq_table; fentry->input_rate;
>>>>>> fentry++) {
>>>>>>            if (fentry->input_rate == pll_ref_freq)
>>>>>> @@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
>>>>>>        fence_udelay(1, clk_base);
>>>>>>        reg |= PLL_ENABLE;
>>>>>>        writel(reg, clk_base + PLLU_BASE);
>>>>>> +    fence_udelay(1, clk_base);
>>>>>>    -    readl_relaxed_poll_timeout_atomic(clk_base + PLLU_BASE, reg,
>>>>>> -                      reg & PLL_BASE_LOCK, 2, 1000);
>>>>>> -    if (!(reg & PLL_BASE_LOCK)) {
>>>>>> +    ret = tegra210_wait_for_mask(&pllu, PLLU_BASE, PLL_BASE_LOCK);
>>>>>> +    if (ret) {
>>>>>>            pr_err("Timed out waiting for PLL_U to lock\n");
>>>>>>            return -ETIMEDOUT;
>>>>>>        }
>>>>>> @@ -3283,6 +3288,103 @@ static void tegra210_disable_cpu_clock(u32
>>>>>> cpu)
>>>>>>    }
>>>>>>      #ifdef CONFIG_PM_SLEEP
>>>>>> +static u32 cpu_softrst_ctx[3];
>>>>>> +static struct platform_device *dfll_pdev;
>>>>>> +#define car_readl(_base, _off) readl_relaxed(clk_base + (_base) +
>>>>>> ((_off) * 4))
>>>>>> +#define car_writel(_val, _base, _off) \
>>>>>> +        writel_relaxed(_val, clk_base + (_base) + ((_off) * 4))
>>>>>> +
>>>>>> +static int tegra210_clk_suspend(void)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +    struct device_node *node;
>>>>>> +
>>>>>> +    tegra_cclkg_burst_policy_save_context();
>>>>>> +
>>>>>> +    if (!dfll_pdev) {
>>>>>> +        node = of_find_compatible_node(NULL, NULL,
>>>>>> +                           "nvidia,tegra210-dfll");
>>>>>> +        if (node)
>>>>>> +            dfll_pdev = of_find_device_by_node(node);
>>>>>> +
>>>>>> +        of_node_put(node);
>>>>>> +        if (!dfll_pdev)
>>>>>> +            pr_err("dfll node not found. no suspend for dfll\n");
>>>>>> +    }
>>>>>> +
>>>>>> +    if (dfll_pdev)
>>>>>> +        tegra_dfll_suspend(dfll_pdev);
>>>>>> +
>>>>>> +    /* Enable PLLP_OUT_CPU after dfll suspend */
>>>>>> +    tegra_clk_set_pllp_out_cpu(true);
>>>>>> +
>>>>>> +    tegra_sclk_cclklp_burst_policy_save_context();
>>>>>> +
>>>>>> +    clk_save_context();
>>>>>> +
>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>>>> +        cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
>>>>>> +
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static void tegra210_clk_resume(void)
>>>>>> +{
>>>>>> +    unsigned int i;
>>>>>> +    struct clk_hw *parent;
>>>>>> +    struct clk *clk;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * clk_restore_context restores clocks as per the clock tree.
>>>>>> +     *
>>>>>> +     * dfllCPU_out is first in the clock tree to get restored and it
>>>>>> +     * involves programming DFLL controller along with restoring
>>>>>> CPUG
>>>>>> +     * clock burst policy.
>>>>>> +     *
>>>>>> +     * DFLL programming needs dfll_ref and dfll_soc peripheral
>>>>>> clocks
>>>>>> +     * to be restores which are part ofthe peripheral clocks.
>>>                                              ^ white-space
>>>
>>> Please use spellchecker to avoid typos.
>>>
>>>>>> +     * So, peripheral clocks restore should happen prior to dfll
>>>>>> clock
>>>>>> +     * restore.
>>>>>> +     */
>>>>>> +
>>>>>> +    tegra_clk_osc_resume(clk_base);
>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>>>> +        car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
>>>>>> +
>>>>>> +    /* restore all plls and peripheral clocks */
>>>>>> +    tegra210_init_pllu();
>>>>>> +    clk_restore_context();
>>>>>> +
>>>>>> +    fence_udelay(5, clk_base);
>>>>>> +
>>>>>> +    /* resume SCLK and CPULP clocks */
>>>>>> +    tegra_sclk_cpulp_burst_policy_restore_context();
>>>>>> +
>>>>>> +    /*
>>>>>> +     * restore CPUG clocks:
>>>>>> +     * - enable DFLL in open loop mode
>>>>>> +     * - switch CPUG to DFLL clock source
>>>>>> +     * - close DFLL loop
>>>>>> +     * - sync PLLX state
>>>>>> +     */
>>>>>> +    if (dfll_pdev)
>>>>>> +        tegra_dfll_resume(dfll_pdev, false);
>>>>>> +
>>>>>> +    tegra_cclkg_burst_policy_restore_context();
>>>>>> +    fence_udelay(2, clk_base);
>>>>>> +
>>>>>> +    if (dfll_pdev)
>>>>>> +        tegra_dfll_resume(dfll_pdev, true);
>>>>>> +
>>>>>> +    parent =
>>>>>> clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
>>>>>> +    clk = clks[TEGRA210_CLK_PLL_X];
>>>>>> +    if (parent != __clk_get_hw(clk))
>>>>>> +        tegra_clk_sync_state_pll(__clk_get_hw(clk));
>>>>>> +
>>>>>> +    /* Disable PLL_OUT_CPU after DFLL resume */
>>>>>> +    tegra_clk_set_pllp_out_cpu(false);
>>>>>> +}
>>>>>> +
>>>>>>    static void tegra210_cpu_clock_suspend(void)
>>>>>>    {
>>>>>>        /* switch coresite to clk_m, save off original source */
>>>>>> @@ -3298,6 +3400,11 @@ static void tegra210_cpu_clock_resume(void)
>>>>>>    }
>>>>>>    #endif
>>>>>>    +static struct syscore_ops tegra_clk_syscore_ops = {
>>>>>> +    .suspend = tegra210_clk_suspend,
>>>>>> +    .resume = tegra210_clk_resume,
>>>>>> +};
>>>>>> +
>>>>>>    static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
>>>>>>        .wait_for_reset    = tegra210_wait_cpu_in_reset,
>>>>>>        .disable_clock    = tegra210_disable_cpu_clock,
>>>>>> @@ -3583,5 +3690,7 @@ static void __init tegra210_clock_init(struct
>>>>>> device_node *np)
>>>>>>        tegra210_mbist_clk_init();
>>>>>>          tegra_cpu_car_ops = &tegra210_cpu_car_ops;
>>>>>> +
>>>>>> +    register_syscore_ops(&tegra_clk_syscore_ops);
>>>>>>    }
>>>>> Is it really worthwhile to use syscore_ops for suspend/resume given
>>>>> that drivers for
>>>>> won't resume before the CLK driver anyway? Are there any other options
>>>>> for CLK
>>>>> suspend/resume?
>>>>>
>>>>> I'm also not sure whether PM runtime API could be used at all in the
>>>>> context of
>>>>> syscore_ops ..
>>>>>
>>>>> Secondly, what about to use generic clk_save_context() /
>>>>> clk_restore_context()
>>>>> helpers for the suspend-resume? It looks to me that some other
>>>>> essential (and proper)
>>>>> platform driver (soc/tegra/? PMC?) should suspend-resume the clocks
>>>>> using the generic
>>>>> CLK Framework API.
>>>> Clock resume should happen very early to restore peripheral and cpu
>>>> clocks very early than peripheral drivers resume happens.
>>> If all peripheral drivers properly requested all of the necessary clocks
>>> and CLK driver was a platform driver, then I guess the probe should have
>>> been naturally ordered. But that's not very achievable with the
>>> currently available infrastructure in the kernel, so I'm not arguing
>>> that the clocks should be explicitly resumed before the users.
>>>
>>>> this patch series uses clk_save_context and clk_restore_context for
>>>> corresponding divider, pll, pllout.. save and restore context.
>>> Now I see that indeed this API is utilized in this patch, thank you for
>>> the clarification.
>>>
>>>> But as there is dependency on dfll resume and cpu and pllx clocks
>>>> restore, couldnt use clk_save_context and clk_restore_context for dfll.
>>>>
>>>> So implemented recommended dfll resume sequence in main Tegra210 clock
>>>> driver along with invoking clk_save_context/clk_restore_context where
>>>> all other clocks save/restore happens as per clock tree traversal.
>>> Could you please clarify what part of peripherals clocks is required for
>>> DFLL's restore? Couldn't DFLL driver be changed to avoid that quirkness
>>> and thus to make DFLL driver suspend/resume the clock?
>>
>> DFLL source ref_clk and soc_clk need to be restored prior to dfll.
>>
>> I see dfllCPU_out parent to CCLK_G first in the clock tree and
>> dfll_ref and dfll_soc peripheral clocks are not resumed by the time
>> dfll resume happens first.
>>
>> ref_clk and soc_clk source is from pll_p and clock tree has these
>> registered under pll_p which happens later.
>>
>> tegra210_clock_init registers in order plls, peripheral clocks,
>> super_clk init for cclk_g during clock driver probe and dfll probe and
>> register happens later.
>>
> One more thing, CLDVFS peripheral clock enable is also needed to be
> enabled to program DFLL Controller and all peripheral clock context is
> restored only after their PLL sources are restored.
> 
> DFLL restore involves dfll source clock resume along with CLDVFS
> periheral clock enable and reset
> 

I don't quite see why you can't simply add suspend/resume callbacks to
the CPUFreq driver to:

On suspend:
1. Switch CPU to PLLP (or whatever "safe" parent)
2. Disable/teardown DFLL

On resume:
1. Enable/restore DFLL
2. Switch CPU back to DFLL

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16  3:50             ` Dmitry Osipenko
@ 2019-07-16  4:37               ` Sowjanya Komatineni
  2019-07-16  5:37                 ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-16  4:37 UTC (permalink / raw)
  To: Dmitry Osipenko, thierry.reding, jonathanh, tglx, jason,
	marc.zyngier, linus.walleij, stefan, mark.rutland
  Cc: pdeschrijver, pgaikwad, sboyd, linux-clk, linux-gpio, jckuo,
	josephl, talho, linux-tegra, linux-kernel, mperttunen, spatra,
	robh+dt, devicetree


On 7/15/19 8:50 PM, Dmitry Osipenko wrote:
> 16.07.2019 6:00, Sowjanya Komatineni пишет:
>> On 7/15/19 5:35 PM, Sowjanya Komatineni wrote:
>>> On 7/14/19 2:41 PM, Dmitry Osipenko wrote:
>>>> 13.07.2019 8:54, Sowjanya Komatineni пишет:
>>>>> On 6/29/19 8:10 AM, Dmitry Osipenko wrote:
>>>>>> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>>>>>>> This patch adds system suspend and resume support for Tegra210
>>>>>>> clocks.
>>>>>>>
>>>>>>> All the CAR controller settings are lost on suspend when core power
>>>>>>> goes off.
>>>>>>>
>>>>>>> This patch has implementation for saving and restoring all the PLLs
>>>>>>> and clocks context during system suspend and resume to have the
>>>>>>> clocks back to same state for normal operation.
>>>>>>>
>>>>>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>>>>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>>>>>>> ---
>>>>>>>     drivers/clk/tegra/clk-tegra210.c | 115
>>>>>>> ++++++++++++++++++++++++++++++++++++++-
>>>>>>>     drivers/clk/tegra/clk.c          |  14 +++++
>>>>>>>     drivers/clk/tegra/clk.h          |   1 +
>>>>>>>     3 files changed, 127 insertions(+), 3 deletions(-)
>>>>>>>
>>>>>>> diff --git a/drivers/clk/tegra/clk-tegra210.c
>>>>>>> b/drivers/clk/tegra/clk-tegra210.c
>>>>>>> index 1c08c53482a5..1b839544e086 100644
>>>>>>> --- a/drivers/clk/tegra/clk-tegra210.c
>>>>>>> +++ b/drivers/clk/tegra/clk-tegra210.c
>>>>>>> @@ -9,10 +9,12 @@
>>>>>>>     #include <linux/clkdev.h>
>>>>>>>     #include <linux/of.h>
>>>>>>>     #include <linux/of_address.h>
>>>>>>> +#include <linux/of_platform.h>
>>>>>>>     #include <linux/delay.h>
>>>>>>>     #include <linux/export.h>
>>>>>>>     #include <linux/mutex.h>
>>>>>>>     #include <linux/clk/tegra.h>
>>>>>>> +#include <linux/syscore_ops.h>
>>>>>>>     #include <dt-bindings/clock/tegra210-car.h>
>>>>>>>     #include <dt-bindings/reset/tegra210-car.h>
>>>>>>>     #include <linux/iopoll.h>
>>>>>>> @@ -20,6 +22,7 @@
>>>>>>>     #include <soc/tegra/pmc.h>
>>>>>>>       #include "clk.h"
>>>>>>> +#include "clk-dfll.h"
>>>>>>>     #include "clk-id.h"
>>>>>>>       /*
>>>>>>> @@ -225,6 +228,7 @@
>>>>>>>       #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
>>>>>>>     #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
>>>>>>> +#define CPU_SOFTRST_CTRL 0x380
>>>>>>>       #define LVL2_CLK_GATE_OVRA 0xf8
>>>>>>>     #define LVL2_CLK_GATE_OVRC 0x3a0
>>>>>>> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
>>>>>>>         struct tegra_clk_pll_freq_table *fentry;
>>>>>>>         struct tegra_clk_pll pllu;
>>>>>>>         u32 reg;
>>>>>>> +    int ret;
>>>>>>>           for (fentry = pll_u_freq_table; fentry->input_rate;
>>>>>>> fentry++) {
>>>>>>>             if (fentry->input_rate == pll_ref_freq)
>>>>>>> @@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
>>>>>>>         fence_udelay(1, clk_base);
>>>>>>>         reg |= PLL_ENABLE;
>>>>>>>         writel(reg, clk_base + PLLU_BASE);
>>>>>>> +    fence_udelay(1, clk_base);
>>>>>>>     -    readl_relaxed_poll_timeout_atomic(clk_base + PLLU_BASE, reg,
>>>>>>> -                      reg & PLL_BASE_LOCK, 2, 1000);
>>>>>>> -    if (!(reg & PLL_BASE_LOCK)) {
>>>>>>> +    ret = tegra210_wait_for_mask(&pllu, PLLU_BASE, PLL_BASE_LOCK);
>>>>>>> +    if (ret) {
>>>>>>>             pr_err("Timed out waiting for PLL_U to lock\n");
>>>>>>>             return -ETIMEDOUT;
>>>>>>>         }
>>>>>>> @@ -3283,6 +3288,103 @@ static void tegra210_disable_cpu_clock(u32
>>>>>>> cpu)
>>>>>>>     }
>>>>>>>       #ifdef CONFIG_PM_SLEEP
>>>>>>> +static u32 cpu_softrst_ctx[3];
>>>>>>> +static struct platform_device *dfll_pdev;
>>>>>>> +#define car_readl(_base, _off) readl_relaxed(clk_base + (_base) +
>>>>>>> ((_off) * 4))
>>>>>>> +#define car_writel(_val, _base, _off) \
>>>>>>> +        writel_relaxed(_val, clk_base + (_base) + ((_off) * 4))
>>>>>>> +
>>>>>>> +static int tegra210_clk_suspend(void)
>>>>>>> +{
>>>>>>> +    unsigned int i;
>>>>>>> +    struct device_node *node;
>>>>>>> +
>>>>>>> +    tegra_cclkg_burst_policy_save_context();
>>>>>>> +
>>>>>>> +    if (!dfll_pdev) {
>>>>>>> +        node = of_find_compatible_node(NULL, NULL,
>>>>>>> +                           "nvidia,tegra210-dfll");
>>>>>>> +        if (node)
>>>>>>> +            dfll_pdev = of_find_device_by_node(node);
>>>>>>> +
>>>>>>> +        of_node_put(node);
>>>>>>> +        if (!dfll_pdev)
>>>>>>> +            pr_err("dfll node not found. no suspend for dfll\n");
>>>>>>> +    }
>>>>>>> +
>>>>>>> +    if (dfll_pdev)
>>>>>>> +        tegra_dfll_suspend(dfll_pdev);
>>>>>>> +
>>>>>>> +    /* Enable PLLP_OUT_CPU after dfll suspend */
>>>>>>> +    tegra_clk_set_pllp_out_cpu(true);
>>>>>>> +
>>>>>>> +    tegra_sclk_cclklp_burst_policy_save_context();
>>>>>>> +
>>>>>>> +    clk_save_context();
>>>>>>> +
>>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>>>>> +        cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
>>>>>>> +
>>>>>>> +    return 0;
>>>>>>> +}
>>>>>>> +
>>>>>>> +static void tegra210_clk_resume(void)
>>>>>>> +{
>>>>>>> +    unsigned int i;
>>>>>>> +    struct clk_hw *parent;
>>>>>>> +    struct clk *clk;
>>>>>>> +
>>>>>>> +    /*
>>>>>>> +     * clk_restore_context restores clocks as per the clock tree.
>>>>>>> +     *
>>>>>>> +     * dfllCPU_out is first in the clock tree to get restored and it
>>>>>>> +     * involves programming DFLL controller along with restoring
>>>>>>> CPUG
>>>>>>> +     * clock burst policy.
>>>>>>> +     *
>>>>>>> +     * DFLL programming needs dfll_ref and dfll_soc peripheral
>>>>>>> clocks
>>>>>>> +     * to be restores which are part ofthe peripheral clocks.
>>>>                                               ^ white-space
>>>>
>>>> Please use spellchecker to avoid typos.
>>>>
>>>>>>> +     * So, peripheral clocks restore should happen prior to dfll
>>>>>>> clock
>>>>>>> +     * restore.
>>>>>>> +     */
>>>>>>> +
>>>>>>> +    tegra_clk_osc_resume(clk_base);
>>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>>>>> +        car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
>>>>>>> +
>>>>>>> +    /* restore all plls and peripheral clocks */
>>>>>>> +    tegra210_init_pllu();
>>>>>>> +    clk_restore_context();
>>>>>>> +
>>>>>>> +    fence_udelay(5, clk_base);
>>>>>>> +
>>>>>>> +    /* resume SCLK and CPULP clocks */
>>>>>>> +    tegra_sclk_cpulp_burst_policy_restore_context();
>>>>>>> +
>>>>>>> +    /*
>>>>>>> +     * restore CPUG clocks:
>>>>>>> +     * - enable DFLL in open loop mode
>>>>>>> +     * - switch CPUG to DFLL clock source
>>>>>>> +     * - close DFLL loop
>>>>>>> +     * - sync PLLX state
>>>>>>> +     */
>>>>>>> +    if (dfll_pdev)
>>>>>>> +        tegra_dfll_resume(dfll_pdev, false);
>>>>>>> +
>>>>>>> +    tegra_cclkg_burst_policy_restore_context();
>>>>>>> +    fence_udelay(2, clk_base);
>>>>>>> +
>>>>>>> +    if (dfll_pdev)
>>>>>>> +        tegra_dfll_resume(dfll_pdev, true);
>>>>>>> +
>>>>>>> +    parent =
>>>>>>> clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
>>>>>>> +    clk = clks[TEGRA210_CLK_PLL_X];
>>>>>>> +    if (parent != __clk_get_hw(clk))
>>>>>>> +        tegra_clk_sync_state_pll(__clk_get_hw(clk));
>>>>>>> +
>>>>>>> +    /* Disable PLL_OUT_CPU after DFLL resume */
>>>>>>> +    tegra_clk_set_pllp_out_cpu(false);
>>>>>>> +}
>>>>>>> +
>>>>>>>     static void tegra210_cpu_clock_suspend(void)
>>>>>>>     {
>>>>>>>         /* switch coresite to clk_m, save off original source */
>>>>>>> @@ -3298,6 +3400,11 @@ static void tegra210_cpu_clock_resume(void)
>>>>>>>     }
>>>>>>>     #endif
>>>>>>>     +static struct syscore_ops tegra_clk_syscore_ops = {
>>>>>>> +    .suspend = tegra210_clk_suspend,
>>>>>>> +    .resume = tegra210_clk_resume,
>>>>>>> +};
>>>>>>> +
>>>>>>>     static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
>>>>>>>         .wait_for_reset    = tegra210_wait_cpu_in_reset,
>>>>>>>         .disable_clock    = tegra210_disable_cpu_clock,
>>>>>>> @@ -3583,5 +3690,7 @@ static void __init tegra210_clock_init(struct
>>>>>>> device_node *np)
>>>>>>>         tegra210_mbist_clk_init();
>>>>>>>           tegra_cpu_car_ops = &tegra210_cpu_car_ops;
>>>>>>> +
>>>>>>> +    register_syscore_ops(&tegra_clk_syscore_ops);
>>>>>>>     }
>>>>>> Is it really worthwhile to use syscore_ops for suspend/resume given
>>>>>> that drivers for
>>>>>> won't resume before the CLK driver anyway? Are there any other options
>>>>>> for CLK
>>>>>> suspend/resume?
>>>>>>
>>>>>> I'm also not sure whether PM runtime API could be used at all in the
>>>>>> context of
>>>>>> syscore_ops ..
>>>>>>
>>>>>> Secondly, what about to use generic clk_save_context() /
>>>>>> clk_restore_context()
>>>>>> helpers for the suspend-resume? It looks to me that some other
>>>>>> essential (and proper)
>>>>>> platform driver (soc/tegra/? PMC?) should suspend-resume the clocks
>>>>>> using the generic
>>>>>> CLK Framework API.
>>>>> Clock resume should happen very early to restore peripheral and cpu
>>>>> clocks very early than peripheral drivers resume happens.
>>>> If all peripheral drivers properly requested all of the necessary clocks
>>>> and CLK driver was a platform driver, then I guess the probe should have
>>>> been naturally ordered. But that's not very achievable with the
>>>> currently available infrastructure in the kernel, so I'm not arguing
>>>> that the clocks should be explicitly resumed before the users.
>>>>
>>>>> this patch series uses clk_save_context and clk_restore_context for
>>>>> corresponding divider, pll, pllout.. save and restore context.
>>>> Now I see that indeed this API is utilized in this patch, thank you for
>>>> the clarification.
>>>>
>>>>> But as there is dependency on dfll resume and cpu and pllx clocks
>>>>> restore, couldnt use clk_save_context and clk_restore_context for dfll.
>>>>>
>>>>> So implemented recommended dfll resume sequence in main Tegra210 clock
>>>>> driver along with invoking clk_save_context/clk_restore_context where
>>>>> all other clocks save/restore happens as per clock tree traversal.
>>>> Could you please clarify what part of peripherals clocks is required for
>>>> DFLL's restore? Couldn't DFLL driver be changed to avoid that quirkness
>>>> and thus to make DFLL driver suspend/resume the clock?
>>> DFLL source ref_clk and soc_clk need to be restored prior to dfll.
>>>
>>> I see dfllCPU_out parent to CCLK_G first in the clock tree and
>>> dfll_ref and dfll_soc peripheral clocks are not resumed by the time
>>> dfll resume happens first.
>>>
>>> ref_clk and soc_clk source is from pll_p and clock tree has these
>>> registered under pll_p which happens later.
>>>
>>> tegra210_clock_init registers in order plls, peripheral clocks,
>>> super_clk init for cclk_g during clock driver probe and dfll probe and
>>> register happens later.
>>>
>> One more thing, CLDVFS peripheral clock enable is also needed to be
>> enabled to program DFLL Controller and all peripheral clock context is
>> restored only after their PLL sources are restored.
>>
>> DFLL restore involves dfll source clock resume along with CLDVFS
>> periheral clock enable and reset
>>
> I don't quite see why you can't simply add suspend/resume callbacks to
> the CPUFreq driver to:
>
> On suspend:
> 1. Switch CPU to PLLP (or whatever "safe" parent)
> 2. Disable/teardown DFLL
>
> On resume:
> 1. Enable/restore DFLL
> 2. Switch CPU back to DFLL

dfll runtime suspend/resume are already part of dfll_pm_ops. Don't we 
want to use it for suspend/resume as well?

currently no APIs are shared b/w clk/tegra driver and CPUFreq driver to 
invoke dfll suspend/resume in CPUFreq driver


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16  4:37               ` Sowjanya Komatineni
@ 2019-07-16  5:37                 ` Dmitry Osipenko
  2019-07-16  6:20                   ` Dmitry Osipenko
  2019-07-16  6:35                   ` Sowjanya Komatineni
  0 siblings, 2 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-16  5:37 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pdeschrijver, pgaikwad,
	sboyd, linux-clk, linux-gpio, jckuo, josephl, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree

В Mon, 15 Jul 2019 21:37:09 -0700
Sowjanya Komatineni <skomatineni@nvidia.com> пишет:

> On 7/15/19 8:50 PM, Dmitry Osipenko wrote:
> > 16.07.2019 6:00, Sowjanya Komatineni пишет:  
> >> On 7/15/19 5:35 PM, Sowjanya Komatineni wrote:  
> >>> On 7/14/19 2:41 PM, Dmitry Osipenko wrote:  
> >>>> 13.07.2019 8:54, Sowjanya Komatineni пишет:  
> >>>>> On 6/29/19 8:10 AM, Dmitry Osipenko wrote:  
> >>>>>> 28.06.2019 5:12, Sowjanya Komatineni пишет:  
> >>>>>>> This patch adds system suspend and resume support for Tegra210
> >>>>>>> clocks.
> >>>>>>>
> >>>>>>> All the CAR controller settings are lost on suspend when core
> >>>>>>> power goes off.
> >>>>>>>
> >>>>>>> This patch has implementation for saving and restoring all
> >>>>>>> the PLLs and clocks context during system suspend and resume
> >>>>>>> to have the clocks back to same state for normal operation.
> >>>>>>>
> >>>>>>> Acked-by: Thierry Reding <treding@nvidia.com>
> >>>>>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> >>>>>>> ---
> >>>>>>>     drivers/clk/tegra/clk-tegra210.c | 115
> >>>>>>> ++++++++++++++++++++++++++++++++++++++-
> >>>>>>>     drivers/clk/tegra/clk.c          |  14 +++++
> >>>>>>>     drivers/clk/tegra/clk.h          |   1 +
> >>>>>>>     3 files changed, 127 insertions(+), 3 deletions(-)
> >>>>>>>
> >>>>>>> diff --git a/drivers/clk/tegra/clk-tegra210.c
> >>>>>>> b/drivers/clk/tegra/clk-tegra210.c
> >>>>>>> index 1c08c53482a5..1b839544e086 100644
> >>>>>>> --- a/drivers/clk/tegra/clk-tegra210.c
> >>>>>>> +++ b/drivers/clk/tegra/clk-tegra210.c
> >>>>>>> @@ -9,10 +9,12 @@
> >>>>>>>     #include <linux/clkdev.h>
> >>>>>>>     #include <linux/of.h>
> >>>>>>>     #include <linux/of_address.h>
> >>>>>>> +#include <linux/of_platform.h>
> >>>>>>>     #include <linux/delay.h>
> >>>>>>>     #include <linux/export.h>
> >>>>>>>     #include <linux/mutex.h>
> >>>>>>>     #include <linux/clk/tegra.h>
> >>>>>>> +#include <linux/syscore_ops.h>
> >>>>>>>     #include <dt-bindings/clock/tegra210-car.h>
> >>>>>>>     #include <dt-bindings/reset/tegra210-car.h>
> >>>>>>>     #include <linux/iopoll.h>
> >>>>>>> @@ -20,6 +22,7 @@
> >>>>>>>     #include <soc/tegra/pmc.h>
> >>>>>>>       #include "clk.h"
> >>>>>>> +#include "clk-dfll.h"
> >>>>>>>     #include "clk-id.h"
> >>>>>>>       /*
> >>>>>>> @@ -225,6 +228,7 @@
> >>>>>>>       #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
> >>>>>>>     #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
> >>>>>>> +#define CPU_SOFTRST_CTRL 0x380
> >>>>>>>       #define LVL2_CLK_GATE_OVRA 0xf8
> >>>>>>>     #define LVL2_CLK_GATE_OVRC 0x3a0
> >>>>>>> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
> >>>>>>>         struct tegra_clk_pll_freq_table *fentry;
> >>>>>>>         struct tegra_clk_pll pllu;
> >>>>>>>         u32 reg;
> >>>>>>> +    int ret;
> >>>>>>>           for (fentry = pll_u_freq_table; fentry->input_rate;
> >>>>>>> fentry++) {
> >>>>>>>             if (fentry->input_rate == pll_ref_freq)
> >>>>>>> @@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
> >>>>>>>         fence_udelay(1, clk_base);
> >>>>>>>         reg |= PLL_ENABLE;
> >>>>>>>         writel(reg, clk_base + PLLU_BASE);
> >>>>>>> +    fence_udelay(1, clk_base);
> >>>>>>>     -    readl_relaxed_poll_timeout_atomic(clk_base +
> >>>>>>> PLLU_BASE, reg,
> >>>>>>> -                      reg & PLL_BASE_LOCK, 2, 1000);
> >>>>>>> -    if (!(reg & PLL_BASE_LOCK)) {
> >>>>>>> +    ret = tegra210_wait_for_mask(&pllu, PLLU_BASE,
> >>>>>>> PLL_BASE_LOCK);
> >>>>>>> +    if (ret) {
> >>>>>>>             pr_err("Timed out waiting for PLL_U to lock\n");
> >>>>>>>             return -ETIMEDOUT;
> >>>>>>>         }
> >>>>>>> @@ -3283,6 +3288,103 @@ static void
> >>>>>>> tegra210_disable_cpu_clock(u32 cpu)
> >>>>>>>     }
> >>>>>>>       #ifdef CONFIG_PM_SLEEP
> >>>>>>> +static u32 cpu_softrst_ctx[3];
> >>>>>>> +static struct platform_device *dfll_pdev;
> >>>>>>> +#define car_readl(_base, _off) readl_relaxed(clk_base +
> >>>>>>> (_base) + ((_off) * 4))
> >>>>>>> +#define car_writel(_val, _base, _off) \
> >>>>>>> +        writel_relaxed(_val, clk_base + (_base) + ((_off) *
> >>>>>>> 4)) +
> >>>>>>> +static int tegra210_clk_suspend(void)
> >>>>>>> +{
> >>>>>>> +    unsigned int i;
> >>>>>>> +    struct device_node *node;
> >>>>>>> +
> >>>>>>> +    tegra_cclkg_burst_policy_save_context();
> >>>>>>> +
> >>>>>>> +    if (!dfll_pdev) {
> >>>>>>> +        node = of_find_compatible_node(NULL, NULL,
> >>>>>>> +                           "nvidia,tegra210-dfll");
> >>>>>>> +        if (node)
> >>>>>>> +            dfll_pdev = of_find_device_by_node(node);
> >>>>>>> +
> >>>>>>> +        of_node_put(node);
> >>>>>>> +        if (!dfll_pdev)
> >>>>>>> +            pr_err("dfll node not found. no suspend for
> >>>>>>> dfll\n");
> >>>>>>> +    }
> >>>>>>> +
> >>>>>>> +    if (dfll_pdev)
> >>>>>>> +        tegra_dfll_suspend(dfll_pdev);
> >>>>>>> +
> >>>>>>> +    /* Enable PLLP_OUT_CPU after dfll suspend */
> >>>>>>> +    tegra_clk_set_pllp_out_cpu(true);
> >>>>>>> +
> >>>>>>> +    tegra_sclk_cclklp_burst_policy_save_context();
> >>>>>>> +
> >>>>>>> +    clk_save_context();
> >>>>>>> +
> >>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
> >>>>>>> +        cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
> >>>>>>> +
> >>>>>>> +    return 0;
> >>>>>>> +}
> >>>>>>> +
> >>>>>>> +static void tegra210_clk_resume(void)
> >>>>>>> +{
> >>>>>>> +    unsigned int i;
> >>>>>>> +    struct clk_hw *parent;
> >>>>>>> +    struct clk *clk;
> >>>>>>> +
> >>>>>>> +    /*
> >>>>>>> +     * clk_restore_context restores clocks as per the clock
> >>>>>>> tree.
> >>>>>>> +     *
> >>>>>>> +     * dfllCPU_out is first in the clock tree to get
> >>>>>>> restored and it
> >>>>>>> +     * involves programming DFLL controller along with
> >>>>>>> restoring CPUG
> >>>>>>> +     * clock burst policy.
> >>>>>>> +     *
> >>>>>>> +     * DFLL programming needs dfll_ref and dfll_soc
> >>>>>>> peripheral clocks
> >>>>>>> +     * to be restores which are part ofthe peripheral
> >>>>>>> clocks.  
> >>>>                                               ^ white-space
> >>>>
> >>>> Please use spellchecker to avoid typos.
> >>>>  
> >>>>>>> +     * So, peripheral clocks restore should happen prior to
> >>>>>>> dfll clock
> >>>>>>> +     * restore.
> >>>>>>> +     */
> >>>>>>> +
> >>>>>>> +    tegra_clk_osc_resume(clk_base);
> >>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
> >>>>>>> +        car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
> >>>>>>> +
> >>>>>>> +    /* restore all plls and peripheral clocks */
> >>>>>>> +    tegra210_init_pllu();
> >>>>>>> +    clk_restore_context();
> >>>>>>> +
> >>>>>>> +    fence_udelay(5, clk_base);
> >>>>>>> +
> >>>>>>> +    /* resume SCLK and CPULP clocks */
> >>>>>>> +    tegra_sclk_cpulp_burst_policy_restore_context();
> >>>>>>> +
> >>>>>>> +    /*
> >>>>>>> +     * restore CPUG clocks:
> >>>>>>> +     * - enable DFLL in open loop mode
> >>>>>>> +     * - switch CPUG to DFLL clock source
> >>>>>>> +     * - close DFLL loop
> >>>>>>> +     * - sync PLLX state
> >>>>>>> +     */
> >>>>>>> +    if (dfll_pdev)
> >>>>>>> +        tegra_dfll_resume(dfll_pdev, false);
> >>>>>>> +
> >>>>>>> +    tegra_cclkg_burst_policy_restore_context();
> >>>>>>> +    fence_udelay(2, clk_base);
> >>>>>>> +
> >>>>>>> +    if (dfll_pdev)
> >>>>>>> +        tegra_dfll_resume(dfll_pdev, true);
> >>>>>>> +
> >>>>>>> +    parent =
> >>>>>>> clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
> >>>>>>> +    clk = clks[TEGRA210_CLK_PLL_X];
> >>>>>>> +    if (parent != __clk_get_hw(clk))
> >>>>>>> +        tegra_clk_sync_state_pll(__clk_get_hw(clk));
> >>>>>>> +
> >>>>>>> +    /* Disable PLL_OUT_CPU after DFLL resume */
> >>>>>>> +    tegra_clk_set_pllp_out_cpu(false);
> >>>>>>> +}
> >>>>>>> +
> >>>>>>>     static void tegra210_cpu_clock_suspend(void)
> >>>>>>>     {
> >>>>>>>         /* switch coresite to clk_m, save off original source
> >>>>>>> */ @@ -3298,6 +3400,11 @@ static void
> >>>>>>> tegra210_cpu_clock_resume(void) }
> >>>>>>>     #endif
> >>>>>>>     +static struct syscore_ops tegra_clk_syscore_ops = {
> >>>>>>> +    .suspend = tegra210_clk_suspend,
> >>>>>>> +    .resume = tegra210_clk_resume,
> >>>>>>> +};
> >>>>>>> +
> >>>>>>>     static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
> >>>>>>>         .wait_for_reset    = tegra210_wait_cpu_in_reset,
> >>>>>>>         .disable_clock    = tegra210_disable_cpu_clock,
> >>>>>>> @@ -3583,5 +3690,7 @@ static void __init
> >>>>>>> tegra210_clock_init(struct device_node *np)
> >>>>>>>         tegra210_mbist_clk_init();
> >>>>>>>           tegra_cpu_car_ops = &tegra210_cpu_car_ops;
> >>>>>>> +
> >>>>>>> +    register_syscore_ops(&tegra_clk_syscore_ops);
> >>>>>>>     }  
> >>>>>> Is it really worthwhile to use syscore_ops for suspend/resume
> >>>>>> given that drivers for
> >>>>>> won't resume before the CLK driver anyway? Are there any other
> >>>>>> options for CLK
> >>>>>> suspend/resume?
> >>>>>>
> >>>>>> I'm also not sure whether PM runtime API could be used at all
> >>>>>> in the context of
> >>>>>> syscore_ops ..
> >>>>>>
> >>>>>> Secondly, what about to use generic clk_save_context() /
> >>>>>> clk_restore_context()
> >>>>>> helpers for the suspend-resume? It looks to me that some other
> >>>>>> essential (and proper)
> >>>>>> platform driver (soc/tegra/? PMC?) should suspend-resume the
> >>>>>> clocks using the generic
> >>>>>> CLK Framework API.  
> >>>>> Clock resume should happen very early to restore peripheral and
> >>>>> cpu clocks very early than peripheral drivers resume happens.  
> >>>> If all peripheral drivers properly requested all of the
> >>>> necessary clocks and CLK driver was a platform driver, then I
> >>>> guess the probe should have been naturally ordered. But that's
> >>>> not very achievable with the currently available infrastructure
> >>>> in the kernel, so I'm not arguing that the clocks should be
> >>>> explicitly resumed before the users. 
> >>>>> this patch series uses clk_save_context and clk_restore_context
> >>>>> for corresponding divider, pll, pllout.. save and restore
> >>>>> context.  
> >>>> Now I see that indeed this API is utilized in this patch, thank
> >>>> you for the clarification.
> >>>>  
> >>>>> But as there is dependency on dfll resume and cpu and pllx
> >>>>> clocks restore, couldnt use clk_save_context and
> >>>>> clk_restore_context for dfll.
> >>>>>
> >>>>> So implemented recommended dfll resume sequence in main
> >>>>> Tegra210 clock driver along with invoking
> >>>>> clk_save_context/clk_restore_context where all other clocks
> >>>>> save/restore happens as per clock tree traversal.  
> >>>> Could you please clarify what part of peripherals clocks is
> >>>> required for DFLL's restore? Couldn't DFLL driver be changed to
> >>>> avoid that quirkness and thus to make DFLL driver suspend/resume
> >>>> the clock?  
> >>> DFLL source ref_clk and soc_clk need to be restored prior to dfll.
> >>>
> >>> I see dfllCPU_out parent to CCLK_G first in the clock tree and
> >>> dfll_ref and dfll_soc peripheral clocks are not resumed by the
> >>> time dfll resume happens first.
> >>>
> >>> ref_clk and soc_clk source is from pll_p and clock tree has these
> >>> registered under pll_p which happens later.
> >>>
> >>> tegra210_clock_init registers in order plls, peripheral clocks,
> >>> super_clk init for cclk_g during clock driver probe and dfll
> >>> probe and register happens later.
> >>>  
> >> One more thing, CLDVFS peripheral clock enable is also needed to be
> >> enabled to program DFLL Controller and all peripheral clock
> >> context is restored only after their PLL sources are restored.
> >>
> >> DFLL restore involves dfll source clock resume along with CLDVFS
> >> periheral clock enable and reset
> >>  
> > I don't quite see why you can't simply add suspend/resume callbacks
> > to the CPUFreq driver to:
> >
> > On suspend:
> > 1. Switch CPU to PLLP (or whatever "safe" parent)
> > 2. Disable/teardown DFLL
> >
> > On resume:
> > 1. Enable/restore DFLL
> > 2. Switch CPU back to DFLL  
> 
> dfll runtime suspend/resume are already part of dfll_pm_ops. Don't we 
> want to use it for suspend/resume as well?

Looks like no. Seems runtime PM of that driver is intended solely for
the DFLL's clk management.

> currently no APIs are shared b/w clk/tegra driver and CPUFreq driver
> to invoke dfll suspend/resume in CPUFreq driver
> 

Just add it. Also, please note that CPUFreq driver is optional and thus
you may need to switch CPU to a safe parent on clk-core suspend as
well in order to resume properly if CPU was running off unsafe parent
during boot and CPUFreq driver is disabled in kernel build (or failed
to load).

The other thing that also need attention is that T124 CPUFreq driver
implicitly relies on DFLL driver to be probed first, which is icky.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16  5:37                 ` Dmitry Osipenko
@ 2019-07-16  6:20                   ` Dmitry Osipenko
  2019-07-16  6:35                   ` Sowjanya Komatineni
  1 sibling, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-16  6:20 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pdeschrijver, pgaikwad,
	sboyd, linux-clk, linux-gpio, jckuo, josephl, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree

В Tue, 16 Jul 2019 08:37:01 +0300
Dmitry Osipenko <digetx@gmail.com> пишет:

> В Mon, 15 Jul 2019 21:37:09 -0700
> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
> 
> > On 7/15/19 8:50 PM, Dmitry Osipenko wrote:  
> > > 16.07.2019 6:00, Sowjanya Komatineni пишет:    
> > >> On 7/15/19 5:35 PM, Sowjanya Komatineni wrote:    
> > >>> On 7/14/19 2:41 PM, Dmitry Osipenko wrote:    
> > >>>> 13.07.2019 8:54, Sowjanya Komatineni пишет:    
> > >>>>> On 6/29/19 8:10 AM, Dmitry Osipenko wrote:    
> > >>>>>> 28.06.2019 5:12, Sowjanya Komatineni пишет:    
> > >>>>>>> This patch adds system suspend and resume support for
> > >>>>>>> Tegra210 clocks.
> > >>>>>>>
> > >>>>>>> All the CAR controller settings are lost on suspend when
> > >>>>>>> core power goes off.
> > >>>>>>>
> > >>>>>>> This patch has implementation for saving and restoring all
> > >>>>>>> the PLLs and clocks context during system suspend and resume
> > >>>>>>> to have the clocks back to same state for normal operation.
> > >>>>>>>
> > >>>>>>> Acked-by: Thierry Reding <treding@nvidia.com>
> > >>>>>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
> > >>>>>>> ---
> > >>>>>>>     drivers/clk/tegra/clk-tegra210.c | 115
> > >>>>>>> ++++++++++++++++++++++++++++++++++++++-
> > >>>>>>>     drivers/clk/tegra/clk.c          |  14 +++++
> > >>>>>>>     drivers/clk/tegra/clk.h          |   1 +
> > >>>>>>>     3 files changed, 127 insertions(+), 3 deletions(-)
> > >>>>>>>
> > >>>>>>> diff --git a/drivers/clk/tegra/clk-tegra210.c
> > >>>>>>> b/drivers/clk/tegra/clk-tegra210.c
> > >>>>>>> index 1c08c53482a5..1b839544e086 100644
> > >>>>>>> --- a/drivers/clk/tegra/clk-tegra210.c
> > >>>>>>> +++ b/drivers/clk/tegra/clk-tegra210.c
> > >>>>>>> @@ -9,10 +9,12 @@
> > >>>>>>>     #include <linux/clkdev.h>
> > >>>>>>>     #include <linux/of.h>
> > >>>>>>>     #include <linux/of_address.h>
> > >>>>>>> +#include <linux/of_platform.h>
> > >>>>>>>     #include <linux/delay.h>
> > >>>>>>>     #include <linux/export.h>
> > >>>>>>>     #include <linux/mutex.h>
> > >>>>>>>     #include <linux/clk/tegra.h>
> > >>>>>>> +#include <linux/syscore_ops.h>
> > >>>>>>>     #include <dt-bindings/clock/tegra210-car.h>
> > >>>>>>>     #include <dt-bindings/reset/tegra210-car.h>
> > >>>>>>>     #include <linux/iopoll.h>
> > >>>>>>> @@ -20,6 +22,7 @@
> > >>>>>>>     #include <soc/tegra/pmc.h>
> > >>>>>>>       #include "clk.h"
> > >>>>>>> +#include "clk-dfll.h"
> > >>>>>>>     #include "clk-id.h"
> > >>>>>>>       /*
> > >>>>>>> @@ -225,6 +228,7 @@
> > >>>>>>>       #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
> > >>>>>>>     #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
> > >>>>>>> +#define CPU_SOFTRST_CTRL 0x380
> > >>>>>>>       #define LVL2_CLK_GATE_OVRA 0xf8
> > >>>>>>>     #define LVL2_CLK_GATE_OVRC 0x3a0
> > >>>>>>> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
> > >>>>>>>         struct tegra_clk_pll_freq_table *fentry;
> > >>>>>>>         struct tegra_clk_pll pllu;
> > >>>>>>>         u32 reg;
> > >>>>>>> +    int ret;
> > >>>>>>>           for (fentry = pll_u_freq_table;
> > >>>>>>> fentry->input_rate; fentry++) {
> > >>>>>>>             if (fentry->input_rate == pll_ref_freq)
> > >>>>>>> @@ -2847,10 +2852,10 @@ static int
> > >>>>>>> tegra210_enable_pllu(void) fence_udelay(1, clk_base);
> > >>>>>>>         reg |= PLL_ENABLE;
> > >>>>>>>         writel(reg, clk_base + PLLU_BASE);
> > >>>>>>> +    fence_udelay(1, clk_base);
> > >>>>>>>     -    readl_relaxed_poll_timeout_atomic(clk_base +
> > >>>>>>> PLLU_BASE, reg,
> > >>>>>>> -                      reg & PLL_BASE_LOCK, 2, 1000);
> > >>>>>>> -    if (!(reg & PLL_BASE_LOCK)) {
> > >>>>>>> +    ret = tegra210_wait_for_mask(&pllu, PLLU_BASE,
> > >>>>>>> PLL_BASE_LOCK);
> > >>>>>>> +    if (ret) {
> > >>>>>>>             pr_err("Timed out waiting for PLL_U to lock\n");
> > >>>>>>>             return -ETIMEDOUT;
> > >>>>>>>         }
> > >>>>>>> @@ -3283,6 +3288,103 @@ static void
> > >>>>>>> tegra210_disable_cpu_clock(u32 cpu)
> > >>>>>>>     }
> > >>>>>>>       #ifdef CONFIG_PM_SLEEP
> > >>>>>>> +static u32 cpu_softrst_ctx[3];
> > >>>>>>> +static struct platform_device *dfll_pdev;
> > >>>>>>> +#define car_readl(_base, _off) readl_relaxed(clk_base +
> > >>>>>>> (_base) + ((_off) * 4))
> > >>>>>>> +#define car_writel(_val, _base, _off) \
> > >>>>>>> +        writel_relaxed(_val, clk_base + (_base) + ((_off) *
> > >>>>>>> 4)) +
> > >>>>>>> +static int tegra210_clk_suspend(void)
> > >>>>>>> +{
> > >>>>>>> +    unsigned int i;
> > >>>>>>> +    struct device_node *node;
> > >>>>>>> +
> > >>>>>>> +    tegra_cclkg_burst_policy_save_context();
> > >>>>>>> +
> > >>>>>>> +    if (!dfll_pdev) {
> > >>>>>>> +        node = of_find_compatible_node(NULL, NULL,
> > >>>>>>> +                           "nvidia,tegra210-dfll");
> > >>>>>>> +        if (node)
> > >>>>>>> +            dfll_pdev = of_find_device_by_node(node);
> > >>>>>>> +
> > >>>>>>> +        of_node_put(node);
> > >>>>>>> +        if (!dfll_pdev)
> > >>>>>>> +            pr_err("dfll node not found. no suspend for
> > >>>>>>> dfll\n");
> > >>>>>>> +    }
> > >>>>>>> +
> > >>>>>>> +    if (dfll_pdev)
> > >>>>>>> +        tegra_dfll_suspend(dfll_pdev);
> > >>>>>>> +
> > >>>>>>> +    /* Enable PLLP_OUT_CPU after dfll suspend */
> > >>>>>>> +    tegra_clk_set_pllp_out_cpu(true);
> > >>>>>>> +
> > >>>>>>> +    tegra_sclk_cclklp_burst_policy_save_context();
> > >>>>>>> +
> > >>>>>>> +    clk_save_context();
> > >>>>>>> +
> > >>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
> > >>>>>>> +        cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL,
> > >>>>>>> i); +
> > >>>>>>> +    return 0;
> > >>>>>>> +}
> > >>>>>>> +
> > >>>>>>> +static void tegra210_clk_resume(void)
> > >>>>>>> +{
> > >>>>>>> +    unsigned int i;
> > >>>>>>> +    struct clk_hw *parent;
> > >>>>>>> +    struct clk *clk;
> > >>>>>>> +
> > >>>>>>> +    /*
> > >>>>>>> +     * clk_restore_context restores clocks as per the clock
> > >>>>>>> tree.
> > >>>>>>> +     *
> > >>>>>>> +     * dfllCPU_out is first in the clock tree to get
> > >>>>>>> restored and it
> > >>>>>>> +     * involves programming DFLL controller along with
> > >>>>>>> restoring CPUG
> > >>>>>>> +     * clock burst policy.
> > >>>>>>> +     *
> > >>>>>>> +     * DFLL programming needs dfll_ref and dfll_soc
> > >>>>>>> peripheral clocks
> > >>>>>>> +     * to be restores which are part ofthe peripheral
> > >>>>>>> clocks.    
> > >>>>                                               ^ white-space
> > >>>>
> > >>>> Please use spellchecker to avoid typos.
> > >>>>    
> > >>>>>>> +     * So, peripheral clocks restore should happen prior to
> > >>>>>>> dfll clock
> > >>>>>>> +     * restore.
> > >>>>>>> +     */
> > >>>>>>> +
> > >>>>>>> +    tegra_clk_osc_resume(clk_base);
> > >>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
> > >>>>>>> +        car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL,
> > >>>>>>> i); +
> > >>>>>>> +    /* restore all plls and peripheral clocks */
> > >>>>>>> +    tegra210_init_pllu();
> > >>>>>>> +    clk_restore_context();
> > >>>>>>> +
> > >>>>>>> +    fence_udelay(5, clk_base);
> > >>>>>>> +
> > >>>>>>> +    /* resume SCLK and CPULP clocks */
> > >>>>>>> +    tegra_sclk_cpulp_burst_policy_restore_context();
> > >>>>>>> +
> > >>>>>>> +    /*
> > >>>>>>> +     * restore CPUG clocks:
> > >>>>>>> +     * - enable DFLL in open loop mode
> > >>>>>>> +     * - switch CPUG to DFLL clock source
> > >>>>>>> +     * - close DFLL loop
> > >>>>>>> +     * - sync PLLX state
> > >>>>>>> +     */
> > >>>>>>> +    if (dfll_pdev)
> > >>>>>>> +        tegra_dfll_resume(dfll_pdev, false);
> > >>>>>>> +
> > >>>>>>> +    tegra_cclkg_burst_policy_restore_context();
> > >>>>>>> +    fence_udelay(2, clk_base);
> > >>>>>>> +
> > >>>>>>> +    if (dfll_pdev)
> > >>>>>>> +        tegra_dfll_resume(dfll_pdev, true);
> > >>>>>>> +
> > >>>>>>> +    parent =
> > >>>>>>> clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
> > >>>>>>> +    clk = clks[TEGRA210_CLK_PLL_X];
> > >>>>>>> +    if (parent != __clk_get_hw(clk))
> > >>>>>>> +        tegra_clk_sync_state_pll(__clk_get_hw(clk));
> > >>>>>>> +
> > >>>>>>> +    /* Disable PLL_OUT_CPU after DFLL resume */
> > >>>>>>> +    tegra_clk_set_pllp_out_cpu(false);
> > >>>>>>> +}
> > >>>>>>> +
> > >>>>>>>     static void tegra210_cpu_clock_suspend(void)
> > >>>>>>>     {
> > >>>>>>>         /* switch coresite to clk_m, save off original
> > >>>>>>> source */ @@ -3298,6 +3400,11 @@ static void
> > >>>>>>> tegra210_cpu_clock_resume(void) }
> > >>>>>>>     #endif
> > >>>>>>>     +static struct syscore_ops tegra_clk_syscore_ops = {
> > >>>>>>> +    .suspend = tegra210_clk_suspend,
> > >>>>>>> +    .resume = tegra210_clk_resume,
> > >>>>>>> +};
> > >>>>>>> +
> > >>>>>>>     static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
> > >>>>>>>         .wait_for_reset    = tegra210_wait_cpu_in_reset,
> > >>>>>>>         .disable_clock    = tegra210_disable_cpu_clock,
> > >>>>>>> @@ -3583,5 +3690,7 @@ static void __init
> > >>>>>>> tegra210_clock_init(struct device_node *np)
> > >>>>>>>         tegra210_mbist_clk_init();
> > >>>>>>>           tegra_cpu_car_ops = &tegra210_cpu_car_ops;
> > >>>>>>> +
> > >>>>>>> +    register_syscore_ops(&tegra_clk_syscore_ops);
> > >>>>>>>     }    
> > >>>>>> Is it really worthwhile to use syscore_ops for suspend/resume
> > >>>>>> given that drivers for
> > >>>>>> won't resume before the CLK driver anyway? Are there any
> > >>>>>> other options for CLK
> > >>>>>> suspend/resume?
> > >>>>>>
> > >>>>>> I'm also not sure whether PM runtime API could be used at all
> > >>>>>> in the context of
> > >>>>>> syscore_ops ..
> > >>>>>>
> > >>>>>> Secondly, what about to use generic clk_save_context() /
> > >>>>>> clk_restore_context()
> > >>>>>> helpers for the suspend-resume? It looks to me that some
> > >>>>>> other essential (and proper)
> > >>>>>> platform driver (soc/tegra/? PMC?) should suspend-resume the
> > >>>>>> clocks using the generic
> > >>>>>> CLK Framework API.    
> > >>>>> Clock resume should happen very early to restore peripheral
> > >>>>> and cpu clocks very early than peripheral drivers resume
> > >>>>> happens.    
> > >>>> If all peripheral drivers properly requested all of the
> > >>>> necessary clocks and CLK driver was a platform driver, then I
> > >>>> guess the probe should have been naturally ordered. But that's
> > >>>> not very achievable with the currently available infrastructure
> > >>>> in the kernel, so I'm not arguing that the clocks should be
> > >>>> explicitly resumed before the users.   
> > >>>>> this patch series uses clk_save_context and
> > >>>>> clk_restore_context for corresponding divider, pll, pllout..
> > >>>>> save and restore context.    
> > >>>> Now I see that indeed this API is utilized in this patch, thank
> > >>>> you for the clarification.
> > >>>>    
> > >>>>> But as there is dependency on dfll resume and cpu and pllx
> > >>>>> clocks restore, couldnt use clk_save_context and
> > >>>>> clk_restore_context for dfll.
> > >>>>>
> > >>>>> So implemented recommended dfll resume sequence in main
> > >>>>> Tegra210 clock driver along with invoking
> > >>>>> clk_save_context/clk_restore_context where all other clocks
> > >>>>> save/restore happens as per clock tree traversal.    
> > >>>> Could you please clarify what part of peripherals clocks is
> > >>>> required for DFLL's restore? Couldn't DFLL driver be changed to
> > >>>> avoid that quirkness and thus to make DFLL driver
> > >>>> suspend/resume the clock?    
> > >>> DFLL source ref_clk and soc_clk need to be restored prior to
> > >>> dfll.
> > >>>
> > >>> I see dfllCPU_out parent to CCLK_G first in the clock tree and
> > >>> dfll_ref and dfll_soc peripheral clocks are not resumed by the
> > >>> time dfll resume happens first.
> > >>>
> > >>> ref_clk and soc_clk source is from pll_p and clock tree has
> > >>> these registered under pll_p which happens later.
> > >>>
> > >>> tegra210_clock_init registers in order plls, peripheral clocks,
> > >>> super_clk init for cclk_g during clock driver probe and dfll
> > >>> probe and register happens later.
> > >>>    
> > >> One more thing, CLDVFS peripheral clock enable is also needed to
> > >> be enabled to program DFLL Controller and all peripheral clock
> > >> context is restored only after their PLL sources are restored.
> > >>
> > >> DFLL restore involves dfll source clock resume along with CLDVFS
> > >> periheral clock enable and reset
> > >>    
> > > I don't quite see why you can't simply add suspend/resume
> > > callbacks to the CPUFreq driver to:
> > >
> > > On suspend:
> > > 1. Switch CPU to PLLP (or whatever "safe" parent)
> > > 2. Disable/teardown DFLL
> > >
> > > On resume:
> > > 1. Enable/restore DFLL
> > > 2. Switch CPU back to DFLL    
> > 
> > dfll runtime suspend/resume are already part of dfll_pm_ops. Don't
> > we want to use it for suspend/resume as well?  
> 
> Looks like no. Seems runtime PM of that driver is intended solely for
> the DFLL's clk management.
> 
> > currently no APIs are shared b/w clk/tegra driver and CPUFreq driver
> > to invoke dfll suspend/resume in CPUFreq driver
> >   
> 
> Just add it. Also, please note that CPUFreq driver is optional and
> thus you may need to switch CPU to a safe parent on clk-core suspend
> as well in order to resume properly if CPU was running off unsafe
> parent during boot and CPUFreq driver is disabled in kernel build (or
> failed to load).

Although, if PLLs are restored before CCLK, then it should be fine
as-is.

> The other thing that also need attention is that T124 CPUFreq driver
> implicitly relies on DFLL driver to be probed first, which is icky.
> 


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16  5:37                 ` Dmitry Osipenko
  2019-07-16  6:20                   ` Dmitry Osipenko
@ 2019-07-16  6:35                   ` Sowjanya Komatineni
  2019-07-16  7:24                     ` Joseph Lo
  1 sibling, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-16  6:35 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pdeschrijver, pgaikwad,
	sboyd, linux-clk, linux-gpio, jckuo, josephl, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree


On 7/15/19 10:37 PM, Dmitry Osipenko wrote:
> В Mon, 15 Jul 2019 21:37:09 -0700
> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>
>> On 7/15/19 8:50 PM, Dmitry Osipenko wrote:
>>> 16.07.2019 6:00, Sowjanya Komatineni пишет:
>>>> On 7/15/19 5:35 PM, Sowjanya Komatineni wrote:
>>>>> On 7/14/19 2:41 PM, Dmitry Osipenko wrote:
>>>>>> 13.07.2019 8:54, Sowjanya Komatineni пишет:
>>>>>>> On 6/29/19 8:10 AM, Dmitry Osipenko wrote:
>>>>>>>> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>>>>>>>>> This patch adds system suspend and resume support for Tegra210
>>>>>>>>> clocks.
>>>>>>>>>
>>>>>>>>> All the CAR controller settings are lost on suspend when core
>>>>>>>>> power goes off.
>>>>>>>>>
>>>>>>>>> This patch has implementation for saving and restoring all
>>>>>>>>> the PLLs and clocks context during system suspend and resume
>>>>>>>>> to have the clocks back to same state for normal operation.
>>>>>>>>>
>>>>>>>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>>>>>>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>>>>>>>>> ---
>>>>>>>>>      drivers/clk/tegra/clk-tegra210.c | 115
>>>>>>>>> ++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>      drivers/clk/tegra/clk.c          |  14 +++++
>>>>>>>>>      drivers/clk/tegra/clk.h          |   1 +
>>>>>>>>>      3 files changed, 127 insertions(+), 3 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/drivers/clk/tegra/clk-tegra210.c
>>>>>>>>> b/drivers/clk/tegra/clk-tegra210.c
>>>>>>>>> index 1c08c53482a5..1b839544e086 100644
>>>>>>>>> --- a/drivers/clk/tegra/clk-tegra210.c
>>>>>>>>> +++ b/drivers/clk/tegra/clk-tegra210.c
>>>>>>>>> @@ -9,10 +9,12 @@
>>>>>>>>>      #include <linux/clkdev.h>
>>>>>>>>>      #include <linux/of.h>
>>>>>>>>>      #include <linux/of_address.h>
>>>>>>>>> +#include <linux/of_platform.h>
>>>>>>>>>      #include <linux/delay.h>
>>>>>>>>>      #include <linux/export.h>
>>>>>>>>>      #include <linux/mutex.h>
>>>>>>>>>      #include <linux/clk/tegra.h>
>>>>>>>>> +#include <linux/syscore_ops.h>
>>>>>>>>>      #include <dt-bindings/clock/tegra210-car.h>
>>>>>>>>>      #include <dt-bindings/reset/tegra210-car.h>
>>>>>>>>>      #include <linux/iopoll.h>
>>>>>>>>> @@ -20,6 +22,7 @@
>>>>>>>>>      #include <soc/tegra/pmc.h>
>>>>>>>>>        #include "clk.h"
>>>>>>>>> +#include "clk-dfll.h"
>>>>>>>>>      #include "clk-id.h"
>>>>>>>>>        /*
>>>>>>>>> @@ -225,6 +228,7 @@
>>>>>>>>>        #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
>>>>>>>>>      #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
>>>>>>>>> +#define CPU_SOFTRST_CTRL 0x380
>>>>>>>>>        #define LVL2_CLK_GATE_OVRA 0xf8
>>>>>>>>>      #define LVL2_CLK_GATE_OVRC 0x3a0
>>>>>>>>> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
>>>>>>>>>          struct tegra_clk_pll_freq_table *fentry;
>>>>>>>>>          struct tegra_clk_pll pllu;
>>>>>>>>>          u32 reg;
>>>>>>>>> +    int ret;
>>>>>>>>>            for (fentry = pll_u_freq_table; fentry->input_rate;
>>>>>>>>> fentry++) {
>>>>>>>>>              if (fentry->input_rate == pll_ref_freq)
>>>>>>>>> @@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
>>>>>>>>>          fence_udelay(1, clk_base);
>>>>>>>>>          reg |= PLL_ENABLE;
>>>>>>>>>          writel(reg, clk_base + PLLU_BASE);
>>>>>>>>> +    fence_udelay(1, clk_base);
>>>>>>>>>      -    readl_relaxed_poll_timeout_atomic(clk_base +
>>>>>>>>> PLLU_BASE, reg,
>>>>>>>>> -                      reg & PLL_BASE_LOCK, 2, 1000);
>>>>>>>>> -    if (!(reg & PLL_BASE_LOCK)) {
>>>>>>>>> +    ret = tegra210_wait_for_mask(&pllu, PLLU_BASE,
>>>>>>>>> PLL_BASE_LOCK);
>>>>>>>>> +    if (ret) {
>>>>>>>>>              pr_err("Timed out waiting for PLL_U to lock\n");
>>>>>>>>>              return -ETIMEDOUT;
>>>>>>>>>          }
>>>>>>>>> @@ -3283,6 +3288,103 @@ static void
>>>>>>>>> tegra210_disable_cpu_clock(u32 cpu)
>>>>>>>>>      }
>>>>>>>>>        #ifdef CONFIG_PM_SLEEP
>>>>>>>>> +static u32 cpu_softrst_ctx[3];
>>>>>>>>> +static struct platform_device *dfll_pdev;
>>>>>>>>> +#define car_readl(_base, _off) readl_relaxed(clk_base +
>>>>>>>>> (_base) + ((_off) * 4))
>>>>>>>>> +#define car_writel(_val, _base, _off) \
>>>>>>>>> +        writel_relaxed(_val, clk_base + (_base) + ((_off) *
>>>>>>>>> 4)) +
>>>>>>>>> +static int tegra210_clk_suspend(void)
>>>>>>>>> +{
>>>>>>>>> +    unsigned int i;
>>>>>>>>> +    struct device_node *node;
>>>>>>>>> +
>>>>>>>>> +    tegra_cclkg_burst_policy_save_context();
>>>>>>>>> +
>>>>>>>>> +    if (!dfll_pdev) {
>>>>>>>>> +        node = of_find_compatible_node(NULL, NULL,
>>>>>>>>> +                           "nvidia,tegra210-dfll");
>>>>>>>>> +        if (node)
>>>>>>>>> +            dfll_pdev = of_find_device_by_node(node);
>>>>>>>>> +
>>>>>>>>> +        of_node_put(node);
>>>>>>>>> +        if (!dfll_pdev)
>>>>>>>>> +            pr_err("dfll node not found. no suspend for
>>>>>>>>> dfll\n");
>>>>>>>>> +    }
>>>>>>>>> +
>>>>>>>>> +    if (dfll_pdev)
>>>>>>>>> +        tegra_dfll_suspend(dfll_pdev);
>>>>>>>>> +
>>>>>>>>> +    /* Enable PLLP_OUT_CPU after dfll suspend */
>>>>>>>>> +    tegra_clk_set_pllp_out_cpu(true);
>>>>>>>>> +
>>>>>>>>> +    tegra_sclk_cclklp_burst_policy_save_context();
>>>>>>>>> +
>>>>>>>>> +    clk_save_context();
>>>>>>>>> +
>>>>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>>>>>>> +        cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
>>>>>>>>> +
>>>>>>>>> +    return 0;
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>> +static void tegra210_clk_resume(void)
>>>>>>>>> +{
>>>>>>>>> +    unsigned int i;
>>>>>>>>> +    struct clk_hw *parent;
>>>>>>>>> +    struct clk *clk;
>>>>>>>>> +
>>>>>>>>> +    /*
>>>>>>>>> +     * clk_restore_context restores clocks as per the clock
>>>>>>>>> tree.
>>>>>>>>> +     *
>>>>>>>>> +     * dfllCPU_out is first in the clock tree to get
>>>>>>>>> restored and it
>>>>>>>>> +     * involves programming DFLL controller along with
>>>>>>>>> restoring CPUG
>>>>>>>>> +     * clock burst policy.
>>>>>>>>> +     *
>>>>>>>>> +     * DFLL programming needs dfll_ref and dfll_soc
>>>>>>>>> peripheral clocks
>>>>>>>>> +     * to be restores which are part ofthe peripheral
>>>>>>>>> clocks.
>>>>>>                                                ^ white-space
>>>>>>
>>>>>> Please use spellchecker to avoid typos.
>>>>>>   
>>>>>>>>> +     * So, peripheral clocks restore should happen prior to
>>>>>>>>> dfll clock
>>>>>>>>> +     * restore.
>>>>>>>>> +     */
>>>>>>>>> +
>>>>>>>>> +    tegra_clk_osc_resume(clk_base);
>>>>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>>>>>>> +        car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
>>>>>>>>> +
>>>>>>>>> +    /* restore all plls and peripheral clocks */
>>>>>>>>> +    tegra210_init_pllu();
>>>>>>>>> +    clk_restore_context();
>>>>>>>>> +
>>>>>>>>> +    fence_udelay(5, clk_base);
>>>>>>>>> +
>>>>>>>>> +    /* resume SCLK and CPULP clocks */
>>>>>>>>> +    tegra_sclk_cpulp_burst_policy_restore_context();
>>>>>>>>> +
>>>>>>>>> +    /*
>>>>>>>>> +     * restore CPUG clocks:
>>>>>>>>> +     * - enable DFLL in open loop mode
>>>>>>>>> +     * - switch CPUG to DFLL clock source
>>>>>>>>> +     * - close DFLL loop
>>>>>>>>> +     * - sync PLLX state
>>>>>>>>> +     */
>>>>>>>>> +    if (dfll_pdev)
>>>>>>>>> +        tegra_dfll_resume(dfll_pdev, false);
>>>>>>>>> +
>>>>>>>>> +    tegra_cclkg_burst_policy_restore_context();
>>>>>>>>> +    fence_udelay(2, clk_base);
>>>>>>>>> +
>>>>>>>>> +    if (dfll_pdev)
>>>>>>>>> +        tegra_dfll_resume(dfll_pdev, true);
>>>>>>>>> +
>>>>>>>>> +    parent =
>>>>>>>>> clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
>>>>>>>>> +    clk = clks[TEGRA210_CLK_PLL_X];
>>>>>>>>> +    if (parent != __clk_get_hw(clk))
>>>>>>>>> +        tegra_clk_sync_state_pll(__clk_get_hw(clk));
>>>>>>>>> +
>>>>>>>>> +    /* Disable PLL_OUT_CPU after DFLL resume */
>>>>>>>>> +    tegra_clk_set_pllp_out_cpu(false);
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>>      static void tegra210_cpu_clock_suspend(void)
>>>>>>>>>      {
>>>>>>>>>          /* switch coresite to clk_m, save off original source
>>>>>>>>> */ @@ -3298,6 +3400,11 @@ static void
>>>>>>>>> tegra210_cpu_clock_resume(void) }
>>>>>>>>>      #endif
>>>>>>>>>      +static struct syscore_ops tegra_clk_syscore_ops = {
>>>>>>>>> +    .suspend = tegra210_clk_suspend,
>>>>>>>>> +    .resume = tegra210_clk_resume,
>>>>>>>>> +};
>>>>>>>>> +
>>>>>>>>>      static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
>>>>>>>>>          .wait_for_reset    = tegra210_wait_cpu_in_reset,
>>>>>>>>>          .disable_clock    = tegra210_disable_cpu_clock,
>>>>>>>>> @@ -3583,5 +3690,7 @@ static void __init
>>>>>>>>> tegra210_clock_init(struct device_node *np)
>>>>>>>>>          tegra210_mbist_clk_init();
>>>>>>>>>            tegra_cpu_car_ops = &tegra210_cpu_car_ops;
>>>>>>>>> +
>>>>>>>>> +    register_syscore_ops(&tegra_clk_syscore_ops);
>>>>>>>>>      }
>>>>>>>> Is it really worthwhile to use syscore_ops for suspend/resume
>>>>>>>> given that drivers for
>>>>>>>> won't resume before the CLK driver anyway? Are there any other
>>>>>>>> options for CLK
>>>>>>>> suspend/resume?
>>>>>>>>
>>>>>>>> I'm also not sure whether PM runtime API could be used at all
>>>>>>>> in the context of
>>>>>>>> syscore_ops ..
>>>>>>>>
>>>>>>>> Secondly, what about to use generic clk_save_context() /
>>>>>>>> clk_restore_context()
>>>>>>>> helpers for the suspend-resume? It looks to me that some other
>>>>>>>> essential (and proper)
>>>>>>>> platform driver (soc/tegra/? PMC?) should suspend-resume the
>>>>>>>> clocks using the generic
>>>>>>>> CLK Framework API.
>>>>>>> Clock resume should happen very early to restore peripheral and
>>>>>>> cpu clocks very early than peripheral drivers resume happens.
>>>>>> If all peripheral drivers properly requested all of the
>>>>>> necessary clocks and CLK driver was a platform driver, then I
>>>>>> guess the probe should have been naturally ordered. But that's
>>>>>> not very achievable with the currently available infrastructure
>>>>>> in the kernel, so I'm not arguing that the clocks should be
>>>>>> explicitly resumed before the users.
>>>>>>> this patch series uses clk_save_context and clk_restore_context
>>>>>>> for corresponding divider, pll, pllout.. save and restore
>>>>>>> context.
>>>>>> Now I see that indeed this API is utilized in this patch, thank
>>>>>> you for the clarification.
>>>>>>   
>>>>>>> But as there is dependency on dfll resume and cpu and pllx
>>>>>>> clocks restore, couldnt use clk_save_context and
>>>>>>> clk_restore_context for dfll.
>>>>>>>
>>>>>>> So implemented recommended dfll resume sequence in main
>>>>>>> Tegra210 clock driver along with invoking
>>>>>>> clk_save_context/clk_restore_context where all other clocks
>>>>>>> save/restore happens as per clock tree traversal.
>>>>>> Could you please clarify what part of peripherals clocks is
>>>>>> required for DFLL's restore? Couldn't DFLL driver be changed to
>>>>>> avoid that quirkness and thus to make DFLL driver suspend/resume
>>>>>> the clock?
>>>>> DFLL source ref_clk and soc_clk need to be restored prior to dfll.
>>>>>
>>>>> I see dfllCPU_out parent to CCLK_G first in the clock tree and
>>>>> dfll_ref and dfll_soc peripheral clocks are not resumed by the
>>>>> time dfll resume happens first.
>>>>>
>>>>> ref_clk and soc_clk source is from pll_p and clock tree has these
>>>>> registered under pll_p which happens later.
>>>>>
>>>>> tegra210_clock_init registers in order plls, peripheral clocks,
>>>>> super_clk init for cclk_g during clock driver probe and dfll
>>>>> probe and register happens later.
>>>>>   
>>>> One more thing, CLDVFS peripheral clock enable is also needed to be
>>>> enabled to program DFLL Controller and all peripheral clock
>>>> context is restored only after their PLL sources are restored.
>>>>
>>>> DFLL restore involves dfll source clock resume along with CLDVFS
>>>> periheral clock enable and reset
>>>>   
>>> I don't quite see why you can't simply add suspend/resume callbacks
>>> to the CPUFreq driver to:
>>>
>>> On suspend:
>>> 1. Switch CPU to PLLP (or whatever "safe" parent)
>>> 2. Disable/teardown DFLL
>>>
>>> On resume:
>>> 1. Enable/restore DFLL
>>> 2. Switch CPU back to DFLL
>> dfll runtime suspend/resume are already part of dfll_pm_ops. Don't we
>> want to use it for suspend/resume as well?
> Looks like no. Seems runtime PM of that driver is intended solely for
> the DFLL's clk management.
>
>> currently no APIs are shared b/w clk/tegra driver and CPUFreq driver
>> to invoke dfll suspend/resume in CPUFreq driver
>>
> Just add it. Also, please note that CPUFreq driver is optional and thus
> you may need to switch CPU to a safe parent on clk-core suspend as
> well in order to resume properly if CPU was running off unsafe parent
> during boot and CPUFreq driver is disabled in kernel build (or failed
> to load).
OK, Will add to CPUFreq driver...
>
> The other thing that also need attention is that T124 CPUFreq driver
> implicitly relies on DFLL driver to be probed first, which is icky.
>
Should I add check for successful dfll clk register explicitly in 
CPUFreq driver probe and defer till dfll clk registers?

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16  6:35                   ` Sowjanya Komatineni
@ 2019-07-16  7:24                     ` Joseph Lo
  2019-07-16  8:06                       ` Peter De Schrijver
  0 siblings, 1 reply; 111+ messages in thread
From: Joseph Lo @ 2019-07-16  7:24 UTC (permalink / raw)
  To: Sowjanya Komatineni, Dmitry Osipenko
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pdeschrijver, pgaikwad,
	sboyd, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree



On 7/16/19 2:35 PM, Sowjanya Komatineni wrote:
> 
> On 7/15/19 10:37 PM, Dmitry Osipenko wrote:
>> В Mon, 15 Jul 2019 21:37:09 -0700
>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>
>>> On 7/15/19 8:50 PM, Dmitry Osipenko wrote:
>>>> 16.07.2019 6:00, Sowjanya Komatineni пишет:
>>>>> On 7/15/19 5:35 PM, Sowjanya Komatineni wrote:
>>>>>> On 7/14/19 2:41 PM, Dmitry Osipenko wrote:
>>>>>>> 13.07.2019 8:54, Sowjanya Komatineni пишет:
>>>>>>>> On 6/29/19 8:10 AM, Dmitry Osipenko wrote:
>>>>>>>>> 28.06.2019 5:12, Sowjanya Komatineni пишет:
>>>>>>>>>> This patch adds system suspend and resume support for Tegra210
>>>>>>>>>> clocks.
>>>>>>>>>>
>>>>>>>>>> All the CAR controller settings are lost on suspend when core
>>>>>>>>>> power goes off.
>>>>>>>>>>
>>>>>>>>>> This patch has implementation for saving and restoring all
>>>>>>>>>> the PLLs and clocks context during system suspend and resume
>>>>>>>>>> to have the clocks back to same state for normal operation.
>>>>>>>>>>
>>>>>>>>>> Acked-by: Thierry Reding <treding@nvidia.com>
>>>>>>>>>> Signed-off-by: Sowjanya Komatineni <skomatineni@nvidia.com>
>>>>>>>>>> ---
>>>>>>>>>>      drivers/clk/tegra/clk-tegra210.c | 115
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>      drivers/clk/tegra/clk.c          |  14 +++++
>>>>>>>>>>      drivers/clk/tegra/clk.h          |   1 +
>>>>>>>>>>      3 files changed, 127 insertions(+), 3 deletions(-)
>>>>>>>>>>
>>>>>>>>>> diff --git a/drivers/clk/tegra/clk-tegra210.c
>>>>>>>>>> b/drivers/clk/tegra/clk-tegra210.c
>>>>>>>>>> index 1c08c53482a5..1b839544e086 100644
>>>>>>>>>> --- a/drivers/clk/tegra/clk-tegra210.c
>>>>>>>>>> +++ b/drivers/clk/tegra/clk-tegra210.c
>>>>>>>>>> @@ -9,10 +9,12 @@
>>>>>>>>>>      #include <linux/clkdev.h>
>>>>>>>>>>      #include <linux/of.h>
>>>>>>>>>>      #include <linux/of_address.h>
>>>>>>>>>> +#include <linux/of_platform.h>
>>>>>>>>>>      #include <linux/delay.h>
>>>>>>>>>>      #include <linux/export.h>
>>>>>>>>>>      #include <linux/mutex.h>
>>>>>>>>>>      #include <linux/clk/tegra.h>
>>>>>>>>>> +#include <linux/syscore_ops.h>
>>>>>>>>>>      #include <dt-bindings/clock/tegra210-car.h>
>>>>>>>>>>      #include <dt-bindings/reset/tegra210-car.h>
>>>>>>>>>>      #include <linux/iopoll.h>
>>>>>>>>>> @@ -20,6 +22,7 @@
>>>>>>>>>>      #include <soc/tegra/pmc.h>
>>>>>>>>>>        #include "clk.h"
>>>>>>>>>> +#include "clk-dfll.h"
>>>>>>>>>>      #include "clk-id.h"
>>>>>>>>>>        /*
>>>>>>>>>> @@ -225,6 +228,7 @@
>>>>>>>>>>        #define CLK_RST_CONTROLLER_RST_DEV_Y_SET 0x2a8
>>>>>>>>>>      #define CLK_RST_CONTROLLER_RST_DEV_Y_CLR 0x2ac
>>>>>>>>>> +#define CPU_SOFTRST_CTRL 0x380
>>>>>>>>>>        #define LVL2_CLK_GATE_OVRA 0xf8
>>>>>>>>>>      #define LVL2_CLK_GATE_OVRC 0x3a0
>>>>>>>>>> @@ -2820,6 +2824,7 @@ static int tegra210_enable_pllu(void)
>>>>>>>>>>          struct tegra_clk_pll_freq_table *fentry;
>>>>>>>>>>          struct tegra_clk_pll pllu;
>>>>>>>>>>          u32 reg;
>>>>>>>>>> +    int ret;
>>>>>>>>>>            for (fentry = pll_u_freq_table; fentry->input_rate;
>>>>>>>>>> fentry++) {
>>>>>>>>>>              if (fentry->input_rate == pll_ref_freq)
>>>>>>>>>> @@ -2847,10 +2852,10 @@ static int tegra210_enable_pllu(void)
>>>>>>>>>>          fence_udelay(1, clk_base);
>>>>>>>>>>          reg |= PLL_ENABLE;
>>>>>>>>>>          writel(reg, clk_base + PLLU_BASE);
>>>>>>>>>> +    fence_udelay(1, clk_base);
>>>>>>>>>>      -    readl_relaxed_poll_timeout_atomic(clk_base +
>>>>>>>>>> PLLU_BASE, reg,
>>>>>>>>>> -                      reg & PLL_BASE_LOCK, 2, 1000);
>>>>>>>>>> -    if (!(reg & PLL_BASE_LOCK)) {
>>>>>>>>>> +    ret = tegra210_wait_for_mask(&pllu, PLLU_BASE,
>>>>>>>>>> PLL_BASE_LOCK);
>>>>>>>>>> +    if (ret) {
>>>>>>>>>>              pr_err("Timed out waiting for PLL_U to lock\n");
>>>>>>>>>>              return -ETIMEDOUT;
>>>>>>>>>>          }
>>>>>>>>>> @@ -3283,6 +3288,103 @@ static void
>>>>>>>>>> tegra210_disable_cpu_clock(u32 cpu)
>>>>>>>>>>      }
>>>>>>>>>>        #ifdef CONFIG_PM_SLEEP
>>>>>>>>>> +static u32 cpu_softrst_ctx[3];
>>>>>>>>>> +static struct platform_device *dfll_pdev;
>>>>>>>>>> +#define car_readl(_base, _off) readl_relaxed(clk_base +
>>>>>>>>>> (_base) + ((_off) * 4))
>>>>>>>>>> +#define car_writel(_val, _base, _off) \
>>>>>>>>>> +        writel_relaxed(_val, clk_base + (_base) + ((_off) *
>>>>>>>>>> 4)) +
>>>>>>>>>> +static int tegra210_clk_suspend(void)
>>>>>>>>>> +{
>>>>>>>>>> +    unsigned int i;
>>>>>>>>>> +    struct device_node *node;
>>>>>>>>>> +
>>>>>>>>>> +    tegra_cclkg_burst_policy_save_context();
>>>>>>>>>> +
>>>>>>>>>> +    if (!dfll_pdev) {
>>>>>>>>>> +        node = of_find_compatible_node(NULL, NULL,
>>>>>>>>>> +                           "nvidia,tegra210-dfll");
>>>>>>>>>> +        if (node)
>>>>>>>>>> +            dfll_pdev = of_find_device_by_node(node);
>>>>>>>>>> +
>>>>>>>>>> +        of_node_put(node);
>>>>>>>>>> +        if (!dfll_pdev)
>>>>>>>>>> +            pr_err("dfll node not found. no suspend for
>>>>>>>>>> dfll\n");
>>>>>>>>>> +    }
>>>>>>>>>> +
>>>>>>>>>> +    if (dfll_pdev)
>>>>>>>>>> +        tegra_dfll_suspend(dfll_pdev);
>>>>>>>>>> +
>>>>>>>>>> +    /* Enable PLLP_OUT_CPU after dfll suspend */
>>>>>>>>>> +    tegra_clk_set_pllp_out_cpu(true);
>>>>>>>>>> +
>>>>>>>>>> +    tegra_sclk_cclklp_burst_policy_save_context();
>>>>>>>>>> +
>>>>>>>>>> +    clk_save_context();
>>>>>>>>>> +
>>>>>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>>>>>>>> +        cpu_softrst_ctx[i] = car_readl(CPU_SOFTRST_CTRL, i);
>>>>>>>>>> +
>>>>>>>>>> +    return 0;
>>>>>>>>>> +}
>>>>>>>>>> +
>>>>>>>>>> +static void tegra210_clk_resume(void)
>>>>>>>>>> +{
>>>>>>>>>> +    unsigned int i;
>>>>>>>>>> +    struct clk_hw *parent;
>>>>>>>>>> +    struct clk *clk;
>>>>>>>>>> +
>>>>>>>>>> +    /*
>>>>>>>>>> +     * clk_restore_context restores clocks as per the clock
>>>>>>>>>> tree.
>>>>>>>>>> +     *
>>>>>>>>>> +     * dfllCPU_out is first in the clock tree to get
>>>>>>>>>> restored and it
>>>>>>>>>> +     * involves programming DFLL controller along with
>>>>>>>>>> restoring CPUG
>>>>>>>>>> +     * clock burst policy.
>>>>>>>>>> +     *
>>>>>>>>>> +     * DFLL programming needs dfll_ref and dfll_soc
>>>>>>>>>> peripheral clocks
>>>>>>>>>> +     * to be restores which are part ofthe peripheral
>>>>>>>>>> clocks.
>>>>>>>                                                ^ white-space
>>>>>>>
>>>>>>> Please use spellchecker to avoid typos.
>>>>>>>>>> +     * So, peripheral clocks restore should happen prior to
>>>>>>>>>> dfll clock
>>>>>>>>>> +     * restore.
>>>>>>>>>> +     */
>>>>>>>>>> +
>>>>>>>>>> +    tegra_clk_osc_resume(clk_base);
>>>>>>>>>> +    for (i = 0; i < ARRAY_SIZE(cpu_softrst_ctx); i++)
>>>>>>>>>> +        car_writel(cpu_softrst_ctx[i], CPU_SOFTRST_CTRL, i);
>>>>>>>>>> +
>>>>>>>>>> +    /* restore all plls and peripheral clocks */
>>>>>>>>>> +    tegra210_init_pllu();
>>>>>>>>>> +    clk_restore_context();
>>>>>>>>>> +
>>>>>>>>>> +    fence_udelay(5, clk_base);
>>>>>>>>>> +
>>>>>>>>>> +    /* resume SCLK and CPULP clocks */
>>>>>>>>>> +    tegra_sclk_cpulp_burst_policy_restore_context();
>>>>>>>>>> +
>>>>>>>>>> +    /*
>>>>>>>>>> +     * restore CPUG clocks:
>>>>>>>>>> +     * - enable DFLL in open loop mode
>>>>>>>>>> +     * - switch CPUG to DFLL clock source
>>>>>>>>>> +     * - close DFLL loop
>>>>>>>>>> +     * - sync PLLX state
>>>>>>>>>> +     */
>>>>>>>>>> +    if (dfll_pdev)
>>>>>>>>>> +        tegra_dfll_resume(dfll_pdev, false);
>>>>>>>>>> +
>>>>>>>>>> +    tegra_cclkg_burst_policy_restore_context();
>>>>>>>>>> +    fence_udelay(2, clk_base);
>>>>>>>>>> +
>>>>>>>>>> +    if (dfll_pdev)
>>>>>>>>>> +        tegra_dfll_resume(dfll_pdev, true);
>>>>>>>>>> +
>>>>>>>>>> +    parent =
>>>>>>>>>> clk_hw_get_parent(__clk_get_hw(clks[TEGRA210_CLK_CCLK_G]));
>>>>>>>>>> +    clk = clks[TEGRA210_CLK_PLL_X];
>>>>>>>>>> +    if (parent != __clk_get_hw(clk))
>>>>>>>>>> +        tegra_clk_sync_state_pll(__clk_get_hw(clk));
>>>>>>>>>> +
>>>>>>>>>> +    /* Disable PLL_OUT_CPU after DFLL resume */
>>>>>>>>>> +    tegra_clk_set_pllp_out_cpu(false);
>>>>>>>>>> +}
>>>>>>>>>> +
>>>>>>>>>>      static void tegra210_cpu_clock_suspend(void)
>>>>>>>>>>      {
>>>>>>>>>>          /* switch coresite to clk_m, save off original source
>>>>>>>>>> */ @@ -3298,6 +3400,11 @@ static void
>>>>>>>>>> tegra210_cpu_clock_resume(void) }
>>>>>>>>>>      #endif
>>>>>>>>>>      +static struct syscore_ops tegra_clk_syscore_ops = {
>>>>>>>>>> +    .suspend = tegra210_clk_suspend,
>>>>>>>>>> +    .resume = tegra210_clk_resume,
>>>>>>>>>> +};
>>>>>>>>>> +
>>>>>>>>>>      static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
>>>>>>>>>>          .wait_for_reset    = tegra210_wait_cpu_in_reset,
>>>>>>>>>>          .disable_clock    = tegra210_disable_cpu_clock,
>>>>>>>>>> @@ -3583,5 +3690,7 @@ static void __init
>>>>>>>>>> tegra210_clock_init(struct device_node *np)
>>>>>>>>>>          tegra210_mbist_clk_init();
>>>>>>>>>>            tegra_cpu_car_ops = &tegra210_cpu_car_ops;
>>>>>>>>>> +
>>>>>>>>>> +    register_syscore_ops(&tegra_clk_syscore_ops);
>>>>>>>>>>      }
>>>>>>>>> Is it really worthwhile to use syscore_ops for suspend/resume
>>>>>>>>> given that drivers for
>>>>>>>>> won't resume before the CLK driver anyway? Are there any other
>>>>>>>>> options for CLK
>>>>>>>>> suspend/resume?
>>>>>>>>>
>>>>>>>>> I'm also not sure whether PM runtime API could be used at all
>>>>>>>>> in the context of
>>>>>>>>> syscore_ops ..
>>>>>>>>>
>>>>>>>>> Secondly, what about to use generic clk_save_context() /
>>>>>>>>> clk_restore_context()
>>>>>>>>> helpers for the suspend-resume? It looks to me that some other
>>>>>>>>> essential (and proper)
>>>>>>>>> platform driver (soc/tegra/? PMC?) should suspend-resume the
>>>>>>>>> clocks using the generic
>>>>>>>>> CLK Framework API.
>>>>>>>> Clock resume should happen very early to restore peripheral and
>>>>>>>> cpu clocks very early than peripheral drivers resume happens.
>>>>>>> If all peripheral drivers properly requested all of the
>>>>>>> necessary clocks and CLK driver was a platform driver, then I
>>>>>>> guess the probe should have been naturally ordered. But that's
>>>>>>> not very achievable with the currently available infrastructure
>>>>>>> in the kernel, so I'm not arguing that the clocks should be
>>>>>>> explicitly resumed before the users.
>>>>>>>> this patch series uses clk_save_context and clk_restore_context
>>>>>>>> for corresponding divider, pll, pllout.. save and restore
>>>>>>>> context.
>>>>>>> Now I see that indeed this API is utilized in this patch, thank
>>>>>>> you for the clarification.
>>>>>>>> But as there is dependency on dfll resume and cpu and pllx
>>>>>>>> clocks restore, couldnt use clk_save_context and
>>>>>>>> clk_restore_context for dfll.
>>>>>>>>
>>>>>>>> So implemented recommended dfll resume sequence in main
>>>>>>>> Tegra210 clock driver along with invoking
>>>>>>>> clk_save_context/clk_restore_context where all other clocks
>>>>>>>> save/restore happens as per clock tree traversal.
>>>>>>> Could you please clarify what part of peripherals clocks is
>>>>>>> required for DFLL's restore? Couldn't DFLL driver be changed to
>>>>>>> avoid that quirkness and thus to make DFLL driver suspend/resume
>>>>>>> the clock?
>>>>>> DFLL source ref_clk and soc_clk need to be restored prior to dfll.
>>>>>>
>>>>>> I see dfllCPU_out parent to CCLK_G first in the clock tree and
>>>>>> dfll_ref and dfll_soc peripheral clocks are not resumed by the
>>>>>> time dfll resume happens first.
>>>>>>
>>>>>> ref_clk and soc_clk source is from pll_p and clock tree has these
>>>>>> registered under pll_p which happens later.
>>>>>>
>>>>>> tegra210_clock_init registers in order plls, peripheral clocks,
>>>>>> super_clk init for cclk_g during clock driver probe and dfll
>>>>>> probe and register happens later.
>>>>> One more thing, CLDVFS peripheral clock enable is also needed to be
>>>>> enabled to program DFLL Controller and all peripheral clock
>>>>> context is restored only after their PLL sources are restored.
>>>>>
>>>>> DFLL restore involves dfll source clock resume along with CLDVFS
>>>>> periheral clock enable and reset
>>>> I don't quite see why you can't simply add suspend/resume callbacks
>>>> to the CPUFreq driver to:
>>>>
>>>> On suspend:
>>>> 1. Switch CPU to PLLP (or whatever "safe" parent)
>>>> 2. Disable/teardown DFLL
>>>>
>>>> On resume:
>>>> 1. Enable/restore DFLL
>>>> 2. Switch CPU back to DFLL
>>> dfll runtime suspend/resume are already part of dfll_pm_ops. Don't we
>>> want to use it for suspend/resume as well?
>> Looks like no. Seems runtime PM of that driver is intended solely for
>> the DFLL's clk management.
>>
>>> currently no APIs are shared b/w clk/tegra driver and CPUFreq driver
>>> to invoke dfll suspend/resume in CPUFreq driver
>>>
>> Just add it. Also, please note that CPUFreq driver is optional and thus
>> you may need to switch CPU to a safe parent on clk-core suspend as
>> well in order to resume properly if CPU was running off unsafe parent
>> during boot and CPUFreq driver is disabled in kernel build (or failed
>> to load).
> OK, Will add to CPUFreq driver...
>>
>> The other thing that also need attention is that T124 CPUFreq driver
>> implicitly relies on DFLL driver to be probed first, which is icky.
>>
> Should I add check for successful dfll clk register explicitly in 
> CPUFreq driver probe and defer till dfll clk registers?

Sorry, I didn't follow the mail thread. Just regarding the DFLL part.

As you know it, the DFLL clock is one of the CPU clock sources and 
integrated with DVFS control logic with the regulator. We will not 
switch CPU to other clock sources once we switched to DFLL. Because the 
CPU has been regulated by the DFLL HW with the DVFS table (CVB or OPP 
table you see in the driver.). We shouldn't reparent it to other sources 
with unknew freq/volt pair. That's not guaranteed to work. We allow 
switching to open-loop mode but different sources.

And I don't exactly understand why we need to switch to PLLP in CPU idle 
driver. Just keep it on CL-DVFS mode all the time.

In SC7 entry, the dfll suspend function moves it the open-loop mode. 
That's all. The sc7-entryfirmware will handle the rest of the sequence 
to turn off the CPU power.

In SC7 resume, the warmboot code will handle the sequence to turn on 
regulator and power up the CPU cluster. And leave it on PLL_P. After 
resuming to the kernel, we re-init DFLL, restore the CPU clock policy 
(CPU runs on DFLL open-loop mode) and then moving to close-loop mode.

The DFLL part looks good to me. BTW, change the patch subject to "Add 
suspend-resume support" seems more appropriate to me.

Thanks.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16  7:24                     ` Joseph Lo
@ 2019-07-16  8:06                       ` Peter De Schrijver
  2019-07-16 15:00                         ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Peter De Schrijver @ 2019-07-16  8:06 UTC (permalink / raw)
  To: Joseph Lo
  Cc: Sowjanya Komatineni, Dmitry Osipenko, thierry.reding, jonathanh,
	tglx, jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree

On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
> > OK, Will add to CPUFreq driver...
> > > 
> > > The other thing that also need attention is that T124 CPUFreq driver
> > > implicitly relies on DFLL driver to be probed first, which is icky.
> > > 
> > Should I add check for successful dfll clk register explicitly in
> > CPUFreq driver probe and defer till dfll clk registers?
> 
> Sorry, I didn't follow the mail thread. Just regarding the DFLL part.
> 
> As you know it, the DFLL clock is one of the CPU clock sources and
> integrated with DVFS control logic with the regulator. We will not switch
> CPU to other clock sources once we switched to DFLL. Because the CPU has
> been regulated by the DFLL HW with the DVFS table (CVB or OPP table you see
> in the driver.). We shouldn't reparent it to other sources with unknew
> freq/volt pair. That's not guaranteed to work. We allow switching to
> open-loop mode but different sources.
> 
> And I don't exactly understand why we need to switch to PLLP in CPU idle
> driver. Just keep it on CL-DVFS mode all the time.
> 
> In SC7 entry, the dfll suspend function moves it the open-loop mode. That's
> all. The sc7-entryfirmware will handle the rest of the sequence to turn off
> the CPU power.
> 
> In SC7 resume, the warmboot code will handle the sequence to turn on
> regulator and power up the CPU cluster. And leave it on PLL_P. After
> resuming to the kernel, we re-init DFLL, restore the CPU clock policy (CPU
> runs on DFLL open-loop mode) and then moving to close-loop mode.
> 
> The DFLL part looks good to me. BTW, change the patch subject to "Add
> suspend-resume support" seems more appropriate to me.
> 

To clarify this, the sequences for DFLL use are as follows (assuming all
required DFLL hw configuration has been done)

Switch to DFLL:
0) Save current parent and frequency
1) Program DFLL to open loop mode
2) Enable DFLL
3) Change cclk_g parent to DFLL
For OVR regulator:
4) Change PWM output pin from tristate to output
5) Enable DFLL PWM output
For I2C regulator:
4) Enable DFLL I2C output
6) Program DFLL to closed loop mode

Switch away from DFLL:
0) Change cclk_g parent to PLLP so the CPU frequency is ok for any vdd_cpu voltage
1) Program DFLL to open loop mode

For OVR regulator:
2) Change PWM output pin from output to tristate: vdd_cpu will go back
   to hardwired boot voltage.
3) Disable DFLL PWM output

For I2C regulator:
2) Program vdd_cpu regulator voltage to the boot voltage
3) Disable DFLL I2C output

4) Reprogram parent saved in step 0 of 'Switch to DFLL' to the saved
   frequency
5) Change cclk_g parent to saved parent
6) Disable DFLL

Peter.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16  8:06                       ` Peter De Schrijver
@ 2019-07-16 15:00                         ` Dmitry Osipenko
  2019-07-16 16:50                           ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-16 15:00 UTC (permalink / raw)
  To: Peter De Schrijver, Joseph Lo, Sowjanya Komatineni
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree

16.07.2019 11:06, Peter De Schrijver пишет:
> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>> OK, Will add to CPUFreq driver...
>>>>
>>>> The other thing that also need attention is that T124 CPUFreq driver
>>>> implicitly relies on DFLL driver to be probed first, which is icky.
>>>>
>>> Should I add check for successful dfll clk register explicitly in
>>> CPUFreq driver probe and defer till dfll clk registers?

Probably you should use the "device links". See [1][2] for the example.

[1]
https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383

[2] https://www.kernel.org/doc/html/latest/driver-api/device_link.html

Return EPROBE_DEFER instead of EINVAL if device_link_add() fails. And
use of_find_device_by_node() to get the DFLL's device, see [3].

[3]
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100

>> Sorry, I didn't follow the mail thread. Just regarding the DFLL part.
>>
>> As you know it, the DFLL clock is one of the CPU clock sources and
>> integrated with DVFS control logic with the regulator. We will not switch
>> CPU to other clock sources once we switched to DFLL. Because the CPU has
>> been regulated by the DFLL HW with the DVFS table (CVB or OPP table you see
>> in the driver.). We shouldn't reparent it to other sources with unknew
>> freq/volt pair. That's not guaranteed to work. We allow switching to
>> open-loop mode but different sources.

Okay, then the CPUFreq driver will have to enforce DFLL freq to PLLP's
rate before switching to PLLP in order to have a proper CPU voltage.

>> And I don't exactly understand why we need to switch to PLLP in CPU idle
>> driver. Just keep it on CL-DVFS mode all the time.
>>
>> In SC7 entry, the dfll suspend function moves it the open-loop mode. That's
>> all. The sc7-entryfirmware will handle the rest of the sequence to turn off
>> the CPU power.
>>
>> In SC7 resume, the warmboot code will handle the sequence to turn on
>> regulator and power up the CPU cluster. And leave it on PLL_P. After
>> resuming to the kernel, we re-init DFLL, restore the CPU clock policy (CPU
>> runs on DFLL open-loop mode) and then moving to close-loop mode.

The DFLL is re-inited after switching CCLK to DFLL parent during of the
early clocks-state restoring by CaR driver. Hence instead of having odd
hacks in the CaR driver, it is much nicer to have a proper
suspend-resume sequencing of the device drivers. In this case CPUFreq
driver is the driver that enables DFLL and switches CPU to that clock
source, which means that this driver is also should be responsible for
management of the DFLL's state during of suspend/resume process. If
CPUFreq driver disables DFLL during suspend and re-enables it during
resume, then looks like the CaR driver hacks around DFLL are not needed.

>> The DFLL part looks good to me. BTW, change the patch subject to "Add
>> suspend-resume support" seems more appropriate to me.
>>
> 
> To clarify this, the sequences for DFLL use are as follows (assuming all
> required DFLL hw configuration has been done)
> 
> Switch to DFLL:
> 0) Save current parent and frequency
> 1) Program DFLL to open loop mode
> 2) Enable DFLL
> 3) Change cclk_g parent to DFLL
> For OVR regulator:
> 4) Change PWM output pin from tristate to output
> 5) Enable DFLL PWM output
> For I2C regulator:
> 4) Enable DFLL I2C output
> 6) Program DFLL to closed loop mode
> 
> Switch away from DFLL:
> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for any vdd_cpu voltage
> 1) Program DFLL to open loop mode
> 
> For OVR regulator:
> 2) Change PWM output pin from output to tristate: vdd_cpu will go back
>    to hardwired boot voltage.
> 3) Disable DFLL PWM output
> 
> For I2C regulator:
> 2) Program vdd_cpu regulator voltage to the boot voltage
> 3) Disable DFLL I2C output
> 
> 4) Reprogram parent saved in step 0 of 'Switch to DFLL' to the saved
>    frequency
> 5) Change cclk_g parent to saved parent
> 6) Disable DFLL

Thanks!

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 15:00                         ` Dmitry Osipenko
@ 2019-07-16 16:50                           ` Sowjanya Komatineni
  2019-07-16 18:19                             ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-16 16:50 UTC (permalink / raw)
  To: Dmitry Osipenko, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree


On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
> 16.07.2019 11:06, Peter De Schrijver пишет:
>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>> OK, Will add to CPUFreq driver...
>>>>> The other thing that also need attention is that T124 CPUFreq driver
>>>>> implicitly relies on DFLL driver to be probed first, which is icky.
>>>>>
>>>> Should I add check for successful dfll clk register explicitly in
>>>> CPUFreq driver probe and defer till dfll clk registers?
> Probably you should use the "device links". See [1][2] for the example.
>
> [1]
> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>
> [2] https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>
> Return EPROBE_DEFER instead of EINVAL if device_link_add() fails. And
> use of_find_device_by_node() to get the DFLL's device, see [3].
>
> [3]
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
Will go thru and add...
>>> Sorry, I didn't follow the mail thread. Just regarding the DFLL part.
>>>
>>> As you know it, the DFLL clock is one of the CPU clock sources and
>>> integrated with DVFS control logic with the regulator. We will not switch
>>> CPU to other clock sources once we switched to DFLL. Because the CPU has
>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP table you see
>>> in the driver.). We shouldn't reparent it to other sources with unknew
>>> freq/volt pair. That's not guaranteed to work. We allow switching to
>>> open-loop mode but different sources.
> Okay, then the CPUFreq driver will have to enforce DFLL freq to PLLP's
> rate before switching to PLLP in order to have a proper CPU voltage.

PLLP freq is safe to work for any CPU voltage. So no need to enforce 
DFLL freq to PLLP rate before changing CCLK_G source to PLLP during suspend

>>> And I don't exactly understand why we need to switch to PLLP in CPU idle
>>> driver. Just keep it on CL-DVFS mode all the time.
>>>
>>> In SC7 entry, the dfll suspend function moves it the open-loop mode. That's
>>> all. The sc7-entryfirmware will handle the rest of the sequence to turn off
>>> the CPU power.
>>>
>>> In SC7 resume, the warmboot code will handle the sequence to turn on
>>> regulator and power up the CPU cluster. And leave it on PLL_P. After
>>> resuming to the kernel, we re-init DFLL, restore the CPU clock policy (CPU
>>> runs on DFLL open-loop mode) and then moving to close-loop mode.
> The DFLL is re-inited after switching CCLK to DFLL parent during of the
> early clocks-state restoring by CaR driver. Hence instead of having odd
> hacks in the CaR driver, it is much nicer to have a proper
> suspend-resume sequencing of the device drivers. In this case CPUFreq
> driver is the driver that enables DFLL and switches CPU to that clock
> source, which means that this driver is also should be responsible for
> management of the DFLL's state during of suspend/resume process. If
> CPUFreq driver disables DFLL during suspend and re-enables it during
> resume, then looks like the CaR driver hacks around DFLL are not needed.
>
>>> The DFLL part looks good to me. BTW, change the patch subject to "Add
>>> suspend-resume support" seems more appropriate to me.
>>>
>> To clarify this, the sequences for DFLL use are as follows (assuming all
>> required DFLL hw configuration has been done)
>>
>> Switch to DFLL:
>> 0) Save current parent and frequency
>> 1) Program DFLL to open loop mode
>> 2) Enable DFLL
>> 3) Change cclk_g parent to DFLL
>> For OVR regulator:
>> 4) Change PWM output pin from tristate to output
>> 5) Enable DFLL PWM output
>> For I2C regulator:
>> 4) Enable DFLL I2C output
>> 6) Program DFLL to closed loop mode
>>
>> Switch away from DFLL:
>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for any vdd_cpu voltage
>> 1) Program DFLL to open loop mode
>>
>> For OVR regulator:
>> 2) Change PWM output pin from output to tristate: vdd_cpu will go back
>>     to hardwired boot voltage.
>> 3) Disable DFLL PWM output
>>
>> For I2C regulator:
>> 2) Program vdd_cpu regulator voltage to the boot voltage
>> 3) Disable DFLL I2C output
>>
>> 4) Reprogram parent saved in step 0 of 'Switch to DFLL' to the saved
>>     frequency
>> 5) Change cclk_g parent to saved parent
>> 6) Disable DFLL

This is the same sequence currently implemented. But dfll suspend/resume 
calls are thru Tegra210 clock driver.

Dmitry wants to have dfll suspend/resume along with CCLK_G restore to 
happen from CPUFreq driver pm_ops rather than tegra210 clock driver or 
tegra dfll driver.

Will move it to CPUFreq driver...

> Thanks!

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 16:50                           ` Sowjanya Komatineni
@ 2019-07-16 18:19                             ` Sowjanya Komatineni
  2019-07-16 18:25                               ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-16 18:19 UTC (permalink / raw)
  To: Dmitry Osipenko, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree


On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>
> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>> OK, Will add to CPUFreq driver...
>>>>>> The other thing that also need attention is that T124 CPUFreq driver
>>>>>> implicitly relies on DFLL driver to be probed first, which is icky.
>>>>>>
>>>>> Should I add check for successful dfll clk register explicitly in
>>>>> CPUFreq driver probe and defer till dfll clk registers?
>> Probably you should use the "device links". See [1][2] for the example.
>>
>> [1]
>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383 
>>
>>
>> [2] https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>
>> Return EPROBE_DEFER instead of EINVAL if device_link_add() fails. And
>> use of_find_device_by_node() to get the DFLL's device, see [3].
>>
>> [3]
>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100 
>>
> Will go thru and add...
>>>> Sorry, I didn't follow the mail thread. Just regarding the DFLL part.
>>>>
>>>> As you know it, the DFLL clock is one of the CPU clock sources and
>>>> integrated with DVFS control logic with the regulator. We will not 
>>>> switch
>>>> CPU to other clock sources once we switched to DFLL. Because the 
>>>> CPU has
>>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP table 
>>>> you see
>>>> in the driver.). We shouldn't reparent it to other sources with unknew
>>>> freq/volt pair. That's not guaranteed to work. We allow switching to
>>>> open-loop mode but different sources.
>> Okay, then the CPUFreq driver will have to enforce DFLL freq to PLLP's
>> rate before switching to PLLP in order to have a proper CPU voltage.
>
> PLLP freq is safe to work for any CPU voltage. So no need to enforce 
> DFLL freq to PLLP rate before changing CCLK_G source to PLLP during 
> suspend
>
Sorry, please ignore my above comment. During suspend, need to change 
CCLK_G source to PLLP when dfll is in closed loop mode first and then 
dfll need to be set to open loop.
>>>> And I don't exactly understand why we need to switch to PLLP in CPU 
>>>> idle
>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>
>>>> In SC7 entry, the dfll suspend function moves it the open-loop 
>>>> mode. That's
>>>> all. The sc7-entryfirmware will handle the rest of the sequence to 
>>>> turn off
>>>> the CPU power.
>>>>
>>>> In SC7 resume, the warmboot code will handle the sequence to turn on
>>>> regulator and power up the CPU cluster. And leave it on PLL_P. After
>>>> resuming to the kernel, we re-init DFLL, restore the CPU clock 
>>>> policy (CPU
>>>> runs on DFLL open-loop mode) and then moving to close-loop mode.
>> The DFLL is re-inited after switching CCLK to DFLL parent during of the
>> early clocks-state restoring by CaR driver. Hence instead of having odd
>> hacks in the CaR driver, it is much nicer to have a proper
>> suspend-resume sequencing of the device drivers. In this case CPUFreq
>> driver is the driver that enables DFLL and switches CPU to that clock
>> source, which means that this driver is also should be responsible for
>> management of the DFLL's state during of suspend/resume process. If
>> CPUFreq driver disables DFLL during suspend and re-enables it during
>> resume, then looks like the CaR driver hacks around DFLL are not needed.
>>
>>>> The DFLL part looks good to me. BTW, change the patch subject to "Add
>>>> suspend-resume support" seems more appropriate to me.
>>>>
>>> To clarify this, the sequences for DFLL use are as follows (assuming 
>>> all
>>> required DFLL hw configuration has been done)
>>>
>>> Switch to DFLL:
>>> 0) Save current parent and frequency
>>> 1) Program DFLL to open loop mode
>>> 2) Enable DFLL
>>> 3) Change cclk_g parent to DFLL
>>> For OVR regulator:
>>> 4) Change PWM output pin from tristate to output
>>> 5) Enable DFLL PWM output
>>> For I2C regulator:
>>> 4) Enable DFLL I2C output
>>> 6) Program DFLL to closed loop mode
>>>
>>> Switch away from DFLL:
>>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for any 
>>> vdd_cpu voltage
>>> 1) Program DFLL to open loop mode
>>>
I see during switch away from DFLL (suspend), cclk_g parent is not 
changed to PLLP before changing dfll to open loop mode.

Will add this ...

>>> For OVR regulator:
>>> 2) Change PWM output pin from output to tristate: vdd_cpu will go back
>>>     to hardwired boot voltage.
>>> 3) Disable DFLL PWM output
>>>
>>> For I2C regulator:
>>> 2) Program vdd_cpu regulator voltage to the boot voltage
>>> 3) Disable DFLL I2C output
>>>
>>> 4) Reprogram parent saved in step 0 of 'Switch to DFLL' to the saved
>>>     frequency
>>> 5) Change cclk_g parent to saved parent
>>> 6) Disable DFLL
>
> This is the same sequence currently implemented. But dfll 
> suspend/resume calls are thru Tegra210 clock driver.
>
> Dmitry wants to have dfll suspend/resume along with CCLK_G restore to 
> happen from CPUFreq driver pm_ops rather than tegra210 clock driver or 
> tegra dfll driver.
>
> Will move it to CPUFreq driver...
>
Thanks!

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 18:19                             ` Sowjanya Komatineni
@ 2019-07-16 18:25                               ` Dmitry Osipenko
  2019-07-16 18:30                                 ` Sowjanya Komatineni
  2019-07-18 19:15                                 ` Peter De Schrijver
  0 siblings, 2 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-16 18:25 UTC (permalink / raw)
  To: Sowjanya Komatineni, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree

16.07.2019 21:19, Sowjanya Komatineni пишет:
> 
> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>
>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>>> OK, Will add to CPUFreq driver...
>>>>>>> The other thing that also need attention is that T124 CPUFreq driver
>>>>>>> implicitly relies on DFLL driver to be probed first, which is icky.
>>>>>>>
>>>>>> Should I add check for successful dfll clk register explicitly in
>>>>>> CPUFreq driver probe and defer till dfll clk registers?
>>> Probably you should use the "device links". See [1][2] for the example.
>>>
>>> [1]
>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>
>>>
>>> [2] https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>
>>> Return EPROBE_DEFER instead of EINVAL if device_link_add() fails. And
>>> use of_find_device_by_node() to get the DFLL's device, see [3].
>>>
>>> [3]
>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>
>> Will go thru and add...

Looks like I initially confused this case with getting orphaned clock.
I'm now seeing that the DFLL driver registers the clock and then
clk_get(dfll) should be returning EPROBE_DEFER until DFLL driver is
probed, hence everything should be fine as-is and there is no real need
for the 'device link'. Sorry for the confusion!

>>>>> Sorry, I didn't follow the mail thread. Just regarding the DFLL part.
>>>>>
>>>>> As you know it, the DFLL clock is one of the CPU clock sources and
>>>>> integrated with DVFS control logic with the regulator. We will not
>>>>> switch
>>>>> CPU to other clock sources once we switched to DFLL. Because the
>>>>> CPU has
>>>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP table
>>>>> you see
>>>>> in the driver.). We shouldn't reparent it to other sources with unknew
>>>>> freq/volt pair. That's not guaranteed to work. We allow switching to
>>>>> open-loop mode but different sources.
>>> Okay, then the CPUFreq driver will have to enforce DFLL freq to PLLP's
>>> rate before switching to PLLP in order to have a proper CPU voltage.
>>
>> PLLP freq is safe to work for any CPU voltage. So no need to enforce
>> DFLL freq to PLLP rate before changing CCLK_G source to PLLP during
>> suspend
>>
> Sorry, please ignore my above comment. During suspend, need to change
> CCLK_G source to PLLP when dfll is in closed loop mode first and then
> dfll need to be set to open loop.

Okay.

>>>>> And I don't exactly understand why we need to switch to PLLP in CPU
>>>>> idle
>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>
>>>>> In SC7 entry, the dfll suspend function moves it the open-loop
>>>>> mode. That's
>>>>> all. The sc7-entryfirmware will handle the rest of the sequence to
>>>>> turn off
>>>>> the CPU power.
>>>>>
>>>>> In SC7 resume, the warmboot code will handle the sequence to turn on
>>>>> regulator and power up the CPU cluster. And leave it on PLL_P. After
>>>>> resuming to the kernel, we re-init DFLL, restore the CPU clock
>>>>> policy (CPU
>>>>> runs on DFLL open-loop mode) and then moving to close-loop mode.
>>> The DFLL is re-inited after switching CCLK to DFLL parent during of the
>>> early clocks-state restoring by CaR driver. Hence instead of having odd
>>> hacks in the CaR driver, it is much nicer to have a proper
>>> suspend-resume sequencing of the device drivers. In this case CPUFreq
>>> driver is the driver that enables DFLL and switches CPU to that clock
>>> source, which means that this driver is also should be responsible for
>>> management of the DFLL's state during of suspend/resume process. If
>>> CPUFreq driver disables DFLL during suspend and re-enables it during
>>> resume, then looks like the CaR driver hacks around DFLL are not needed.
>>>
>>>>> The DFLL part looks good to me. BTW, change the patch subject to "Add
>>>>> suspend-resume support" seems more appropriate to me.
>>>>>
>>>> To clarify this, the sequences for DFLL use are as follows (assuming
>>>> all
>>>> required DFLL hw configuration has been done)
>>>>
>>>> Switch to DFLL:
>>>> 0) Save current parent and frequency
>>>> 1) Program DFLL to open loop mode
>>>> 2) Enable DFLL
>>>> 3) Change cclk_g parent to DFLL
>>>> For OVR regulator:
>>>> 4) Change PWM output pin from tristate to output
>>>> 5) Enable DFLL PWM output
>>>> For I2C regulator:
>>>> 4) Enable DFLL I2C output
>>>> 6) Program DFLL to closed loop mode
>>>>
>>>> Switch away from DFLL:
>>>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for any
>>>> vdd_cpu voltage
>>>> 1) Program DFLL to open loop mode
>>>>
> I see during switch away from DFLL (suspend), cclk_g parent is not
> changed to PLLP before changing dfll to open loop mode.
> 
> Will add this ...

The CPUFreq driver switches parent to PLLP during the probe, similar
should be done on suspend.

I'm also wondering if it's always safe to switch to PLLP in the probe.
If CPU is running on a lower freq than PLLP, then some other more
appropriate intermediate parent should be selected.

>>>> For OVR regulator:
>>>> 2) Change PWM output pin from output to tristate: vdd_cpu will go back
>>>>     to hardwired boot voltage.
>>>> 3) Disable DFLL PWM output
>>>>
>>>> For I2C regulator:
>>>> 2) Program vdd_cpu regulator voltage to the boot voltage
>>>> 3) Disable DFLL I2C output
>>>>
>>>> 4) Reprogram parent saved in step 0 of 'Switch to DFLL' to the saved
>>>>     frequency
>>>> 5) Change cclk_g parent to saved parent
>>>> 6) Disable DFLL
>>
>> This is the same sequence currently implemented. But dfll
>> suspend/resume calls are thru Tegra210 clock driver.
>>
>> Dmitry wants to have dfll suspend/resume along with CCLK_G restore to
>> happen from CPUFreq driver pm_ops rather than tegra210 clock driver or
>> tegra dfll driver.
>>
>> Will move it to CPUFreq driver...
>>
> Thanks!

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 18:25                               ` Dmitry Osipenko
@ 2019-07-16 18:30                                 ` Sowjanya Komatineni
  2019-07-16 18:43                                   ` Dmitry Osipenko
  2019-07-18 19:15                                 ` Peter De Schrijver
  1 sibling, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-16 18:30 UTC (permalink / raw)
  To: Dmitry Osipenko, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree


On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>> The other thing that also need attention is that T124 CPUFreq driver
>>>>>>>> implicitly relies on DFLL driver to be probed first, which is icky.
>>>>>>>>
>>>>>>> Should I add check for successful dfll clk register explicitly in
>>>>>>> CPUFreq driver probe and defer till dfll clk registers?
>>>> Probably you should use the "device links". See [1][2] for the example.
>>>>
>>>> [1]
>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>
>>>>
>>>> [2] https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>
>>>> Return EPROBE_DEFER instead of EINVAL if device_link_add() fails. And
>>>> use of_find_device_by_node() to get the DFLL's device, see [3].
>>>>
>>>> [3]
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>
>>> Will go thru and add...
> Looks like I initially confused this case with getting orphaned clock.
> I'm now seeing that the DFLL driver registers the clock and then
> clk_get(dfll) should be returning EPROBE_DEFER until DFLL driver is
> probed, hence everything should be fine as-is and there is no real need
> for the 'device link'. Sorry for the confusion!
>
>>>>>> Sorry, I didn't follow the mail thread. Just regarding the DFLL part.
>>>>>>
>>>>>> As you know it, the DFLL clock is one of the CPU clock sources and
>>>>>> integrated with DVFS control logic with the regulator. We will not
>>>>>> switch
>>>>>> CPU to other clock sources once we switched to DFLL. Because the
>>>>>> CPU has
>>>>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP table
>>>>>> you see
>>>>>> in the driver.). We shouldn't reparent it to other sources with unknew
>>>>>> freq/volt pair. That's not guaranteed to work. We allow switching to
>>>>>> open-loop mode but different sources.
>>>> Okay, then the CPUFreq driver will have to enforce DFLL freq to PLLP's
>>>> rate before switching to PLLP in order to have a proper CPU voltage.
>>> PLLP freq is safe to work for any CPU voltage. So no need to enforce
>>> DFLL freq to PLLP rate before changing CCLK_G source to PLLP during
>>> suspend
>>>
>> Sorry, please ignore my above comment. During suspend, need to change
>> CCLK_G source to PLLP when dfll is in closed loop mode first and then
>> dfll need to be set to open loop.
> Okay.
>
>>>>>> And I don't exactly understand why we need to switch to PLLP in CPU
>>>>>> idle
>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>
>>>>>> In SC7 entry, the dfll suspend function moves it the open-loop
>>>>>> mode. That's
>>>>>> all. The sc7-entryfirmware will handle the rest of the sequence to
>>>>>> turn off
>>>>>> the CPU power.
>>>>>>
>>>>>> In SC7 resume, the warmboot code will handle the sequence to turn on
>>>>>> regulator and power up the CPU cluster. And leave it on PLL_P. After
>>>>>> resuming to the kernel, we re-init DFLL, restore the CPU clock
>>>>>> policy (CPU
>>>>>> runs on DFLL open-loop mode) and then moving to close-loop mode.
>>>> The DFLL is re-inited after switching CCLK to DFLL parent during of the
>>>> early clocks-state restoring by CaR driver. Hence instead of having odd
>>>> hacks in the CaR driver, it is much nicer to have a proper
>>>> suspend-resume sequencing of the device drivers. In this case CPUFreq
>>>> driver is the driver that enables DFLL and switches CPU to that clock
>>>> source, which means that this driver is also should be responsible for
>>>> management of the DFLL's state during of suspend/resume process. If
>>>> CPUFreq driver disables DFLL during suspend and re-enables it during
>>>> resume, then looks like the CaR driver hacks around DFLL are not needed.
>>>>
>>>>>> The DFLL part looks good to me. BTW, change the patch subject to "Add
>>>>>> suspend-resume support" seems more appropriate to me.
>>>>>>
>>>>> To clarify this, the sequences for DFLL use are as follows (assuming
>>>>> all
>>>>> required DFLL hw configuration has been done)
>>>>>
>>>>> Switch to DFLL:
>>>>> 0) Save current parent and frequency
>>>>> 1) Program DFLL to open loop mode
>>>>> 2) Enable DFLL
>>>>> 3) Change cclk_g parent to DFLL
>>>>> For OVR regulator:
>>>>> 4) Change PWM output pin from tristate to output
>>>>> 5) Enable DFLL PWM output
>>>>> For I2C regulator:
>>>>> 4) Enable DFLL I2C output
>>>>> 6) Program DFLL to closed loop mode
>>>>>
>>>>> Switch away from DFLL:
>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for any
>>>>> vdd_cpu voltage
>>>>> 1) Program DFLL to open loop mode
>>>>>
>> I see during switch away from DFLL (suspend), cclk_g parent is not
>> changed to PLLP before changing dfll to open loop mode.
>>
>> Will add this ...
> The CPUFreq driver switches parent to PLLP during the probe, similar
> should be done on suspend.
>
> I'm also wondering if it's always safe to switch to PLLP in the probe.
> If CPU is running on a lower freq than PLLP, then some other more
> appropriate intermediate parent should be selected.
>
CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher rate 
so switching to PLL_P during CPUFreq probe prior to dfll clock enable 
should be safe.
>>>>> For OVR regulator:
>>>>> 2) Change PWM output pin from output to tristate: vdd_cpu will go back
>>>>>      to hardwired boot voltage.
>>>>> 3) Disable DFLL PWM output
>>>>>
>>>>> For I2C regulator:
>>>>> 2) Program vdd_cpu regulator voltage to the boot voltage
>>>>> 3) Disable DFLL I2C output
>>>>>
>>>>> 4) Reprogram parent saved in step 0 of 'Switch to DFLL' to the saved
>>>>>      frequency
>>>>> 5) Change cclk_g parent to saved parent
>>>>> 6) Disable DFLL
>>> This is the same sequence currently implemented. But dfll
>>> suspend/resume calls are thru Tegra210 clock driver.
>>>
>>> Dmitry wants to have dfll suspend/resume along with CCLK_G restore to
>>> happen from CPUFreq driver pm_ops rather than tegra210 clock driver or
>>> tegra dfll driver.
>>>
>>> Will move it to CPUFreq driver...
>>>
>> Thanks!

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 18:30                                 ` Sowjanya Komatineni
@ 2019-07-16 18:43                                   ` Dmitry Osipenko
  2019-07-16 19:26                                     ` Sowjanya Komatineni
  2019-07-18 19:18                                     ` Peter De Schrijver
  0 siblings, 2 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-16 18:43 UTC (permalink / raw)
  To: Sowjanya Komatineni, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree

16.07.2019 21:30, Sowjanya Komatineni пишет:
> 
> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>> The other thing that also need attention is that T124 CPUFreq
>>>>>>>>> driver
>>>>>>>>> implicitly relies on DFLL driver to be probed first, which is
>>>>>>>>> icky.
>>>>>>>>>
>>>>>>>> Should I add check for successful dfll clk register explicitly in
>>>>>>>> CPUFreq driver probe and defer till dfll clk registers?
>>>>> Probably you should use the "device links". See [1][2] for the
>>>>> example.
>>>>>
>>>>> [1]
>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>
>>>>>
>>>>>
>>>>> [2] https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>
>>>>> Return EPROBE_DEFER instead of EINVAL if device_link_add() fails. And
>>>>> use of_find_device_by_node() to get the DFLL's device, see [3].
>>>>>
>>>>> [3]
>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>
>>>>>
>>>> Will go thru and add...
>> Looks like I initially confused this case with getting orphaned clock.
>> I'm now seeing that the DFLL driver registers the clock and then
>> clk_get(dfll) should be returning EPROBE_DEFER until DFLL driver is
>> probed, hence everything should be fine as-is and there is no real need
>> for the 'device link'. Sorry for the confusion!
>>
>>>>>>> Sorry, I didn't follow the mail thread. Just regarding the DFLL
>>>>>>> part.
>>>>>>>
>>>>>>> As you know it, the DFLL clock is one of the CPU clock sources and
>>>>>>> integrated with DVFS control logic with the regulator. We will not
>>>>>>> switch
>>>>>>> CPU to other clock sources once we switched to DFLL. Because the
>>>>>>> CPU has
>>>>>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP table
>>>>>>> you see
>>>>>>> in the driver.). We shouldn't reparent it to other sources with
>>>>>>> unknew
>>>>>>> freq/volt pair. That's not guaranteed to work. We allow switching to
>>>>>>> open-loop mode but different sources.
>>>>> Okay, then the CPUFreq driver will have to enforce DFLL freq to PLLP's
>>>>> rate before switching to PLLP in order to have a proper CPU voltage.
>>>> PLLP freq is safe to work for any CPU voltage. So no need to enforce
>>>> DFLL freq to PLLP rate before changing CCLK_G source to PLLP during
>>>> suspend
>>>>
>>> Sorry, please ignore my above comment. During suspend, need to change
>>> CCLK_G source to PLLP when dfll is in closed loop mode first and then
>>> dfll need to be set to open loop.
>> Okay.
>>
>>>>>>> And I don't exactly understand why we need to switch to PLLP in CPU
>>>>>>> idle
>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>>
>>>>>>> In SC7 entry, the dfll suspend function moves it the open-loop
>>>>>>> mode. That's
>>>>>>> all. The sc7-entryfirmware will handle the rest of the sequence to
>>>>>>> turn off
>>>>>>> the CPU power.
>>>>>>>
>>>>>>> In SC7 resume, the warmboot code will handle the sequence to turn on
>>>>>>> regulator and power up the CPU cluster. And leave it on PLL_P. After
>>>>>>> resuming to the kernel, we re-init DFLL, restore the CPU clock
>>>>>>> policy (CPU
>>>>>>> runs on DFLL open-loop mode) and then moving to close-loop mode.
>>>>> The DFLL is re-inited after switching CCLK to DFLL parent during of
>>>>> the
>>>>> early clocks-state restoring by CaR driver. Hence instead of having
>>>>> odd
>>>>> hacks in the CaR driver, it is much nicer to have a proper
>>>>> suspend-resume sequencing of the device drivers. In this case CPUFreq
>>>>> driver is the driver that enables DFLL and switches CPU to that clock
>>>>> source, which means that this driver is also should be responsible for
>>>>> management of the DFLL's state during of suspend/resume process. If
>>>>> CPUFreq driver disables DFLL during suspend and re-enables it during
>>>>> resume, then looks like the CaR driver hacks around DFLL are not
>>>>> needed.
>>>>>
>>>>>>> The DFLL part looks good to me. BTW, change the patch subject to
>>>>>>> "Add
>>>>>>> suspend-resume support" seems more appropriate to me.
>>>>>>>
>>>>>> To clarify this, the sequences for DFLL use are as follows (assuming
>>>>>> all
>>>>>> required DFLL hw configuration has been done)
>>>>>>
>>>>>> Switch to DFLL:
>>>>>> 0) Save current parent and frequency
>>>>>> 1) Program DFLL to open loop mode
>>>>>> 2) Enable DFLL
>>>>>> 3) Change cclk_g parent to DFLL
>>>>>> For OVR regulator:
>>>>>> 4) Change PWM output pin from tristate to output
>>>>>> 5) Enable DFLL PWM output
>>>>>> For I2C regulator:
>>>>>> 4) Enable DFLL I2C output
>>>>>> 6) Program DFLL to closed loop mode
>>>>>>
>>>>>> Switch away from DFLL:
>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for any
>>>>>> vdd_cpu voltage
>>>>>> 1) Program DFLL to open loop mode
>>>>>>
>>> I see during switch away from DFLL (suspend), cclk_g parent is not
>>> changed to PLLP before changing dfll to open loop mode.
>>>
>>> Will add this ...
>> The CPUFreq driver switches parent to PLLP during the probe, similar
>> should be done on suspend.
>>
>> I'm also wondering if it's always safe to switch to PLLP in the probe.
>> If CPU is running on a lower freq than PLLP, then some other more
>> appropriate intermediate parent should be selected.
>>
> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher rate
> so switching to PLL_P during CPUFreq probe prior to dfll clock enable
> should be safe.

AFAIK, PLLX could run at ~200MHz. There is also a divided output of PLLP
which CCLKG supports, the PLLP_OUT4.

Probably, realistically, CPU is always running off a fast PLLX during
boot, but I'm wondering what may happen on KEXEC. I guess ideally
CPUFreq driver should also have a 'shutdown' callback to teardown DFLL
on a reboot, but likely that there are other clock-related problems as
well that may break KEXEC and thus it is not very important at the moment.

[snip]

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 18:43                                   ` Dmitry Osipenko
@ 2019-07-16 19:26                                     ` Sowjanya Komatineni
  2019-07-16 20:47                                       ` Dmitry Osipenko
  2019-07-18 19:18                                     ` Peter De Schrijver
  1 sibling, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-16 19:26 UTC (permalink / raw)
  To: Dmitry Osipenko, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree


On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>> The other thing that also need attention is that T124 CPUFreq
>>>>>>>>>> driver
>>>>>>>>>> implicitly relies on DFLL driver to be probed first, which is
>>>>>>>>>> icky.
>>>>>>>>>>
>>>>>>>>> Should I add check for successful dfll clk register explicitly in
>>>>>>>>> CPUFreq driver probe and defer till dfll clk registers?
>>>>>> Probably you should use the "device links". See [1][2] for the
>>>>>> example.
>>>>>>
>>>>>> [1]
>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>
>>>>>>
>>>>>>
>>>>>> [2] https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>
>>>>>> Return EPROBE_DEFER instead of EINVAL if device_link_add() fails. And
>>>>>> use of_find_device_by_node() to get the DFLL's device, see [3].
>>>>>>
>>>>>> [3]
>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>
>>>>>>
>>>>> Will go thru and add...
>>> Looks like I initially confused this case with getting orphaned clock.
>>> I'm now seeing that the DFLL driver registers the clock and then
>>> clk_get(dfll) should be returning EPROBE_DEFER until DFLL driver is
>>> probed, hence everything should be fine as-is and there is no real need
>>> for the 'device link'. Sorry for the confusion!
>>>
>>>>>>>> Sorry, I didn't follow the mail thread. Just regarding the DFLL
>>>>>>>> part.
>>>>>>>>
>>>>>>>> As you know it, the DFLL clock is one of the CPU clock sources and
>>>>>>>> integrated with DVFS control logic with the regulator. We will not
>>>>>>>> switch
>>>>>>>> CPU to other clock sources once we switched to DFLL. Because the
>>>>>>>> CPU has
>>>>>>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP table
>>>>>>>> you see
>>>>>>>> in the driver.). We shouldn't reparent it to other sources with
>>>>>>>> unknew
>>>>>>>> freq/volt pair. That's not guaranteed to work. We allow switching to
>>>>>>>> open-loop mode but different sources.
>>>>>> Okay, then the CPUFreq driver will have to enforce DFLL freq to PLLP's
>>>>>> rate before switching to PLLP in order to have a proper CPU voltage.
>>>>> PLLP freq is safe to work for any CPU voltage. So no need to enforce
>>>>> DFLL freq to PLLP rate before changing CCLK_G source to PLLP during
>>>>> suspend
>>>>>
>>>> Sorry, please ignore my above comment. During suspend, need to change
>>>> CCLK_G source to PLLP when dfll is in closed loop mode first and then
>>>> dfll need to be set to open loop.
>>> Okay.
>>>
>>>>>>>> And I don't exactly understand why we need to switch to PLLP in CPU
>>>>>>>> idle
>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>>>
>>>>>>>> In SC7 entry, the dfll suspend function moves it the open-loop
>>>>>>>> mode. That's
>>>>>>>> all. The sc7-entryfirmware will handle the rest of the sequence to
>>>>>>>> turn off
>>>>>>>> the CPU power.
>>>>>>>>
>>>>>>>> In SC7 resume, the warmboot code will handle the sequence to turn on
>>>>>>>> regulator and power up the CPU cluster. And leave it on PLL_P. After
>>>>>>>> resuming to the kernel, we re-init DFLL, restore the CPU clock
>>>>>>>> policy (CPU
>>>>>>>> runs on DFLL open-loop mode) and then moving to close-loop mode.
>>>>>> The DFLL is re-inited after switching CCLK to DFLL parent during of
>>>>>> the
>>>>>> early clocks-state restoring by CaR driver. Hence instead of having
>>>>>> odd
>>>>>> hacks in the CaR driver, it is much nicer to have a proper
>>>>>> suspend-resume sequencing of the device drivers. In this case CPUFreq
>>>>>> driver is the driver that enables DFLL and switches CPU to that clock
>>>>>> source, which means that this driver is also should be responsible for
>>>>>> management of the DFLL's state during of suspend/resume process. If
>>>>>> CPUFreq driver disables DFLL during suspend and re-enables it during
>>>>>> resume, then looks like the CaR driver hacks around DFLL are not
>>>>>> needed.
>>>>>>
>>>>>>>> The DFLL part looks good to me. BTW, change the patch subject to
>>>>>>>> "Add
>>>>>>>> suspend-resume support" seems more appropriate to me.
>>>>>>>>
>>>>>>> To clarify this, the sequences for DFLL use are as follows (assuming
>>>>>>> all
>>>>>>> required DFLL hw configuration has been done)
>>>>>>>
>>>>>>> Switch to DFLL:
>>>>>>> 0) Save current parent and frequency
>>>>>>> 1) Program DFLL to open loop mode
>>>>>>> 2) Enable DFLL
>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>> For OVR regulator:
>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>> 5) Enable DFLL PWM output
>>>>>>> For I2C regulator:
>>>>>>> 4) Enable DFLL I2C output
>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>
>>>>>>> Switch away from DFLL:
>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for any
>>>>>>> vdd_cpu voltage
>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>
>>>> I see during switch away from DFLL (suspend), cclk_g parent is not
>>>> changed to PLLP before changing dfll to open loop mode.
>>>>
>>>> Will add this ...
>>> The CPUFreq driver switches parent to PLLP during the probe, similar
>>> should be done on suspend.
>>>
>>> I'm also wondering if it's always safe to switch to PLLP in the probe.
>>> If CPU is running on a lower freq than PLLP, then some other more
>>> appropriate intermediate parent should be selected.
>>>
>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher rate
>> so switching to PLL_P during CPUFreq probe prior to dfll clock enable
>> should be safe.
> AFAIK, PLLX could run at ~200MHz. There is also a divided output of PLLP
> which CCLKG supports, the PLLP_OUT4.
>
> Probably, realistically, CPU is always running off a fast PLLX during
> boot, but I'm wondering what may happen on KEXEC. I guess ideally
> CPUFreq driver should also have a 'shutdown' callback to teardown DFLL
> on a reboot, but likely that there are other clock-related problems as
> well that may break KEXEC and thus it is not very important at the moment.
>
> [snip]

During bootup CPUG sources from PLL_X. By PLL_P source above I meant 
PLL_P_OUT4.

As per clock policies, PLL_X is always used for high freq like >800Mhz 
and for low frequency it will be sourced from PLLP.



^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 19:26                                     ` Sowjanya Komatineni
@ 2019-07-16 20:47                                       ` Dmitry Osipenko
  2019-07-16 21:12                                         ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-16 20:47 UTC (permalink / raw)
  To: Sowjanya Komatineni, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree

16.07.2019 22:26, Sowjanya Komatineni пишет:
> 
> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>> The other thing that also need attention is that T124 CPUFreq
>>>>>>>>>>> driver
>>>>>>>>>>> implicitly relies on DFLL driver to be probed first, which is
>>>>>>>>>>> icky.
>>>>>>>>>>>
>>>>>>>>>> Should I add check for successful dfll clk register explicitly in
>>>>>>>>>> CPUFreq driver probe and defer till dfll clk registers?
>>>>>>> Probably you should use the "device links". See [1][2] for the
>>>>>>> example.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [2]
>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>
>>>>>>> Return EPROBE_DEFER instead of EINVAL if device_link_add() fails.
>>>>>>> And
>>>>>>> use of_find_device_by_node() to get the DFLL's device, see [3].
>>>>>>>
>>>>>>> [3]
>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Will go thru and add...
>>>> Looks like I initially confused this case with getting orphaned clock.
>>>> I'm now seeing that the DFLL driver registers the clock and then
>>>> clk_get(dfll) should be returning EPROBE_DEFER until DFLL driver is
>>>> probed, hence everything should be fine as-is and there is no real need
>>>> for the 'device link'. Sorry for the confusion!
>>>>
>>>>>>>>> Sorry, I didn't follow the mail thread. Just regarding the DFLL
>>>>>>>>> part.
>>>>>>>>>
>>>>>>>>> As you know it, the DFLL clock is one of the CPU clock sources and
>>>>>>>>> integrated with DVFS control logic with the regulator. We will not
>>>>>>>>> switch
>>>>>>>>> CPU to other clock sources once we switched to DFLL. Because the
>>>>>>>>> CPU has
>>>>>>>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP
>>>>>>>>> table
>>>>>>>>> you see
>>>>>>>>> in the driver.). We shouldn't reparent it to other sources with
>>>>>>>>> unknew
>>>>>>>>> freq/volt pair. That's not guaranteed to work. We allow
>>>>>>>>> switching to
>>>>>>>>> open-loop mode but different sources.
>>>>>>> Okay, then the CPUFreq driver will have to enforce DFLL freq to
>>>>>>> PLLP's
>>>>>>> rate before switching to PLLP in order to have a proper CPU voltage.
>>>>>> PLLP freq is safe to work for any CPU voltage. So no need to enforce
>>>>>> DFLL freq to PLLP rate before changing CCLK_G source to PLLP during
>>>>>> suspend
>>>>>>
>>>>> Sorry, please ignore my above comment. During suspend, need to change
>>>>> CCLK_G source to PLLP when dfll is in closed loop mode first and then
>>>>> dfll need to be set to open loop.
>>>> Okay.
>>>>
>>>>>>>>> And I don't exactly understand why we need to switch to PLLP in
>>>>>>>>> CPU
>>>>>>>>> idle
>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>>>>
>>>>>>>>> In SC7 entry, the dfll suspend function moves it the open-loop
>>>>>>>>> mode. That's
>>>>>>>>> all. The sc7-entryfirmware will handle the rest of the sequence to
>>>>>>>>> turn off
>>>>>>>>> the CPU power.
>>>>>>>>>
>>>>>>>>> In SC7 resume, the warmboot code will handle the sequence to
>>>>>>>>> turn on
>>>>>>>>> regulator and power up the CPU cluster. And leave it on PLL_P.
>>>>>>>>> After
>>>>>>>>> resuming to the kernel, we re-init DFLL, restore the CPU clock
>>>>>>>>> policy (CPU
>>>>>>>>> runs on DFLL open-loop mode) and then moving to close-loop mode.
>>>>>>> The DFLL is re-inited after switching CCLK to DFLL parent during of
>>>>>>> the
>>>>>>> early clocks-state restoring by CaR driver. Hence instead of having
>>>>>>> odd
>>>>>>> hacks in the CaR driver, it is much nicer to have a proper
>>>>>>> suspend-resume sequencing of the device drivers. In this case
>>>>>>> CPUFreq
>>>>>>> driver is the driver that enables DFLL and switches CPU to that
>>>>>>> clock
>>>>>>> source, which means that this driver is also should be
>>>>>>> responsible for
>>>>>>> management of the DFLL's state during of suspend/resume process. If
>>>>>>> CPUFreq driver disables DFLL during suspend and re-enables it during
>>>>>>> resume, then looks like the CaR driver hacks around DFLL are not
>>>>>>> needed.
>>>>>>>
>>>>>>>>> The DFLL part looks good to me. BTW, change the patch subject to
>>>>>>>>> "Add
>>>>>>>>> suspend-resume support" seems more appropriate to me.
>>>>>>>>>
>>>>>>>> To clarify this, the sequences for DFLL use are as follows
>>>>>>>> (assuming
>>>>>>>> all
>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>
>>>>>>>> Switch to DFLL:
>>>>>>>> 0) Save current parent and frequency
>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>> 2) Enable DFLL
>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>> For OVR regulator:
>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>> For I2C regulator:
>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>
>>>>>>>> Switch away from DFLL:
>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for any
>>>>>>>> vdd_cpu voltage
>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>
>>>>> I see during switch away from DFLL (suspend), cclk_g parent is not
>>>>> changed to PLLP before changing dfll to open loop mode.
>>>>>
>>>>> Will add this ...
>>>> The CPUFreq driver switches parent to PLLP during the probe, similar
>>>> should be done on suspend.
>>>>
>>>> I'm also wondering if it's always safe to switch to PLLP in the probe.
>>>> If CPU is running on a lower freq than PLLP, then some other more
>>>> appropriate intermediate parent should be selected.
>>>>
>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher rate
>>> so switching to PLL_P during CPUFreq probe prior to dfll clock enable
>>> should be safe.
>> AFAIK, PLLX could run at ~200MHz. There is also a divided output of PLLP
>> which CCLKG supports, the PLLP_OUT4.
>>
>> Probably, realistically, CPU is always running off a fast PLLX during
>> boot, but I'm wondering what may happen on KEXEC. I guess ideally
>> CPUFreq driver should also have a 'shutdown' callback to teardown DFLL
>> on a reboot, but likely that there are other clock-related problems as
>> well that may break KEXEC and thus it is not very important at the
>> moment.
>>
>> [snip]
> 
> During bootup CPUG sources from PLL_X. By PLL_P source above I meant
> PLL_P_OUT4.
> 
> As per clock policies, PLL_X is always used for high freq like >800Mhz
> and for low frequency it will be sourced from PLLP.

Alright, then please don't forget to pre-initialize PLLP_OUT4 rate to a
reasonable value using tegra_clk_init_table or assigned-clocks.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 20:47                                       ` Dmitry Osipenko
@ 2019-07-16 21:12                                         ` Sowjanya Komatineni
  2019-07-16 21:21                                           ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-16 21:12 UTC (permalink / raw)
  To: Dmitry Osipenko, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree


On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>> The other thing that also need attention is that T124 CPUFreq
>>>>>>>>>>>> driver
>>>>>>>>>>>> implicitly relies on DFLL driver to be probed first, which is
>>>>>>>>>>>> icky.
>>>>>>>>>>>>
>>>>>>>>>>> Should I add check for successful dfll clk register explicitly in
>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk registers?
>>>>>>>> Probably you should use the "device links". See [1][2] for the
>>>>>>>> example.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [2]
>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>
>>>>>>>> Return EPROBE_DEFER instead of EINVAL if device_link_add() fails.
>>>>>>>> And
>>>>>>>> use of_find_device_by_node() to get the DFLL's device, see [3].
>>>>>>>>
>>>>>>>> [3]
>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Will go thru and add...
>>>>> Looks like I initially confused this case with getting orphaned clock.
>>>>> I'm now seeing that the DFLL driver registers the clock and then
>>>>> clk_get(dfll) should be returning EPROBE_DEFER until DFLL driver is
>>>>> probed, hence everything should be fine as-is and there is no real need
>>>>> for the 'device link'. Sorry for the confusion!
>>>>>
>>>>>>>>>> Sorry, I didn't follow the mail thread. Just regarding the DFLL
>>>>>>>>>> part.
>>>>>>>>>>
>>>>>>>>>> As you know it, the DFLL clock is one of the CPU clock sources and
>>>>>>>>>> integrated with DVFS control logic with the regulator. We will not
>>>>>>>>>> switch
>>>>>>>>>> CPU to other clock sources once we switched to DFLL. Because the
>>>>>>>>>> CPU has
>>>>>>>>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP
>>>>>>>>>> table
>>>>>>>>>> you see
>>>>>>>>>> in the driver.). We shouldn't reparent it to other sources with
>>>>>>>>>> unknew
>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We allow
>>>>>>>>>> switching to
>>>>>>>>>> open-loop mode but different sources.
>>>>>>>> Okay, then the CPUFreq driver will have to enforce DFLL freq to
>>>>>>>> PLLP's
>>>>>>>> rate before switching to PLLP in order to have a proper CPU voltage.
>>>>>>> PLLP freq is safe to work for any CPU voltage. So no need to enforce
>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source to PLLP during
>>>>>>> suspend
>>>>>>>
>>>>>> Sorry, please ignore my above comment. During suspend, need to change
>>>>>> CCLK_G source to PLLP when dfll is in closed loop mode first and then
>>>>>> dfll need to be set to open loop.
>>>>> Okay.
>>>>>
>>>>>>>>>> And I don't exactly understand why we need to switch to PLLP in
>>>>>>>>>> CPU
>>>>>>>>>> idle
>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>>>>>
>>>>>>>>>> In SC7 entry, the dfll suspend function moves it the open-loop
>>>>>>>>>> mode. That's
>>>>>>>>>> all. The sc7-entryfirmware will handle the rest of the sequence to
>>>>>>>>>> turn off
>>>>>>>>>> the CPU power.
>>>>>>>>>>
>>>>>>>>>> In SC7 resume, the warmboot code will handle the sequence to
>>>>>>>>>> turn on
>>>>>>>>>> regulator and power up the CPU cluster. And leave it on PLL_P.
>>>>>>>>>> After
>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore the CPU clock
>>>>>>>>>> policy (CPU
>>>>>>>>>> runs on DFLL open-loop mode) and then moving to close-loop mode.
>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL parent during of
>>>>>>>> the
>>>>>>>> early clocks-state restoring by CaR driver. Hence instead of having
>>>>>>>> odd
>>>>>>>> hacks in the CaR driver, it is much nicer to have a proper
>>>>>>>> suspend-resume sequencing of the device drivers. In this case
>>>>>>>> CPUFreq
>>>>>>>> driver is the driver that enables DFLL and switches CPU to that
>>>>>>>> clock
>>>>>>>> source, which means that this driver is also should be
>>>>>>>> responsible for
>>>>>>>> management of the DFLL's state during of suspend/resume process. If
>>>>>>>> CPUFreq driver disables DFLL during suspend and re-enables it during
>>>>>>>> resume, then looks like the CaR driver hacks around DFLL are not
>>>>>>>> needed.
>>>>>>>>
>>>>>>>>>> The DFLL part looks good to me. BTW, change the patch subject to
>>>>>>>>>> "Add
>>>>>>>>>> suspend-resume support" seems more appropriate to me.
>>>>>>>>>>
>>>>>>>>> To clarify this, the sequences for DFLL use are as follows
>>>>>>>>> (assuming
>>>>>>>>> all
>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>
>>>>>>>>> Switch to DFLL:
>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>> 2) Enable DFLL
>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>> For OVR regulator:
>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>> For I2C regulator:
>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>
>>>>>>>>> Switch away from DFLL:
>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for any
>>>>>>>>> vdd_cpu voltage
>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>
>>>>>> I see during switch away from DFLL (suspend), cclk_g parent is not
>>>>>> changed to PLLP before changing dfll to open loop mode.
>>>>>>
>>>>>> Will add this ...
>>>>> The CPUFreq driver switches parent to PLLP during the probe, similar
>>>>> should be done on suspend.
>>>>>
>>>>> I'm also wondering if it's always safe to switch to PLLP in the probe.
>>>>> If CPU is running on a lower freq than PLLP, then some other more
>>>>> appropriate intermediate parent should be selected.
>>>>>
>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher rate
>>>> so switching to PLL_P during CPUFreq probe prior to dfll clock enable
>>>> should be safe.
>>> AFAIK, PLLX could run at ~200MHz. There is also a divided output of PLLP
>>> which CCLKG supports, the PLLP_OUT4.
>>>
>>> Probably, realistically, CPU is always running off a fast PLLX during
>>> boot, but I'm wondering what may happen on KEXEC. I guess ideally
>>> CPUFreq driver should also have a 'shutdown' callback to teardown DFLL
>>> on a reboot, but likely that there are other clock-related problems as
>>> well that may break KEXEC and thus it is not very important at the
>>> moment.
>>>
>>> [snip]
>> During bootup CPUG sources from PLL_X. By PLL_P source above I meant
>> PLL_P_OUT4.
>>
>> As per clock policies, PLL_X is always used for high freq like >800Mhz
>> and for low frequency it will be sourced from PLLP.
> Alright, then please don't forget to pre-initialize PLLP_OUT4 rate to a
> reasonable value using tegra_clk_init_table or assigned-clocks.

PLLP_OUT4 rate update is not needed as it is safe to run at 408Mhz 
because it is below fmax @ Vmin



^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 21:12                                         ` Sowjanya Komatineni
@ 2019-07-16 21:21                                           ` Dmitry Osipenko
  2019-07-16 21:35                                             ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-16 21:21 UTC (permalink / raw)
  To: Sowjanya Komatineni, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree

17.07.2019 0:12, Sowjanya Komatineni пишет:
> 
> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>> The other thing that also need attention is that T124 CPUFreq
>>>>>>>>>>>>> driver
>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed first, which is
>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>
>>>>>>>>>>>> Should I add check for successful dfll clk register
>>>>>>>>>>>> explicitly in
>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk registers?
>>>>>>>>> Probably you should use the "device links". See [1][2] for the
>>>>>>>>> example.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [2]
>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>
>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if device_link_add() fails.
>>>>>>>>> And
>>>>>>>>> use of_find_device_by_node() to get the DFLL's device, see [3].
>>>>>>>>>
>>>>>>>>> [3]
>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Will go thru and add...
>>>>>> Looks like I initially confused this case with getting orphaned
>>>>>> clock.
>>>>>> I'm now seeing that the DFLL driver registers the clock and then
>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until DFLL driver is
>>>>>> probed, hence everything should be fine as-is and there is no real
>>>>>> need
>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>
>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just regarding the DFLL
>>>>>>>>>>> part.
>>>>>>>>>>>
>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU clock
>>>>>>>>>>> sources and
>>>>>>>>>>> integrated with DVFS control logic with the regulator. We
>>>>>>>>>>> will not
>>>>>>>>>>> switch
>>>>>>>>>>> CPU to other clock sources once we switched to DFLL. Because the
>>>>>>>>>>> CPU has
>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP
>>>>>>>>>>> table
>>>>>>>>>>> you see
>>>>>>>>>>> in the driver.). We shouldn't reparent it to other sources with
>>>>>>>>>>> unknew
>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We allow
>>>>>>>>>>> switching to
>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>> Okay, then the CPUFreq driver will have to enforce DFLL freq to
>>>>>>>>> PLLP's
>>>>>>>>> rate before switching to PLLP in order to have a proper CPU
>>>>>>>>> voltage.
>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no need to
>>>>>>>> enforce
>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source to PLLP during
>>>>>>>> suspend
>>>>>>>>
>>>>>>> Sorry, please ignore my above comment. During suspend, need to
>>>>>>> change
>>>>>>> CCLK_G source to PLLP when dfll is in closed loop mode first and
>>>>>>> then
>>>>>>> dfll need to be set to open loop.
>>>>>> Okay.
>>>>>>
>>>>>>>>>>> And I don't exactly understand why we need to switch to PLLP in
>>>>>>>>>>> CPU
>>>>>>>>>>> idle
>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>>>>>>
>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it the open-loop
>>>>>>>>>>> mode. That's
>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest of the
>>>>>>>>>>> sequence to
>>>>>>>>>>> turn off
>>>>>>>>>>> the CPU power.
>>>>>>>>>>>
>>>>>>>>>>> In SC7 resume, the warmboot code will handle the sequence to
>>>>>>>>>>> turn on
>>>>>>>>>>> regulator and power up the CPU cluster. And leave it on PLL_P.
>>>>>>>>>>> After
>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore the CPU clock
>>>>>>>>>>> policy (CPU
>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to close-loop mode.
>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL parent
>>>>>>>>> during of
>>>>>>>>> the
>>>>>>>>> early clocks-state restoring by CaR driver. Hence instead of
>>>>>>>>> having
>>>>>>>>> odd
>>>>>>>>> hacks in the CaR driver, it is much nicer to have a proper
>>>>>>>>> suspend-resume sequencing of the device drivers. In this case
>>>>>>>>> CPUFreq
>>>>>>>>> driver is the driver that enables DFLL and switches CPU to that
>>>>>>>>> clock
>>>>>>>>> source, which means that this driver is also should be
>>>>>>>>> responsible for
>>>>>>>>> management of the DFLL's state during of suspend/resume
>>>>>>>>> process. If
>>>>>>>>> CPUFreq driver disables DFLL during suspend and re-enables it
>>>>>>>>> during
>>>>>>>>> resume, then looks like the CaR driver hacks around DFLL are not
>>>>>>>>> needed.
>>>>>>>>>
>>>>>>>>>>> The DFLL part looks good to me. BTW, change the patch subject to
>>>>>>>>>>> "Add
>>>>>>>>>>> suspend-resume support" seems more appropriate to me.
>>>>>>>>>>>
>>>>>>>>>> To clarify this, the sequences for DFLL use are as follows
>>>>>>>>>> (assuming
>>>>>>>>>> all
>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>
>>>>>>>>>> Switch to DFLL:
>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>> For OVR regulator:
>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>> For I2C regulator:
>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>
>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for
>>>>>>>>>> any
>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>
>>>>>>> I see during switch away from DFLL (suspend), cclk_g parent is not
>>>>>>> changed to PLLP before changing dfll to open loop mode.
>>>>>>>
>>>>>>> Will add this ...
>>>>>> The CPUFreq driver switches parent to PLLP during the probe, similar
>>>>>> should be done on suspend.
>>>>>>
>>>>>> I'm also wondering if it's always safe to switch to PLLP in the
>>>>>> probe.
>>>>>> If CPU is running on a lower freq than PLLP, then some other more
>>>>>> appropriate intermediate parent should be selected.
>>>>>>
>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher
>>>>> rate
>>>>> so switching to PLL_P during CPUFreq probe prior to dfll clock enable
>>>>> should be safe.
>>>> AFAIK, PLLX could run at ~200MHz. There is also a divided output of
>>>> PLLP
>>>> which CCLKG supports, the PLLP_OUT4.
>>>>
>>>> Probably, realistically, CPU is always running off a fast PLLX during
>>>> boot, but I'm wondering what may happen on KEXEC. I guess ideally
>>>> CPUFreq driver should also have a 'shutdown' callback to teardown DFLL
>>>> on a reboot, but likely that there are other clock-related problems as
>>>> well that may break KEXEC and thus it is not very important at the
>>>> moment.
>>>>
>>>> [snip]
>>> During bootup CPUG sources from PLL_X. By PLL_P source above I meant
>>> PLL_P_OUT4.
>>>
>>> As per clock policies, PLL_X is always used for high freq like >800Mhz
>>> and for low frequency it will be sourced from PLLP.
>> Alright, then please don't forget to pre-initialize PLLP_OUT4 rate to a
>> reasonable value using tegra_clk_init_table or assigned-clocks.
> 
> PLLP_OUT4 rate update is not needed as it is safe to run at 408Mhz
> because it is below fmax @ Vmin

So even 204MHz CVB entries are having the same voltage as 408MHz,
correct? It's not instantly obvious to me from the DFLL driver's code
where the fmax @ Vmin is defined, I see that there is the min_millivolts
and frequency entries starting from 204MHZ defined per-table.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 21:21                                           ` Dmitry Osipenko
@ 2019-07-16 21:35                                             ` Sowjanya Komatineni
  2019-07-16 22:00                                               ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-16 21:35 UTC (permalink / raw)
  To: Dmitry Osipenko, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree


On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>> The other thing that also need attention is that T124 CPUFreq
>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed first, which is
>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Should I add check for successful dfll clk register
>>>>>>>>>>>>> explicitly in
>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk registers?
>>>>>>>>>> Probably you should use the "device links". See [1][2] for the
>>>>>>>>>> example.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [2]
>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>
>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if device_link_add() fails.
>>>>>>>>>> And
>>>>>>>>>> use of_find_device_by_node() to get the DFLL's device, see [3].
>>>>>>>>>>
>>>>>>>>>> [3]
>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Will go thru and add...
>>>>>>> Looks like I initially confused this case with getting orphaned
>>>>>>> clock.
>>>>>>> I'm now seeing that the DFLL driver registers the clock and then
>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until DFLL driver is
>>>>>>> probed, hence everything should be fine as-is and there is no real
>>>>>>> need
>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>
>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just regarding the DFLL
>>>>>>>>>>>> part.
>>>>>>>>>>>>
>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU clock
>>>>>>>>>>>> sources and
>>>>>>>>>>>> integrated with DVFS control logic with the regulator. We
>>>>>>>>>>>> will not
>>>>>>>>>>>> switch
>>>>>>>>>>>> CPU to other clock sources once we switched to DFLL. Because the
>>>>>>>>>>>> CPU has
>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP
>>>>>>>>>>>> table
>>>>>>>>>>>> you see
>>>>>>>>>>>> in the driver.). We shouldn't reparent it to other sources with
>>>>>>>>>>>> unknew
>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We allow
>>>>>>>>>>>> switching to
>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce DFLL freq to
>>>>>>>>>> PLLP's
>>>>>>>>>> rate before switching to PLLP in order to have a proper CPU
>>>>>>>>>> voltage.
>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no need to
>>>>>>>>> enforce
>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source to PLLP during
>>>>>>>>> suspend
>>>>>>>>>
>>>>>>>> Sorry, please ignore my above comment. During suspend, need to
>>>>>>>> change
>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop mode first and
>>>>>>>> then
>>>>>>>> dfll need to be set to open loop.
>>>>>>> Okay.
>>>>>>>
>>>>>>>>>>>> And I don't exactly understand why we need to switch to PLLP in
>>>>>>>>>>>> CPU
>>>>>>>>>>>> idle
>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>>>>>>>
>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it the open-loop
>>>>>>>>>>>> mode. That's
>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest of the
>>>>>>>>>>>> sequence to
>>>>>>>>>>>> turn off
>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>
>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the sequence to
>>>>>>>>>>>> turn on
>>>>>>>>>>>> regulator and power up the CPU cluster. And leave it on PLL_P.
>>>>>>>>>>>> After
>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore the CPU clock
>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to close-loop mode.
>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL parent
>>>>>>>>>> during of
>>>>>>>>>> the
>>>>>>>>>> early clocks-state restoring by CaR driver. Hence instead of
>>>>>>>>>> having
>>>>>>>>>> odd
>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a proper
>>>>>>>>>> suspend-resume sequencing of the device drivers. In this case
>>>>>>>>>> CPUFreq
>>>>>>>>>> driver is the driver that enables DFLL and switches CPU to that
>>>>>>>>>> clock
>>>>>>>>>> source, which means that this driver is also should be
>>>>>>>>>> responsible for
>>>>>>>>>> management of the DFLL's state during of suspend/resume
>>>>>>>>>> process. If
>>>>>>>>>> CPUFreq driver disables DFLL during suspend and re-enables it
>>>>>>>>>> during
>>>>>>>>>> resume, then looks like the CaR driver hacks around DFLL are not
>>>>>>>>>> needed.
>>>>>>>>>>
>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the patch subject to
>>>>>>>>>>>> "Add
>>>>>>>>>>>> suspend-resume support" seems more appropriate to me.
>>>>>>>>>>>>
>>>>>>>>>>> To clarify this, the sequences for DFLL use are as follows
>>>>>>>>>>> (assuming
>>>>>>>>>>> all
>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>
>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>
>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for
>>>>>>>>>>> any
>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>
>>>>>>>> I see during switch away from DFLL (suspend), cclk_g parent is not
>>>>>>>> changed to PLLP before changing dfll to open loop mode.
>>>>>>>>
>>>>>>>> Will add this ...
>>>>>>> The CPUFreq driver switches parent to PLLP during the probe, similar
>>>>>>> should be done on suspend.
>>>>>>>
>>>>>>> I'm also wondering if it's always safe to switch to PLLP in the
>>>>>>> probe.
>>>>>>> If CPU is running on a lower freq than PLLP, then some other more
>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>
>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher
>>>>>> rate
>>>>>> so switching to PLL_P during CPUFreq probe prior to dfll clock enable
>>>>>> should be safe.
>>>>> AFAIK, PLLX could run at ~200MHz. There is also a divided output of
>>>>> PLLP
>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>
>>>>> Probably, realistically, CPU is always running off a fast PLLX during
>>>>> boot, but I'm wondering what may happen on KEXEC. I guess ideally
>>>>> CPUFreq driver should also have a 'shutdown' callback to teardown DFLL
>>>>> on a reboot, but likely that there are other clock-related problems as
>>>>> well that may break KEXEC and thus it is not very important at the
>>>>> moment.
>>>>>
>>>>> [snip]
>>>> During bootup CPUG sources from PLL_X. By PLL_P source above I meant
>>>> PLL_P_OUT4.
>>>>
>>>> As per clock policies, PLL_X is always used for high freq like >800Mhz
>>>> and for low frequency it will be sourced from PLLP.
>>> Alright, then please don't forget to pre-initialize PLLP_OUT4 rate to a
>>> reasonable value using tegra_clk_init_table or assigned-clocks.
>> PLLP_OUT4 rate update is not needed as it is safe to run at 408Mhz
>> because it is below fmax @ Vmin
> So even 204MHz CVB entries are having the same voltage as 408MHz,
> correct? It's not instantly obvious to me from the DFLL driver's code
> where the fmax @ Vmin is defined, I see that there is the min_millivolts
> and frequency entries starting from 204MHZ defined per-table.
Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will work at 
Vmin voltage and PLLP max is 408Mhz.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 21:35                                             ` Sowjanya Komatineni
@ 2019-07-16 22:00                                               ` Dmitry Osipenko
  2019-07-16 22:06                                                 ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-16 22:00 UTC (permalink / raw)
  To: Sowjanya Komatineni, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree

17.07.2019 0:35, Sowjanya Komatineni пишет:
> 
> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>> The other thing that also need attention is that T124
>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed first,
>>>>>>>>>>>>>>> which is
>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Should I add check for successful dfll clk register
>>>>>>>>>>>>>> explicitly in
>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk registers?
>>>>>>>>>>> Probably you should use the "device links". See [1][2] for the
>>>>>>>>>>> example.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [2]
>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if device_link_add()
>>>>>>>>>>> fails.
>>>>>>>>>>> And
>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's device, see [3].
>>>>>>>>>>>
>>>>>>>>>>> [3]
>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Will go thru and add...
>>>>>>>> Looks like I initially confused this case with getting orphaned
>>>>>>>> clock.
>>>>>>>> I'm now seeing that the DFLL driver registers the clock and then
>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until DFLL driver is
>>>>>>>> probed, hence everything should be fine as-is and there is no real
>>>>>>>> need
>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>
>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just regarding the
>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>> part.
>>>>>>>>>>>>>
>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU clock
>>>>>>>>>>>>> sources and
>>>>>>>>>>>>> integrated with DVFS control logic with the regulator. We
>>>>>>>>>>>>> will not
>>>>>>>>>>>>> switch
>>>>>>>>>>>>> CPU to other clock sources once we switched to DFLL.
>>>>>>>>>>>>> Because the
>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP
>>>>>>>>>>>>> table
>>>>>>>>>>>>> you see
>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to other sources
>>>>>>>>>>>>> with
>>>>>>>>>>>>> unknew
>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We allow
>>>>>>>>>>>>> switching to
>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce DFLL freq to
>>>>>>>>>>> PLLP's
>>>>>>>>>>> rate before switching to PLLP in order to have a proper CPU
>>>>>>>>>>> voltage.
>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no need to
>>>>>>>>>> enforce
>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source to PLLP
>>>>>>>>>> during
>>>>>>>>>> suspend
>>>>>>>>>>
>>>>>>>>> Sorry, please ignore my above comment. During suspend, need to
>>>>>>>>> change
>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop mode first and
>>>>>>>>> then
>>>>>>>>> dfll need to be set to open loop.
>>>>>>>> Okay.
>>>>>>>>
>>>>>>>>>>>>> And I don't exactly understand why we need to switch to
>>>>>>>>>>>>> PLLP in
>>>>>>>>>>>>> CPU
>>>>>>>>>>>>> idle
>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it the open-loop
>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest of the
>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>> turn off
>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the sequence to
>>>>>>>>>>>>> turn on
>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave it on PLL_P.
>>>>>>>>>>>>> After
>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore the CPU clock
>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to close-loop
>>>>>>>>>>>>> mode.
>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL parent
>>>>>>>>>>> during of
>>>>>>>>>>> the
>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence instead of
>>>>>>>>>>> having
>>>>>>>>>>> odd
>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a proper
>>>>>>>>>>> suspend-resume sequencing of the device drivers. In this case
>>>>>>>>>>> CPUFreq
>>>>>>>>>>> driver is the driver that enables DFLL and switches CPU to that
>>>>>>>>>>> clock
>>>>>>>>>>> source, which means that this driver is also should be
>>>>>>>>>>> responsible for
>>>>>>>>>>> management of the DFLL's state during of suspend/resume
>>>>>>>>>>> process. If
>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and re-enables it
>>>>>>>>>>> during
>>>>>>>>>>> resume, then looks like the CaR driver hacks around DFLL are not
>>>>>>>>>>> needed.
>>>>>>>>>>>
>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the patch
>>>>>>>>>>>>> subject to
>>>>>>>>>>>>> "Add
>>>>>>>>>>>>> suspend-resume support" seems more appropriate to me.
>>>>>>>>>>>>>
>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as follows
>>>>>>>>>>>> (assuming
>>>>>>>>>>>> all
>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>
>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>
>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for
>>>>>>>>>>>> any
>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>
>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g parent is not
>>>>>>>>> changed to PLLP before changing dfll to open loop mode.
>>>>>>>>>
>>>>>>>>> Will add this ...
>>>>>>>> The CPUFreq driver switches parent to PLLP during the probe,
>>>>>>>> similar
>>>>>>>> should be done on suspend.
>>>>>>>>
>>>>>>>> I'm also wondering if it's always safe to switch to PLLP in the
>>>>>>>> probe.
>>>>>>>> If CPU is running on a lower freq than PLLP, then some other more
>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>
>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher
>>>>>>> rate
>>>>>>> so switching to PLL_P during CPUFreq probe prior to dfll clock
>>>>>>> enable
>>>>>>> should be safe.
>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a divided output of
>>>>>> PLLP
>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>
>>>>>> Probably, realistically, CPU is always running off a fast PLLX during
>>>>>> boot, but I'm wondering what may happen on KEXEC. I guess ideally
>>>>>> CPUFreq driver should also have a 'shutdown' callback to teardown
>>>>>> DFLL
>>>>>> on a reboot, but likely that there are other clock-related
>>>>>> problems as
>>>>>> well that may break KEXEC and thus it is not very important at the
>>>>>> moment.
>>>>>>
>>>>>> [snip]
>>>>> During bootup CPUG sources from PLL_X. By PLL_P source above I meant
>>>>> PLL_P_OUT4.
>>>>>
>>>>> As per clock policies, PLL_X is always used for high freq like >800Mhz
>>>>> and for low frequency it will be sourced from PLLP.
>>>> Alright, then please don't forget to pre-initialize PLLP_OUT4 rate to a
>>>> reasonable value using tegra_clk_init_table or assigned-clocks.
>>> PLLP_OUT4 rate update is not needed as it is safe to run at 408Mhz
>>> because it is below fmax @ Vmin
>> So even 204MHz CVB entries are having the same voltage as 408MHz,
>> correct? It's not instantly obvious to me from the DFLL driver's code
>> where the fmax @ Vmin is defined, I see that there is the min_millivolts
>> and frequency entries starting from 204MHZ defined per-table.
> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will work at
> Vmin voltage and PLLP max is 408Mhz.

Thank you for the clarification. It would be good to have that commented
in the code as well.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 22:00                                               ` Dmitry Osipenko
@ 2019-07-16 22:06                                                 ` Sowjanya Komatineni
  2019-07-17  2:18                                                   ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-16 22:06 UTC (permalink / raw)
  To: Dmitry Osipenko, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree


On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>> The other thing that also need attention is that T124
>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed first,
>>>>>>>>>>>>>>>> which is
>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Should I add check for successful dfll clk register
>>>>>>>>>>>>>>> explicitly in
>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk registers?
>>>>>>>>>>>> Probably you should use the "device links". See [1][2] for the
>>>>>>>>>>>> example.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [2]
>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if device_link_add()
>>>>>>>>>>>> fails.
>>>>>>>>>>>> And
>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's device, see [3].
>>>>>>>>>>>>
>>>>>>>>>>>> [3]
>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Will go thru and add...
>>>>>>>>> Looks like I initially confused this case with getting orphaned
>>>>>>>>> clock.
>>>>>>>>> I'm now seeing that the DFLL driver registers the clock and then
>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until DFLL driver is
>>>>>>>>> probed, hence everything should be fine as-is and there is no real
>>>>>>>>> need
>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>
>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just regarding the
>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU clock
>>>>>>>>>>>>>> sources and
>>>>>>>>>>>>>> integrated with DVFS control logic with the regulator. We
>>>>>>>>>>>>>> will not
>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>> CPU to other clock sources once we switched to DFLL.
>>>>>>>>>>>>>> Because the
>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP
>>>>>>>>>>>>>> table
>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to other sources
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We allow
>>>>>>>>>>>>>> switching to
>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce DFLL freq to
>>>>>>>>>>>> PLLP's
>>>>>>>>>>>> rate before switching to PLLP in order to have a proper CPU
>>>>>>>>>>>> voltage.
>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no need to
>>>>>>>>>>> enforce
>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source to PLLP
>>>>>>>>>>> during
>>>>>>>>>>> suspend
>>>>>>>>>>>
>>>>>>>>>> Sorry, please ignore my above comment. During suspend, need to
>>>>>>>>>> change
>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop mode first and
>>>>>>>>>> then
>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>> Okay.
>>>>>>>>>
>>>>>>>>>>>>>> And I don't exactly understand why we need to switch to
>>>>>>>>>>>>>> PLLP in
>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it the open-loop
>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest of the
>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the sequence to
>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave it on PLL_P.
>>>>>>>>>>>>>> After
>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore the CPU clock
>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to close-loop
>>>>>>>>>>>>>> mode.
>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL parent
>>>>>>>>>>>> during of
>>>>>>>>>>>> the
>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence instead of
>>>>>>>>>>>> having
>>>>>>>>>>>> odd
>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a proper
>>>>>>>>>>>> suspend-resume sequencing of the device drivers. In this case
>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>> driver is the driver that enables DFLL and switches CPU to that
>>>>>>>>>>>> clock
>>>>>>>>>>>> source, which means that this driver is also should be
>>>>>>>>>>>> responsible for
>>>>>>>>>>>> management of the DFLL's state during of suspend/resume
>>>>>>>>>>>> process. If
>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and re-enables it
>>>>>>>>>>>> during
>>>>>>>>>>>> resume, then looks like the CaR driver hacks around DFLL are not
>>>>>>>>>>>> needed.
>>>>>>>>>>>>
>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the patch
>>>>>>>>>>>>>> subject to
>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to me.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as follows
>>>>>>>>>>>>> (assuming
>>>>>>>>>>>>> all
>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>
>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for
>>>>>>>>>>>>> any
>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>
>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g parent is not
>>>>>>>>>> changed to PLLP before changing dfll to open loop mode.
>>>>>>>>>>
>>>>>>>>>> Will add this ...
>>>>>>>>> The CPUFreq driver switches parent to PLLP during the probe,
>>>>>>>>> similar
>>>>>>>>> should be done on suspend.
>>>>>>>>>
>>>>>>>>> I'm also wondering if it's always safe to switch to PLLP in the
>>>>>>>>> probe.
>>>>>>>>> If CPU is running on a lower freq than PLLP, then some other more
>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>
>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher
>>>>>>>> rate
>>>>>>>> so switching to PLL_P during CPUFreq probe prior to dfll clock
>>>>>>>> enable
>>>>>>>> should be safe.
>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a divided output of
>>>>>>> PLLP
>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>
>>>>>>> Probably, realistically, CPU is always running off a fast PLLX during
>>>>>>> boot, but I'm wondering what may happen on KEXEC. I guess ideally
>>>>>>> CPUFreq driver should also have a 'shutdown' callback to teardown
>>>>>>> DFLL
>>>>>>> on a reboot, but likely that there are other clock-related
>>>>>>> problems as
>>>>>>> well that may break KEXEC and thus it is not very important at the
>>>>>>> moment.
>>>>>>>
>>>>>>> [snip]
>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source above I meant
>>>>>> PLL_P_OUT4.
>>>>>>
>>>>>> As per clock policies, PLL_X is always used for high freq like >800Mhz
>>>>>> and for low frequency it will be sourced from PLLP.
>>>>> Alright, then please don't forget to pre-initialize PLLP_OUT4 rate to a
>>>>> reasonable value using tegra_clk_init_table or assigned-clocks.
>>>> PLLP_OUT4 rate update is not needed as it is safe to run at 408Mhz
>>>> because it is below fmax @ Vmin
>>> So even 204MHz CVB entries are having the same voltage as 408MHz,
>>> correct? It's not instantly obvious to me from the DFLL driver's code
>>> where the fmax @ Vmin is defined, I see that there is the min_millivolts
>>> and frequency entries starting from 204MHZ defined per-table.
>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will work at
>> Vmin voltage and PLLP max is 408Mhz.
> Thank you for the clarification. It would be good to have that commented
> in the code as well.
OK, Will add...

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 22:06                                                 ` Sowjanya Komatineni
@ 2019-07-17  2:18                                                   ` Sowjanya Komatineni
  2019-07-17  2:35                                                     ` Sowjanya Komatineni
  2019-07-17  3:54                                                     ` Dmitry Osipenko
  0 siblings, 2 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17  2:18 UTC (permalink / raw)
  To: Dmitry Osipenko, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree


On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>
> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>> The other thing that also need attention is that T124
>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed first,
>>>>>>>>>>>>>>>>> which is
>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Should I add check for successful dfll clk register
>>>>>>>>>>>>>>>> explicitly in
>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk registers?
>>>>>>>>>>>>> Probably you should use the "device links". See [1][2] for 
>>>>>>>>>>>>> the
>>>>>>>>>>>>> example.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383 
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [2]
>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html 
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if device_link_add()
>>>>>>>>>>>>> fails.
>>>>>>>>>>>>> And
>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's device, see 
>>>>>>>>>>>>> [3].
>>>>>>>>>>>>>
>>>>>>>>>>>>> [3]
>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100 
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>> Looks like I initially confused this case with getting orphaned
>>>>>>>>>> clock.
>>>>>>>>>> I'm now seeing that the DFLL driver registers the clock and then
>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until DFLL 
>>>>>>>>>> driver is
>>>>>>>>>> probed, hence everything should be fine as-is and there is no 
>>>>>>>>>> real
>>>>>>>>>> need
>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>
>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just regarding the
>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU clock
>>>>>>>>>>>>>>> sources and
>>>>>>>>>>>>>>> integrated with DVFS control logic with the regulator. We
>>>>>>>>>>>>>>> will not
>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>> CPU to other clock sources once we switched to DFLL.
>>>>>>>>>>>>>>> Because the
>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table (CVB 
>>>>>>>>>>>>>>> or OPP
>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to other sources
>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We allow
>>>>>>>>>>>>>>> switching to
>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce DFLL 
>>>>>>>>>>>>> freq to
>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>> rate before switching to PLLP in order to have a proper CPU
>>>>>>>>>>>>> voltage.
>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no need to
>>>>>>>>>>>> enforce
>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source to PLLP
>>>>>>>>>>>> during
>>>>>>>>>>>> suspend
>>>>>>>>>>>>
>>>>>>>>>>> Sorry, please ignore my above comment. During suspend, need to
>>>>>>>>>>> change
>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop mode first 
>>>>>>>>>>> and
>>>>>>>>>>> then
>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>> Okay.
>>>>>>>>>>
>>>>>>>>>>>>>>> And I don't exactly understand why we need to switch to
>>>>>>>>>>>>>>> PLLP in
>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it the 
>>>>>>>>>>>>>>> open-loop
>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest of the
>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the 
>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave it on 
>>>>>>>>>>>>>>> PLL_P.
>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore the CPU 
>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to close-loop
>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL parent
>>>>>>>>>>>>> during of
>>>>>>>>>>>>> the
>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence instead of
>>>>>>>>>>>>> having
>>>>>>>>>>>>> odd
>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a proper
>>>>>>>>>>>>> suspend-resume sequencing of the device drivers. In this case
>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>> driver is the driver that enables DFLL and switches CPU to 
>>>>>>>>>>>>> that
>>>>>>>>>>>>> clock
>>>>>>>>>>>>> source, which means that this driver is also should be
>>>>>>>>>>>>> responsible for
>>>>>>>>>>>>> management of the DFLL's state during of suspend/resume
>>>>>>>>>>>>> process. If
>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and re-enables it
>>>>>>>>>>>>> during
>>>>>>>>>>>>> resume, then looks like the CaR driver hacks around DFLL 
>>>>>>>>>>>>> are not
>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the patch
>>>>>>>>>>>>>>> subject to
>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as follows
>>>>>>>>>>>>>> (assuming
>>>>>>>>>>>>>> all
>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency is 
>>>>>>>>>>>>>> ok for
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>
>>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g parent 
>>>>>>>>>>> is not
>>>>>>>>>>> changed to PLLP before changing dfll to open loop mode.
>>>>>>>>>>>
>>>>>>>>>>> Will add this ...
>>>>>>>>>> The CPUFreq driver switches parent to PLLP during the probe,
>>>>>>>>>> similar
>>>>>>>>>> should be done on suspend.
>>>>>>>>>>
>>>>>>>>>> I'm also wondering if it's always safe to switch to PLLP in the
>>>>>>>>>> probe.
>>>>>>>>>> If CPU is running on a lower freq than PLLP, then some other 
>>>>>>>>>> more
>>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>>
>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at 
>>>>>>>>> higher
>>>>>>>>> rate
>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to dfll clock
>>>>>>>>> enable
>>>>>>>>> should be safe.
>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a divided 
>>>>>>>> output of
>>>>>>>> PLLP
>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>
>>>>>>>> Probably, realistically, CPU is always running off a fast PLLX 
>>>>>>>> during
>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I guess ideally
>>>>>>>> CPUFreq driver should also have a 'shutdown' callback to teardown
>>>>>>>> DFLL
>>>>>>>> on a reboot, but likely that there are other clock-related
>>>>>>>> problems as
>>>>>>>> well that may break KEXEC and thus it is not very important at the
>>>>>>>> moment.
>>>>>>>>
>>>>>>>> [snip]
>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source above I 
>>>>>>> meant
>>>>>>> PLL_P_OUT4.
>>>>>>>
>>>>>>> As per clock policies, PLL_X is always used for high freq like 
>>>>>>> >800Mhz
>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>> Alright, then please don't forget to pre-initialize PLLP_OUT4 
>>>>>> rate to a
>>>>>> reasonable value using tegra_clk_init_table or assigned-clocks.
>>>>> PLLP_OUT4 rate update is not needed as it is safe to run at 408Mhz
>>>>> because it is below fmax @ Vmin
>>>> So even 204MHz CVB entries are having the same voltage as 408MHz,
>>>> correct? It's not instantly obvious to me from the DFLL driver's code
>>>> where the fmax @ Vmin is defined, I see that there is the 
>>>> min_millivolts
>>>> and frequency entries starting from 204MHZ defined per-table.
>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will work at
>>> Vmin voltage and PLLP max is 408Mhz.
>> Thank you for the clarification. It would be good to have that commented
>> in the code as well.
> OK, Will add...

Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend happens 
very early even before disabling non-boot CPUs and also need to export 
clock driver APIs to CPUFreq.

Was thinking of below way of implementing this...


Clock DFLL driver Suspend:

         - Save CPU clock policy registers, and Perform dfll suspend 
which sets in open loop mode

CPU Freq driver Suspend: does nothing


Clock DFLL driver Resume:

         - Re-init DFLL, Set in Open-Loop mode, restore CPU Clock policy 
registers which actually sets source to DFLL along with other         
CPU Policy register restore.

CPU Freq driver Resume:

         - do clk_prepare_enable which acutally sets DFLL in Closed loop 
mode




^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17  2:18                                                   ` Sowjanya Komatineni
@ 2019-07-17  2:35                                                     ` Sowjanya Komatineni
  2019-07-17  4:11                                                       ` Dmitry Osipenko
  2019-07-17  3:54                                                     ` Dmitry Osipenko
  1 sibling, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17  2:35 UTC (permalink / raw)
  To: Dmitry Osipenko, Peter De Schrijver, Joseph Lo
  Cc: thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree


On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>
> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>
>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>> The other thing that also need attention is that T124
>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed first,
>>>>>>>>>>>>>>>>>> which is
>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk register
>>>>>>>>>>>>>>>>> explicitly in
>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk registers?
>>>>>>>>>>>>>> Probably you should use the "device links". See [1][2] 
>>>>>>>>>>>>>> for the
>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if device_link_add()
>>>>>>>>>>>>>> fails.
>>>>>>>>>>>>>> And
>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's device, 
>>>>>>>>>>>>>> see [3].
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>> Looks like I initially confused this case with getting orphaned
>>>>>>>>>>> clock.
>>>>>>>>>>> I'm now seeing that the DFLL driver registers the clock and 
>>>>>>>>>>> then
>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until DFLL 
>>>>>>>>>>> driver is
>>>>>>>>>>> probed, hence everything should be fine as-is and there is 
>>>>>>>>>>> no real
>>>>>>>>>>> need
>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just regarding the
>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU clock
>>>>>>>>>>>>>>>> sources and
>>>>>>>>>>>>>>>> integrated with DVFS control logic with the regulator. We
>>>>>>>>>>>>>>>> will not
>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to DFLL.
>>>>>>>>>>>>>>>> Because the
>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table (CVB 
>>>>>>>>>>>>>>>> or OPP
>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to other sources
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We allow
>>>>>>>>>>>>>>>> switching to
>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce DFLL 
>>>>>>>>>>>>>> freq to
>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>> rate before switching to PLLP in order to have a proper CPU
>>>>>>>>>>>>>> voltage.
>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no need to
>>>>>>>>>>>>> enforce
>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source to PLLP
>>>>>>>>>>>>> during
>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>
>>>>>>>>>>>> Sorry, please ignore my above comment. During suspend, need to
>>>>>>>>>>>> change
>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop mode 
>>>>>>>>>>>> first and
>>>>>>>>>>>> then
>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>> Okay.
>>>>>>>>>>>
>>>>>>>>>>>>>>>> And I don't exactly understand why we need to switch to
>>>>>>>>>>>>>>>> PLLP in
>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it the 
>>>>>>>>>>>>>>>> open-loop
>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest of the
>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the 
>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave it on 
>>>>>>>>>>>>>>>> PLL_P.
>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore the 
>>>>>>>>>>>>>>>> CPU clock
>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to close-loop
>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL parent
>>>>>>>>>>>>>> during of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence instead of
>>>>>>>>>>>>>> having
>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a proper
>>>>>>>>>>>>>> suspend-resume sequencing of the device drivers. In this 
>>>>>>>>>>>>>> case
>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>> driver is the driver that enables DFLL and switches CPU 
>>>>>>>>>>>>>> to that
>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>> source, which means that this driver is also should be
>>>>>>>>>>>>>> responsible for
>>>>>>>>>>>>>> management of the DFLL's state during of suspend/resume
>>>>>>>>>>>>>> process. If
>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and 
>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>> during
>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks around DFLL 
>>>>>>>>>>>>>> are not
>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the patch
>>>>>>>>>>>>>>>> subject to
>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to me.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as follows
>>>>>>>>>>>>>>> (assuming
>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency is 
>>>>>>>>>>>>>>> ok for
>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>
>>>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g parent 
>>>>>>>>>>>> is not
>>>>>>>>>>>> changed to PLLP before changing dfll to open loop mode.
>>>>>>>>>>>>
>>>>>>>>>>>> Will add this ...
>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during the probe,
>>>>>>>>>>> similar
>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>
>>>>>>>>>>> I'm also wondering if it's always safe to switch to PLLP in the
>>>>>>>>>>> probe.
>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then some other 
>>>>>>>>>>> more
>>>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>>>
>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at 
>>>>>>>>>> higher
>>>>>>>>>> rate
>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to dfll clock
>>>>>>>>>> enable
>>>>>>>>>> should be safe.
>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a divided 
>>>>>>>>> output of
>>>>>>>>> PLLP
>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>
>>>>>>>>> Probably, realistically, CPU is always running off a fast PLLX 
>>>>>>>>> during
>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I guess ideally
>>>>>>>>> CPUFreq driver should also have a 'shutdown' callback to teardown
>>>>>>>>> DFLL
>>>>>>>>> on a reboot, but likely that there are other clock-related
>>>>>>>>> problems as
>>>>>>>>> well that may break KEXEC and thus it is not very important at 
>>>>>>>>> the
>>>>>>>>> moment.
>>>>>>>>>
>>>>>>>>> [snip]
>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source above I 
>>>>>>>> meant
>>>>>>>> PLL_P_OUT4.
>>>>>>>>
>>>>>>>> As per clock policies, PLL_X is always used for high freq like 
>>>>>>>> >800Mhz
>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>> Alright, then please don't forget to pre-initialize PLLP_OUT4 
>>>>>>> rate to a
>>>>>>> reasonable value using tegra_clk_init_table or assigned-clocks.
>>>>>> PLLP_OUT4 rate update is not needed as it is safe to run at 408Mhz
>>>>>> because it is below fmax @ Vmin
>>>>> So even 204MHz CVB entries are having the same voltage as 408MHz,
>>>>> correct? It's not instantly obvious to me from the DFLL driver's code
>>>>> where the fmax @ Vmin is defined, I see that there is the 
>>>>> min_millivolts
>>>>> and frequency entries starting from 204MHZ defined per-table.
>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will work at
>>>> Vmin voltage and PLLP max is 408Mhz.
>>> Thank you for the clarification. It would be good to have that 
>>> commented
>>> in the code as well.
>> OK, Will add...
>
> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend happens 
> very early even before disabling non-boot CPUs and also need to export 
> clock driver APIs to CPUFreq.
>
> Was thinking of below way of implementing this...
>
>
> Clock DFLL driver Suspend:
>
>         - Save CPU clock policy registers, and Perform dfll suspend 
> which sets in open loop mode
>
> CPU Freq driver Suspend: does nothing
>
>
> Clock DFLL driver Resume:
>
>         - Re-init DFLL, Set in Open-Loop mode, restore CPU Clock 
> policy registers which actually sets source to DFLL along with other 
>         CPU Policy register restore.
>
> CPU Freq driver Resume:
>
>         - do clk_prepare_enable which acutally sets DFLL in Closed 
> loop mode
>
>
> Adding one more note: Switching CPU Clock to PLLP is not needed as CPU 
> CLock can be from dfll in open-loop mode as DFLL is not disabled 
> anywhere throught the suspend/resume path and SC7 entry FW and Warm 
> boot code will switch CPU source to PLLP.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17  2:18                                                   ` Sowjanya Komatineni
  2019-07-17  2:35                                                     ` Sowjanya Komatineni
@ 2019-07-17  3:54                                                     ` Dmitry Osipenko
  2019-07-17  4:01                                                       ` Sowjanya Komatineni
  1 sibling, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-17  3:54 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree

В Tue, 16 Jul 2019 19:18:19 -0700
Sowjanya Komatineni <skomatineni@nvidia.com> пишет:

> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
> >
> > On 7/16/19 3:00 PM, Dmitry Osipenko wrote:  
> >> 17.07.2019 0:35, Sowjanya Komatineni пишет:  
> >>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:  
> >>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:  
> >>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:  
> >>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:  
> >>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:  
> >>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:  
> >>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:  
> >>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:  
> >>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:  
> >>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:  
> >>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:  
> >>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo
> >>>>>>>>>>>>>> wrote:  
> >>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...  
> >>>>>>>>>>>>>>>>> The other thing that also need attention is that
> >>>>>>>>>>>>>>>>> T124 CPUFreq
> >>>>>>>>>>>>>>>>> driver
> >>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed first,
> >>>>>>>>>>>>>>>>> which is
> >>>>>>>>>>>>>>>>> icky.
> >>>>>>>>>>>>>>>>>  
> >>>>>>>>>>>>>>>> Should I add check for successful dfll clk register
> >>>>>>>>>>>>>>>> explicitly in
> >>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
> >>>>>>>>>>>>>>>> registers?  
> >>>>>>>>>>>>> Probably you should use the "device links". See [1][2]
> >>>>>>>>>>>>> for the
> >>>>>>>>>>>>> example.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [1]
> >>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383 
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [2]
> >>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html 
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
> >>>>>>>>>>>>> device_link_add() fails.
> >>>>>>>>>>>>> And
> >>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's device,
> >>>>>>>>>>>>> see [3].
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [3]
> >>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100 
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>  
> >>>>>>>>>>>> Will go thru and add...  
> >>>>>>>>>> Looks like I initially confused this case with getting
> >>>>>>>>>> orphaned clock.
> >>>>>>>>>> I'm now seeing that the DFLL driver registers the clock
> >>>>>>>>>> and then clk_get(dfll) should be returning EPROBE_DEFER
> >>>>>>>>>> until DFLL driver is
> >>>>>>>>>> probed, hence everything should be fine as-is and there is
> >>>>>>>>>> no real
> >>>>>>>>>> need
> >>>>>>>>>> for the 'device link'. Sorry for the confusion!
> >>>>>>>>>>  
> >>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
> >>>>>>>>>>>>>>> regarding the DFLL
> >>>>>>>>>>>>>>> part.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU clock
> >>>>>>>>>>>>>>> sources and
> >>>>>>>>>>>>>>> integrated with DVFS control logic with the
> >>>>>>>>>>>>>>> regulator. We will not
> >>>>>>>>>>>>>>> switch
> >>>>>>>>>>>>>>> CPU to other clock sources once we switched to DFLL.
> >>>>>>>>>>>>>>> Because the
> >>>>>>>>>>>>>>> CPU has
> >>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table
> >>>>>>>>>>>>>>> (CVB or OPP
> >>>>>>>>>>>>>>> table
> >>>>>>>>>>>>>>> you see
> >>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to other
> >>>>>>>>>>>>>>> sources with
> >>>>>>>>>>>>>>> unknew
> >>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We
> >>>>>>>>>>>>>>> allow switching to
> >>>>>>>>>>>>>>> open-loop mode but different sources.  
> >>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce DFLL 
> >>>>>>>>>>>>> freq to
> >>>>>>>>>>>>> PLLP's
> >>>>>>>>>>>>> rate before switching to PLLP in order to have a proper
> >>>>>>>>>>>>> CPU voltage.  
> >>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no
> >>>>>>>>>>>> need to enforce
> >>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source to
> >>>>>>>>>>>> PLLP during
> >>>>>>>>>>>> suspend
> >>>>>>>>>>>>  
> >>>>>>>>>>> Sorry, please ignore my above comment. During suspend,
> >>>>>>>>>>> need to change
> >>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop mode
> >>>>>>>>>>> first and
> >>>>>>>>>>> then
> >>>>>>>>>>> dfll need to be set to open loop.  
> >>>>>>>>>> Okay.
> >>>>>>>>>>  
> >>>>>>>>>>>>>>> And I don't exactly understand why we need to switch
> >>>>>>>>>>>>>>> to PLLP in
> >>>>>>>>>>>>>>> CPU
> >>>>>>>>>>>>>>> idle
> >>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it the 
> >>>>>>>>>>>>>>> open-loop
> >>>>>>>>>>>>>>> mode. That's
> >>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest of the
> >>>>>>>>>>>>>>> sequence to
> >>>>>>>>>>>>>>> turn off
> >>>>>>>>>>>>>>> the CPU power.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the 
> >>>>>>>>>>>>>>> sequence to
> >>>>>>>>>>>>>>> turn on
> >>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave it
> >>>>>>>>>>>>>>> on PLL_P.
> >>>>>>>>>>>>>>> After
> >>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore the
> >>>>>>>>>>>>>>> CPU clock
> >>>>>>>>>>>>>>> policy (CPU
> >>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
> >>>>>>>>>>>>>>> close-loop mode.  
> >>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL
> >>>>>>>>>>>>> parent during of
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
> >>>>>>>>>>>>> instead of having
> >>>>>>>>>>>>> odd
> >>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a
> >>>>>>>>>>>>> proper suspend-resume sequencing of the device drivers.
> >>>>>>>>>>>>> In this case CPUFreq
> >>>>>>>>>>>>> driver is the driver that enables DFLL and switches CPU
> >>>>>>>>>>>>> to that
> >>>>>>>>>>>>> clock
> >>>>>>>>>>>>> source, which means that this driver is also should be
> >>>>>>>>>>>>> responsible for
> >>>>>>>>>>>>> management of the DFLL's state during of suspend/resume
> >>>>>>>>>>>>> process. If
> >>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
> >>>>>>>>>>>>> re-enables it during
> >>>>>>>>>>>>> resume, then looks like the CaR driver hacks around
> >>>>>>>>>>>>> DFLL are not
> >>>>>>>>>>>>> needed.
> >>>>>>>>>>>>>  
> >>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the patch
> >>>>>>>>>>>>>>> subject to
> >>>>>>>>>>>>>>> "Add
> >>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to me.
> >>>>>>>>>>>>>>>  
> >>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as
> >>>>>>>>>>>>>> follows (assuming
> >>>>>>>>>>>>>> all
> >>>>>>>>>>>>>> required DFLL hw configuration has been done)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Switch to DFLL:
> >>>>>>>>>>>>>> 0) Save current parent and frequency
> >>>>>>>>>>>>>> 1) Program DFLL to open loop mode
> >>>>>>>>>>>>>> 2) Enable DFLL
> >>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
> >>>>>>>>>>>>>> For OVR regulator:
> >>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
> >>>>>>>>>>>>>> 5) Enable DFLL PWM output
> >>>>>>>>>>>>>> For I2C regulator:
> >>>>>>>>>>>>>> 4) Enable DFLL I2C output
> >>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Switch away from DFLL:
> >>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency
> >>>>>>>>>>>>>> is ok for
> >>>>>>>>>>>>>> any
> >>>>>>>>>>>>>> vdd_cpu voltage
> >>>>>>>>>>>>>> 1) Program DFLL to open loop mode
> >>>>>>>>>>>>>>  
> >>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g
> >>>>>>>>>>> parent is not
> >>>>>>>>>>> changed to PLLP before changing dfll to open loop mode.
> >>>>>>>>>>>
> >>>>>>>>>>> Will add this ...  
> >>>>>>>>>> The CPUFreq driver switches parent to PLLP during the
> >>>>>>>>>> probe, similar
> >>>>>>>>>> should be done on suspend.
> >>>>>>>>>>
> >>>>>>>>>> I'm also wondering if it's always safe to switch to PLLP
> >>>>>>>>>> in the probe.
> >>>>>>>>>> If CPU is running on a lower freq than PLLP, then some
> >>>>>>>>>> other more
> >>>>>>>>>> appropriate intermediate parent should be selected.
> >>>>>>>>>>  
> >>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs
> >>>>>>>>> at higher
> >>>>>>>>> rate
> >>>>>>>>> so switching to PLL_P during CPUFreq probe prior to dfll
> >>>>>>>>> clock enable
> >>>>>>>>> should be safe.  
> >>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a divided 
> >>>>>>>> output of
> >>>>>>>> PLLP
> >>>>>>>> which CCLKG supports, the PLLP_OUT4.
> >>>>>>>>
> >>>>>>>> Probably, realistically, CPU is always running off a fast
> >>>>>>>> PLLX during
> >>>>>>>> boot, but I'm wondering what may happen on KEXEC. I guess
> >>>>>>>> ideally CPUFreq driver should also have a 'shutdown'
> >>>>>>>> callback to teardown DFLL
> >>>>>>>> on a reboot, but likely that there are other clock-related
> >>>>>>>> problems as
> >>>>>>>> well that may break KEXEC and thus it is not very important
> >>>>>>>> at the moment.
> >>>>>>>>
> >>>>>>>> [snip]  
> >>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source above
> >>>>>>> I meant
> >>>>>>> PLL_P_OUT4.
> >>>>>>>
> >>>>>>> As per clock policies, PLL_X is always used for high freq
> >>>>>>> like   
> >>>>>>> >800Mhz  
> >>>>>>> and for low frequency it will be sourced from PLLP.  
> >>>>>> Alright, then please don't forget to pre-initialize PLLP_OUT4 
> >>>>>> rate to a
> >>>>>> reasonable value using tegra_clk_init_table or
> >>>>>> assigned-clocks.  
> >>>>> PLLP_OUT4 rate update is not needed as it is safe to run at
> >>>>> 408Mhz because it is below fmax @ Vmin  
> >>>> So even 204MHz CVB entries are having the same voltage as 408MHz,
> >>>> correct? It's not instantly obvious to me from the DFLL driver's
> >>>> code where the fmax @ Vmin is defined, I see that there is the 
> >>>> min_millivolts
> >>>> and frequency entries starting from 204MHZ defined per-table.  
> >>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will work
> >>> at Vmin voltage and PLLP max is 408Mhz.  
> >> Thank you for the clarification. It would be good to have that
> >> commented in the code as well.  
> > OK, Will add...  
> 
> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend happens 
> very early even before disabling non-boot CPUs and also need to
> export clock driver APIs to CPUFreq.
> 
> Was thinking of below way of implementing this...
> 
> 
> Clock DFLL driver Suspend:
> 
>          - Save CPU clock policy registers, and Perform dfll suspend 
> which sets in open loop mode
> 
> CPU Freq driver Suspend: does nothing
> 
> 
> Clock DFLL driver Resume:
> 
>          - Re-init DFLL, Set in Open-Loop mode, restore CPU Clock
> policy registers which actually sets source to DFLL along with other
> CPU Policy register restore.
> 
> CPU Freq driver Resume:
> 
>          - do clk_prepare_enable which acutally sets DFLL in Closed
> loop mode


It doesn't matter much when CPUFreq driver suspends, it's only
important that it suspends before CaR.

I'm not sure why do you need anything else from DFLL driver other than
what is already exposed via generic CCF API. It looks to me
that switching CPU's parent clock away from DFLL and then disabling
DFLL's clock is enough for suspend, accordingly to what Peter wrote. And
resuming is the same as what's done on CPUFreq's driver probe. The CCLK
policy should be saved and restored by the CaR driver, you don't need to
care about it. The cpufreq-dt driver sets
CPUFREQ_NEED_INITIAL_FREQ_CHECK, hence you don't need to care about
restoring the original CPU freq on resume, IIUC.



^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17  3:54                                                     ` Dmitry Osipenko
@ 2019-07-17  4:01                                                       ` Sowjanya Komatineni
  0 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17  4:01 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree


On 7/16/19 8:54 PM, Dmitry Osipenko wrote:
> В Tue, 16 Jul 2019 19:18:19 -0700
> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>
>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>> The other thing that also need attention is that
>>>>>>>>>>>>>>>>>>> T124 CPUFreq
>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed first,
>>>>>>>>>>>>>>>>>>> which is
>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>   
>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk register
>>>>>>>>>>>>>>>>>> explicitly in
>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>> Probably you should use the "device links". See [1][2]
>>>>>>>>>>>>>>> for the
>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's device,
>>>>>>>>>>>>>>> see [3].
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>   
>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>> Looks like I initially confused this case with getting
>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the clock
>>>>>>>>>>>> and then clk_get(dfll) should be returning EPROBE_DEFER
>>>>>>>>>>>> until DFLL driver is
>>>>>>>>>>>> probed, hence everything should be fine as-is and there is
>>>>>>>>>>>> no real
>>>>>>>>>>>> need
>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>   
>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU clock
>>>>>>>>>>>>>>>>> sources and
>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to DFLL.
>>>>>>>>>>>>>>>>> Because the
>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table
>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to other
>>>>>>>>>>>>>>>>> sources with
>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We
>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce DFLL
>>>>>>>>>>>>>>> freq to
>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>> rate before switching to PLLP in order to have a proper
>>>>>>>>>>>>>>> CPU voltage.
>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no
>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source to
>>>>>>>>>>>>>> PLLP during
>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>   
>>>>>>>>>>>>> Sorry, please ignore my above comment. During suspend,
>>>>>>>>>>>>> need to change
>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop mode
>>>>>>>>>>>>> first and
>>>>>>>>>>>>> then
>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>> Okay.
>>>>>>>>>>>>   
>>>>>>>>>>>>>>>>> And I don't exactly understand why we need to switch
>>>>>>>>>>>>>>>>> to PLLP in
>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it the
>>>>>>>>>>>>>>>>> open-loop
>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest of the
>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the
>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave it
>>>>>>>>>>>>>>>>> on PLL_P.
>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore the
>>>>>>>>>>>>>>>>> CPU clock
>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL
>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a
>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device drivers.
>>>>>>>>>>>>>>> In this case CPUFreq
>>>>>>>>>>>>>>> driver is the driver that enables DFLL and switches CPU
>>>>>>>>>>>>>>> to that
>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>> source, which means that this driver is also should be
>>>>>>>>>>>>>>> responsible for
>>>>>>>>>>>>>>> management of the DFLL's state during of suspend/resume
>>>>>>>>>>>>>>> process. If
>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
>>>>>>>>>>>>>>> re-enables it during
>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks around
>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>   
>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the patch
>>>>>>>>>>>>>>>>> subject to
>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to me.
>>>>>>>>>>>>>>>>>   
>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as
>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency
>>>>>>>>>>>>>>>> is ok for
>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>   
>>>>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g
>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>> changed to PLLP before changing dfll to open loop mode.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during the
>>>>>>>>>>>> probe, similar
>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm also wondering if it's always safe to switch to PLLP
>>>>>>>>>>>> in the probe.
>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then some
>>>>>>>>>>>> other more
>>>>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>>>>   
>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs
>>>>>>>>>>> at higher
>>>>>>>>>>> rate
>>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to dfll
>>>>>>>>>>> clock enable
>>>>>>>>>>> should be safe.
>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a divided
>>>>>>>>>> output of
>>>>>>>>>> PLLP
>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>
>>>>>>>>>> Probably, realistically, CPU is always running off a fast
>>>>>>>>>> PLLX during
>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I guess
>>>>>>>>>> ideally CPUFreq driver should also have a 'shutdown'
>>>>>>>>>> callback to teardown DFLL
>>>>>>>>>> on a reboot, but likely that there are other clock-related
>>>>>>>>>> problems as
>>>>>>>>>> well that may break KEXEC and thus it is not very important
>>>>>>>>>> at the moment.
>>>>>>>>>>
>>>>>>>>>> [snip]
>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source above
>>>>>>>>> I meant
>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>
>>>>>>>>> As per clock policies, PLL_X is always used for high freq
>>>>>>>>> like
>>>>>>>>>> 800Mhz
>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>> Alright, then please don't forget to pre-initialize PLLP_OUT4
>>>>>>>> rate to a
>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>> assigned-clocks.
>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to run at
>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>> So even 204MHz CVB entries are having the same voltage as 408MHz,
>>>>>> correct? It's not instantly obvious to me from the DFLL driver's
>>>>>> code where the fmax @ Vmin is defined, I see that there is the
>>>>>> min_millivolts
>>>>>> and frequency entries starting from 204MHZ defined per-table.
>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will work
>>>>> at Vmin voltage and PLLP max is 408Mhz.
>>>> Thank you for the clarification. It would be good to have that
>>>> commented in the code as well.
>>> OK, Will add...
>> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend happens
>> very early even before disabling non-boot CPUs and also need to
>> export clock driver APIs to CPUFreq.
>>
>> Was thinking of below way of implementing this...
>>
>>
>> Clock DFLL driver Suspend:
>>
>>           - Save CPU clock policy registers, and Perform dfll suspend
>> which sets in open loop mode
>>
>> CPU Freq driver Suspend: does nothing
>>
>>
>> Clock DFLL driver Resume:
>>
>>           - Re-init DFLL, Set in Open-Loop mode, restore CPU Clock
>> policy registers which actually sets source to DFLL along with other
>> CPU Policy register restore.
>>
>> CPU Freq driver Resume:
>>
>>           - do clk_prepare_enable which acutally sets DFLL in Closed
>> loop mode
>
> It doesn't matter much when CPUFreq driver suspends, it's only
> important that it suspends before CaR.
>
> I'm not sure why do you need anything else from DFLL driver other than
> what is already exposed via generic CCF API. It looks to me
> that switching CPU's parent clock away from DFLL and then disabling
> DFLL's clock is enough for suspend, accordingly to what Peter wrote. And
> resuming is the same as what's done on CPUFreq's driver probe. The CCLK
> policy should be saved and restored by the CaR driver, you don't need to
> care about it. The cpufreq-dt driver sets
> CPUFREQ_NEED_INITIAL_FREQ_CHECK, hence you don't need to care about
> restoring the original CPU freq on resume, IIUC.
>
>
CPU policy save/restore are part of this patch and it wasn't there 
earlier and when I moved suspend/resume to clock-dfll I moved cpu 
restore also to clock-dfll driver.

Thought to restore cpu source to dfll in CPUFreq driver which requires 
exporting API from clock driver. But leaving CPU restore in Tegra210 
clock driver, we can do dfll suspend/resume thru CPUFreq driver pm_ops 
without exporting clock APIs

Will update ...

> With

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17  2:35                                                     ` Sowjanya Komatineni
@ 2019-07-17  4:11                                                       ` Dmitry Osipenko
       [not found]                                                         ` <77df234f-aa40-0319-a593-f1f19f0f1c2a@nvidia.com>
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-17  4:11 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree

В Tue, 16 Jul 2019 19:35:49 -0700
Sowjanya Komatineni <skomatineni@nvidia.com> пишет:

> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
> >
> > On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:  
> >>
> >> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:  
> >>> 17.07.2019 0:35, Sowjanya Komatineni пишет:  
> >>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:  
> >>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:  
> >>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:  
> >>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:  
> >>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:  
> >>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:  
> >>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:  
> >>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:  
> >>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:  
> >>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:  
> >>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:  
> >>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo
> >>>>>>>>>>>>>>> wrote:  
> >>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...  
> >>>>>>>>>>>>>>>>>> The other thing that also need attention is that
> >>>>>>>>>>>>>>>>>> T124 CPUFreq
> >>>>>>>>>>>>>>>>>> driver
> >>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed
> >>>>>>>>>>>>>>>>>> first, which is
> >>>>>>>>>>>>>>>>>> icky.
> >>>>>>>>>>>>>>>>>>  
> >>>>>>>>>>>>>>>>> Should I add check for successful dfll clk register
> >>>>>>>>>>>>>>>>> explicitly in
> >>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
> >>>>>>>>>>>>>>>>> registers?  
> >>>>>>>>>>>>>> Probably you should use the "device links". See [1][2] 
> >>>>>>>>>>>>>> for the
> >>>>>>>>>>>>>> example.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383 
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [2]
> >>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html 
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
> >>>>>>>>>>>>>> device_link_add() fails.
> >>>>>>>>>>>>>> And
> >>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's device, 
> >>>>>>>>>>>>>> see [3].
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [3]
> >>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100 
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>  
> >>>>>>>>>>>>> Will go thru and add...  
> >>>>>>>>>>> Looks like I initially confused this case with getting
> >>>>>>>>>>> orphaned clock.
> >>>>>>>>>>> I'm now seeing that the DFLL driver registers the clock
> >>>>>>>>>>> and then
> >>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until DFLL 
> >>>>>>>>>>> driver is
> >>>>>>>>>>> probed, hence everything should be fine as-is and there
> >>>>>>>>>>> is no real
> >>>>>>>>>>> need
> >>>>>>>>>>> for the 'device link'. Sorry for the confusion!
> >>>>>>>>>>>  
> >>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
> >>>>>>>>>>>>>>>> regarding the DFLL
> >>>>>>>>>>>>>>>> part.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU
> >>>>>>>>>>>>>>>> clock sources and
> >>>>>>>>>>>>>>>> integrated with DVFS control logic with the
> >>>>>>>>>>>>>>>> regulator. We will not
> >>>>>>>>>>>>>>>> switch
> >>>>>>>>>>>>>>>> CPU to other clock sources once we switched to DFLL.
> >>>>>>>>>>>>>>>> Because the
> >>>>>>>>>>>>>>>> CPU has
> >>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table
> >>>>>>>>>>>>>>>> (CVB or OPP
> >>>>>>>>>>>>>>>> table
> >>>>>>>>>>>>>>>> you see
> >>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to other
> >>>>>>>>>>>>>>>> sources with
> >>>>>>>>>>>>>>>> unknew
> >>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We
> >>>>>>>>>>>>>>>> allow switching to
> >>>>>>>>>>>>>>>> open-loop mode but different sources.  
> >>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce
> >>>>>>>>>>>>>> DFLL freq to
> >>>>>>>>>>>>>> PLLP's
> >>>>>>>>>>>>>> rate before switching to PLLP in order to have a
> >>>>>>>>>>>>>> proper CPU voltage.  
> >>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no
> >>>>>>>>>>>>> need to enforce
> >>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source to
> >>>>>>>>>>>>> PLLP during
> >>>>>>>>>>>>> suspend
> >>>>>>>>>>>>>  
> >>>>>>>>>>>> Sorry, please ignore my above comment. During suspend,
> >>>>>>>>>>>> need to change
> >>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop mode 
> >>>>>>>>>>>> first and
> >>>>>>>>>>>> then
> >>>>>>>>>>>> dfll need to be set to open loop.  
> >>>>>>>>>>> Okay.
> >>>>>>>>>>>  
> >>>>>>>>>>>>>>>> And I don't exactly understand why we need to switch
> >>>>>>>>>>>>>>>> to PLLP in
> >>>>>>>>>>>>>>>> CPU
> >>>>>>>>>>>>>>>> idle
> >>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it the 
> >>>>>>>>>>>>>>>> open-loop
> >>>>>>>>>>>>>>>> mode. That's
> >>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest of
> >>>>>>>>>>>>>>>> the sequence to
> >>>>>>>>>>>>>>>> turn off
> >>>>>>>>>>>>>>>> the CPU power.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the 
> >>>>>>>>>>>>>>>> sequence to
> >>>>>>>>>>>>>>>> turn on
> >>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave it
> >>>>>>>>>>>>>>>> on PLL_P.
> >>>>>>>>>>>>>>>> After
> >>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore the 
> >>>>>>>>>>>>>>>> CPU clock
> >>>>>>>>>>>>>>>> policy (CPU
> >>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
> >>>>>>>>>>>>>>>> close-loop mode.  
> >>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL
> >>>>>>>>>>>>>> parent during of
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
> >>>>>>>>>>>>>> instead of having
> >>>>>>>>>>>>>> odd
> >>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a
> >>>>>>>>>>>>>> proper suspend-resume sequencing of the device
> >>>>>>>>>>>>>> drivers. In this case
> >>>>>>>>>>>>>> CPUFreq
> >>>>>>>>>>>>>> driver is the driver that enables DFLL and switches
> >>>>>>>>>>>>>> CPU to that
> >>>>>>>>>>>>>> clock
> >>>>>>>>>>>>>> source, which means that this driver is also should be
> >>>>>>>>>>>>>> responsible for
> >>>>>>>>>>>>>> management of the DFLL's state during of suspend/resume
> >>>>>>>>>>>>>> process. If
> >>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and 
> >>>>>>>>>>>>>> re-enables it
> >>>>>>>>>>>>>> during
> >>>>>>>>>>>>>> resume, then looks like the CaR driver hacks around
> >>>>>>>>>>>>>> DFLL are not
> >>>>>>>>>>>>>> needed.
> >>>>>>>>>>>>>>  
> >>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the patch
> >>>>>>>>>>>>>>>> subject to
> >>>>>>>>>>>>>>>> "Add
> >>>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to me.
> >>>>>>>>>>>>>>>>  
> >>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as
> >>>>>>>>>>>>>>> follows (assuming
> >>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>> required DFLL hw configuration has been done)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Switch to DFLL:
> >>>>>>>>>>>>>>> 0) Save current parent and frequency
> >>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
> >>>>>>>>>>>>>>> 2) Enable DFLL
> >>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
> >>>>>>>>>>>>>>> For OVR regulator:
> >>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
> >>>>>>>>>>>>>>> 5) Enable DFLL PWM output
> >>>>>>>>>>>>>>> For I2C regulator:
> >>>>>>>>>>>>>>> 4) Enable DFLL I2C output
> >>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Switch away from DFLL:
> >>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency
> >>>>>>>>>>>>>>> is ok for
> >>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>> vdd_cpu voltage
> >>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
> >>>>>>>>>>>>>>>  
> >>>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g
> >>>>>>>>>>>> parent is not
> >>>>>>>>>>>> changed to PLLP before changing dfll to open loop mode.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Will add this ...  
> >>>>>>>>>>> The CPUFreq driver switches parent to PLLP during the
> >>>>>>>>>>> probe, similar
> >>>>>>>>>>> should be done on suspend.
> >>>>>>>>>>>
> >>>>>>>>>>> I'm also wondering if it's always safe to switch to PLLP
> >>>>>>>>>>> in the probe.
> >>>>>>>>>>> If CPU is running on a lower freq than PLLP, then some
> >>>>>>>>>>> other more
> >>>>>>>>>>> appropriate intermediate parent should be selected.
> >>>>>>>>>>>  
> >>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs
> >>>>>>>>>> at higher
> >>>>>>>>>> rate
> >>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to dfll
> >>>>>>>>>> clock enable
> >>>>>>>>>> should be safe.  
> >>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a divided 
> >>>>>>>>> output of
> >>>>>>>>> PLLP
> >>>>>>>>> which CCLKG supports, the PLLP_OUT4.
> >>>>>>>>>
> >>>>>>>>> Probably, realistically, CPU is always running off a fast
> >>>>>>>>> PLLX during
> >>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I guess
> >>>>>>>>> ideally CPUFreq driver should also have a 'shutdown'
> >>>>>>>>> callback to teardown DFLL
> >>>>>>>>> on a reboot, but likely that there are other clock-related
> >>>>>>>>> problems as
> >>>>>>>>> well that may break KEXEC and thus it is not very important
> >>>>>>>>> at the
> >>>>>>>>> moment.
> >>>>>>>>>
> >>>>>>>>> [snip]  
> >>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source above
> >>>>>>>> I meant
> >>>>>>>> PLL_P_OUT4.
> >>>>>>>>
> >>>>>>>> As per clock policies, PLL_X is always used for high freq
> >>>>>>>> like   
> >>>>>>>> >800Mhz  
> >>>>>>>> and for low frequency it will be sourced from PLLP.  
> >>>>>>> Alright, then please don't forget to pre-initialize PLLP_OUT4 
> >>>>>>> rate to a
> >>>>>>> reasonable value using tegra_clk_init_table or
> >>>>>>> assigned-clocks.  
> >>>>>> PLLP_OUT4 rate update is not needed as it is safe to run at
> >>>>>> 408Mhz because it is below fmax @ Vmin  
> >>>>> So even 204MHz CVB entries are having the same voltage as
> >>>>> 408MHz, correct? It's not instantly obvious to me from the DFLL
> >>>>> driver's code where the fmax @ Vmin is defined, I see that
> >>>>> there is the min_millivolts
> >>>>> and frequency entries starting from 204MHZ defined per-table.  
> >>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will
> >>>> work at Vmin voltage and PLLP max is 408Mhz.  
> >>> Thank you for the clarification. It would be good to have that 
> >>> commented
> >>> in the code as well.  
> >> OK, Will add...  
> >
> > Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend
> > happens very early even before disabling non-boot CPUs and also
> > need to export clock driver APIs to CPUFreq.
> >
> > Was thinking of below way of implementing this...
> >
> >
> > Clock DFLL driver Suspend:
> >
> >         - Save CPU clock policy registers, and Perform dfll suspend 
> > which sets in open loop mode
> >
> > CPU Freq driver Suspend: does nothing
> >
> >
> > Clock DFLL driver Resume:
> >
> >         - Re-init DFLL, Set in Open-Loop mode, restore CPU Clock 
> > policy registers which actually sets source to DFLL along with
> > other CPU Policy register restore.
> >
> > CPU Freq driver Resume:
> >
> >         - do clk_prepare_enable which acutally sets DFLL in Closed 
> > loop mode
> >
> >
> > Adding one more note: Switching CPU Clock to PLLP is not needed as
> > CPU CLock can be from dfll in open-loop mode as DFLL is not
> > disabled anywhere throught the suspend/resume path and SC7 entry FW
> > and Warm boot code will switch CPU source to PLLP.  
> 

Since CPU resumes on PLLP, it will be cleaner to suspend it on PLLP as
well. And besides, seems that currently disabling DFLL clock will
disable DFLL completely and then you'd want to re-init the DFLL on
resume any ways. So better to just disable DFLL completely on suspend,
which should happen on clk_disable(dfll).

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
       [not found]                                                         ` <77df234f-aa40-0319-a593-f1f19f0f1c2a@nvidia.com>
@ 2019-07-17  5:42                                                           ` Dmitry Osipenko
  2019-07-17  5:55                                                             ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-17  5:42 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree

В Tue, 16 Jul 2019 22:25:25 -0700
Sowjanya Komatineni <skomatineni@nvidia.com> пишет:

> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
> > В Tue, 16 Jul 2019 19:35:49 -0700
> > Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
> >  
> >> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:  
> >>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:  
> >>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:  
> >>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:  
> >>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:  
> >>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:  
> >>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:  
> >>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:  
> >>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:  
> >>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:  
> >>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:  
> >>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:  
> >>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:  
> >>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:  
> >>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:  
> >>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo
> >>>>>>>>>>>>>>>>> wrote:  
> >>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...  
> >>>>>>>>>>>>>>>>>>>> The other thing that also need attention is that
> >>>>>>>>>>>>>>>>>>>> T124 CPUFreq
> >>>>>>>>>>>>>>>>>>>> driver
> >>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed
> >>>>>>>>>>>>>>>>>>>> first, which is
> >>>>>>>>>>>>>>>>>>>> icky.
> >>>>>>>>>>>>>>>>>>>>     
> >>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
> >>>>>>>>>>>>>>>>>>> register explicitly in
> >>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
> >>>>>>>>>>>>>>>>>>> registers?  
> >>>>>>>>>>>>>>>> Probably you should use the "device links". See
> >>>>>>>>>>>>>>>> [1][2] for the
> >>>>>>>>>>>>>>>> example.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [2]
> >>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
> >>>>>>>>>>>>>>>> device_link_add() fails.
> >>>>>>>>>>>>>>>> And
> >>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
> >>>>>>>>>>>>>>>> device, see [3].
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [3]
> >>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>     
> >>>>>>>>>>>>>>> Will go thru and add...  
> >>>>>>>>>>>>> Looks like I initially confused this case with getting
> >>>>>>>>>>>>> orphaned clock.
> >>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the clock
> >>>>>>>>>>>>> and then
> >>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until
> >>>>>>>>>>>>> DFLL driver is
> >>>>>>>>>>>>> probed, hence everything should be fine as-is and there
> >>>>>>>>>>>>> is no real
> >>>>>>>>>>>>> need
> >>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
> >>>>>>>>>>>>>     
> >>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
> >>>>>>>>>>>>>>>>>> regarding the DFLL
> >>>>>>>>>>>>>>>>>> part.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU
> >>>>>>>>>>>>>>>>>> clock sources and
> >>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
> >>>>>>>>>>>>>>>>>> regulator. We will not
> >>>>>>>>>>>>>>>>>> switch
> >>>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to
> >>>>>>>>>>>>>>>>>> DFLL. Because the
> >>>>>>>>>>>>>>>>>> CPU has
> >>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table
> >>>>>>>>>>>>>>>>>> (CVB or OPP
> >>>>>>>>>>>>>>>>>> table
> >>>>>>>>>>>>>>>>>> you see
> >>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to other
> >>>>>>>>>>>>>>>>>> sources with
> >>>>>>>>>>>>>>>>>> unknew
> >>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We
> >>>>>>>>>>>>>>>>>> allow switching to
> >>>>>>>>>>>>>>>>>> open-loop mode but different sources.  
> >>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce
> >>>>>>>>>>>>>>>> DFLL freq to
> >>>>>>>>>>>>>>>> PLLP's
> >>>>>>>>>>>>>>>> rate before switching to PLLP in order to have a
> >>>>>>>>>>>>>>>> proper CPU voltage.  
> >>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no
> >>>>>>>>>>>>>>> need to enforce
> >>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source
> >>>>>>>>>>>>>>> to PLLP during
> >>>>>>>>>>>>>>> suspend
> >>>>>>>>>>>>>>>     
> >>>>>>>>>>>>>> Sorry, please ignore my above comment. During suspend,
> >>>>>>>>>>>>>> need to change
> >>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop mode
> >>>>>>>>>>>>>> first and
> >>>>>>>>>>>>>> then
> >>>>>>>>>>>>>> dfll need to be set to open loop.  
> >>>>>>>>>>>>> Okay.
> >>>>>>>>>>>>>     
> >>>>>>>>>>>>>>>>>> And I don't exactly understand why we need to
> >>>>>>>>>>>>>>>>>> switch to PLLP in
> >>>>>>>>>>>>>>>>>> CPU
> >>>>>>>>>>>>>>>>>> idle
> >>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it
> >>>>>>>>>>>>>>>>>> the open-loop
> >>>>>>>>>>>>>>>>>> mode. That's
> >>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest of
> >>>>>>>>>>>>>>>>>> the sequence to
> >>>>>>>>>>>>>>>>>> turn off
> >>>>>>>>>>>>>>>>>> the CPU power.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the
> >>>>>>>>>>>>>>>>>> sequence to
> >>>>>>>>>>>>>>>>>> turn on
> >>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave
> >>>>>>>>>>>>>>>>>> it on PLL_P.
> >>>>>>>>>>>>>>>>>> After
> >>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore
> >>>>>>>>>>>>>>>>>> the CPU clock
> >>>>>>>>>>>>>>>>>> policy (CPU
> >>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
> >>>>>>>>>>>>>>>>>> close-loop mode.  
> >>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL
> >>>>>>>>>>>>>>>> parent during of
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
> >>>>>>>>>>>>>>>> instead of having
> >>>>>>>>>>>>>>>> odd
> >>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a
> >>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
> >>>>>>>>>>>>>>>> drivers. In this case
> >>>>>>>>>>>>>>>> CPUFreq
> >>>>>>>>>>>>>>>> driver is the driver that enables DFLL and switches
> >>>>>>>>>>>>>>>> CPU to that
> >>>>>>>>>>>>>>>> clock
> >>>>>>>>>>>>>>>> source, which means that this driver is also should
> >>>>>>>>>>>>>>>> be responsible for
> >>>>>>>>>>>>>>>> management of the DFLL's state during of
> >>>>>>>>>>>>>>>> suspend/resume process. If
> >>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
> >>>>>>>>>>>>>>>> re-enables it
> >>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks around
> >>>>>>>>>>>>>>>> DFLL are not
> >>>>>>>>>>>>>>>> needed.
> >>>>>>>>>>>>>>>>     
> >>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the
> >>>>>>>>>>>>>>>>>> patch subject to
> >>>>>>>>>>>>>>>>>> "Add
> >>>>>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to
> >>>>>>>>>>>>>>>>>> me. 
> >>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as
> >>>>>>>>>>>>>>>>> follows (assuming
> >>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Switch to DFLL:
> >>>>>>>>>>>>>>>>> 0) Save current parent and frequency
> >>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
> >>>>>>>>>>>>>>>>> 2) Enable DFLL
> >>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
> >>>>>>>>>>>>>>>>> For OVR regulator:
> >>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
> >>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
> >>>>>>>>>>>>>>>>> For I2C regulator:
> >>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
> >>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Switch away from DFLL:
> >>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency
> >>>>>>>>>>>>>>>>> is ok for
> >>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>> vdd_cpu voltage
> >>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
> >>>>>>>>>>>>>>>>>     
> >>>>>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g
> >>>>>>>>>>>>>> parent is not
> >>>>>>>>>>>>>> changed to PLLP before changing dfll to open loop mode.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Will add this ...  
> >>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during the
> >>>>>>>>>>>>> probe, similar
> >>>>>>>>>>>>> should be done on suspend.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'm also wondering if it's always safe to switch to PLLP
> >>>>>>>>>>>>> in the probe.
> >>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then some
> >>>>>>>>>>>>> other more
> >>>>>>>>>>>>> appropriate intermediate parent should be selected.
> >>>>>>>>>>>>>     
> >>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs
> >>>>>>>>>>>> at higher
> >>>>>>>>>>>> rate
> >>>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to dfll
> >>>>>>>>>>>> clock enable
> >>>>>>>>>>>> should be safe.  
> >>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a divided
> >>>>>>>>>>> output of
> >>>>>>>>>>> PLLP
> >>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
> >>>>>>>>>>>
> >>>>>>>>>>> Probably, realistically, CPU is always running off a fast
> >>>>>>>>>>> PLLX during
> >>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I guess
> >>>>>>>>>>> ideally CPUFreq driver should also have a 'shutdown'
> >>>>>>>>>>> callback to teardown DFLL
> >>>>>>>>>>> on a reboot, but likely that there are other clock-related
> >>>>>>>>>>> problems as
> >>>>>>>>>>> well that may break KEXEC and thus it is not very
> >>>>>>>>>>> important at the
> >>>>>>>>>>> moment.
> >>>>>>>>>>>
> >>>>>>>>>>> [snip]  
> >>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source
> >>>>>>>>>> above I meant
> >>>>>>>>>> PLL_P_OUT4.
> >>>>>>>>>>
> >>>>>>>>>> As per clock policies, PLL_X is always used for high freq
> >>>>>>>>>> like  
> >>>>>>>>>>> 800Mhz  
> >>>>>>>>>> and for low frequency it will be sourced from PLLP.  
> >>>>>>>>> Alright, then please don't forget to pre-initialize
> >>>>>>>>> PLLP_OUT4 rate to a
> >>>>>>>>> reasonable value using tegra_clk_init_table or
> >>>>>>>>> assigned-clocks.  
> >>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to run at
> >>>>>>>> 408Mhz because it is below fmax @ Vmin  
> >>>>>>> So even 204MHz CVB entries are having the same voltage as
> >>>>>>> 408MHz, correct? It's not instantly obvious to me from the
> >>>>>>> DFLL driver's code where the fmax @ Vmin is defined, I see
> >>>>>>> that there is the min_millivolts
> >>>>>>> and frequency entries starting from 204MHZ defined
> >>>>>>> per-table.  
> >>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will
> >>>>>> work at Vmin voltage and PLLP max is 408Mhz.  
> >>>>> Thank you for the clarification. It would be good to have that
> >>>>> commented
> >>>>> in the code as well.  
> >>>> OK, Will add...  
> >>> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend
> >>> happens very early even before disabling non-boot CPUs and also
> >>> need to export clock driver APIs to CPUFreq.
> >>>
> >>> Was thinking of below way of implementing this...
> >>>
> >>>
> >>> Clock DFLL driver Suspend:
> >>>
> >>>          - Save CPU clock policy registers, and Perform dfll
> >>> suspend which sets in open loop mode
> >>>
> >>> CPU Freq driver Suspend: does nothing
> >>>
> >>>
> >>> Clock DFLL driver Resume:
> >>>
> >>>          - Re-init DFLL, Set in Open-Loop mode, restore CPU Clock
> >>> policy registers which actually sets source to DFLL along with
> >>> other CPU Policy register restore.
> >>>
> >>> CPU Freq driver Resume:
> >>>
> >>>          - do clk_prepare_enable which acutally sets DFLL in
> >>> Closed loop mode
> >>>
> >>>
> >>> Adding one more note: Switching CPU Clock to PLLP is not needed as
> >>> CPU CLock can be from dfll in open-loop mode as DFLL is not
> >>> disabled anywhere throught the suspend/resume path and SC7 entry
> >>> FW and Warm boot code will switch CPU source to PLLP.  
> > Since CPU resumes on PLLP, it will be cleaner to suspend it on PLLP
> > as well. And besides, seems that currently disabling DFLL clock will
> > disable DFLL completely and then you'd want to re-init the DFLL on
> > resume any ways. So better to just disable DFLL completely on
> > suspend, which should happen on clk_disable(dfll).  
> 
> Will switch to PLLP during CPUFreq suspend. With decision of using 
> clk_disable during suspend, its mandatory to switch to PLLP as DFLL
> is completely disabled.
> 
> My earlier concern was on restoring CPU policy as we can't do that
> from CPUFreq driver and need export from clock driver.
> 
> Clear now and will do CPU clock policy restore in after dfll re-init.

Why the policy can't be saved/restored by the CaR driver as a context of
any other clock?

> Also I don't see Tegra124 CPU Freq driver using flag 
> CPUFREQ_NEED_INITIAL_FREQ_CHECK.
> 
> Tegra124 CPUFreq driver is not using cpufreq_driver
> 
> 
> <https://elixir.bootlin.com/linux/v5.2.1/ident/cpufreq_driver>
> 

T124 driver is a wrapper around cpufreq-dt driver.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17  5:42                                                           ` Dmitry Osipenko
@ 2019-07-17  5:55                                                             ` Sowjanya Komatineni
  2019-07-17  6:33                                                               ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17  5:55 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree


On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
> В Tue, 16 Jul 2019 22:25:25 -0700
> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>
>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>   
>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>> The other thing that also need attention is that
>>>>>>>>>>>>>>>>>>>>>> T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed
>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>      
>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>> Probably you should use the "device links". See
>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>      
>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>> Looks like I initially confused this case with getting
>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the clock
>>>>>>>>>>>>>>> and then
>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until
>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>> probed, hence everything should be fine as-is and there
>>>>>>>>>>>>>>> is no real
>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>      
>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU
>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to
>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table
>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to other
>>>>>>>>>>>>>>>>>>>> sources with
>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We
>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce
>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to have a
>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no
>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source
>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>      
>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During suspend,
>>>>>>>>>>>>>>>> need to change
>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop mode
>>>>>>>>>>>>>>>> first and
>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>      
>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we need to
>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the time.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it
>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest of
>>>>>>>>>>>>>>>>>>>> the sequence to
>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the
>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave
>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore
>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL
>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a
>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and switches
>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>> source, which means that this driver is also should
>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks around
>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>      
>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the
>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to
>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as
>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU frequency
>>>>>>>>>>>>>>>>>>> is ok for
>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>      
>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g
>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open loop mode.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during the
>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm also wondering if it's always safe to switch to PLLP
>>>>>>>>>>>>>>> in the probe.
>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then some
>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>>>>>>>      
>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs
>>>>>>>>>>>>>> at higher
>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to dfll
>>>>>>>>>>>>>> clock enable
>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a divided
>>>>>>>>>>>>> output of
>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Probably, realistically, CPU is always running off a fast
>>>>>>>>>>>>> PLLX during
>>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I guess
>>>>>>>>>>>>> ideally CPUFreq driver should also have a 'shutdown'
>>>>>>>>>>>>> callback to teardown DFLL
>>>>>>>>>>>>> on a reboot, but likely that there are other clock-related
>>>>>>>>>>>>> problems as
>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>> important at the
>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [snip]
>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source
>>>>>>>>>>>> above I meant
>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>
>>>>>>>>>>>> As per clock policies, PLL_X is always used for high freq
>>>>>>>>>>>> like
>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>> assigned-clocks.
>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to run at
>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>> So even 204MHz CVB entries are having the same voltage as
>>>>>>>>> 408MHz, correct? It's not instantly obvious to me from the
>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined, I see
>>>>>>>>> that there is the min_millivolts
>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>> per-table.
>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will
>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>> Thank you for the clarification. It would be good to have that
>>>>>>> commented
>>>>>>> in the code as well.
>>>>>> OK, Will add...
>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend
>>>>> happens very early even before disabling non-boot CPUs and also
>>>>> need to export clock driver APIs to CPUFreq.
>>>>>
>>>>> Was thinking of below way of implementing this...
>>>>>
>>>>>
>>>>> Clock DFLL driver Suspend:
>>>>>
>>>>>           - Save CPU clock policy registers, and Perform dfll
>>>>> suspend which sets in open loop mode
>>>>>
>>>>> CPU Freq driver Suspend: does nothing
>>>>>
>>>>>
>>>>> Clock DFLL driver Resume:
>>>>>
>>>>>           - Re-init DFLL, Set in Open-Loop mode, restore CPU Clock
>>>>> policy registers which actually sets source to DFLL along with
>>>>> other CPU Policy register restore.
>>>>>
>>>>> CPU Freq driver Resume:
>>>>>
>>>>>           - do clk_prepare_enable which acutally sets DFLL in
>>>>> Closed loop mode
>>>>>
>>>>>
>>>>> Adding one more note: Switching CPU Clock to PLLP is not needed as
>>>>> CPU CLock can be from dfll in open-loop mode as DFLL is not
>>>>> disabled anywhere throught the suspend/resume path and SC7 entry
>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>> Since CPU resumes on PLLP, it will be cleaner to suspend it on PLLP
>>> as well. And besides, seems that currently disabling DFLL clock will
>>> disable DFLL completely and then you'd want to re-init the DFLL on
>>> resume any ways. So better to just disable DFLL completely on
>>> suspend, which should happen on clk_disable(dfll).
>> Will switch to PLLP during CPUFreq suspend. With decision of using
>> clk_disable during suspend, its mandatory to switch to PLLP as DFLL
>> is completely disabled.
>>
>> My earlier concern was on restoring CPU policy as we can't do that
>> from CPUFreq driver and need export from clock driver.
>>
>> Clear now and will do CPU clock policy restore in after dfll re-init.
> Why the policy can't be saved/restored by the CaR driver as a context of
> any other clock?

restoring cpu clock policy involves programming source and 
super_cclkg_divider.

cclk_g is registered as clk_super_mux and it doesn't use frac_div ops to 
do save/restore its divider.

Also, during clock context we cant restore cclk_g as cclk_g source will 
be dfll and dfll will not be resumed/re-initialized by the time 
clk_super_mux save/restore happens.

we can't use save/restore context for dfll clk_ops because dfllCPU_out 
parent to CCLK_G is first in the clock tree and dfll_ref and dfll_soc 
peripheral clocks are not restored by the time dfll restore happens. 
Also dfll peripheral clock enables need to be restored before dfll 
restore happens which involves programming dfll controller for 
re-initialization.

So dfll resume/re-init is done in clk-tegra210 at end of all clocks 
restore in V5 series but instead of in clk-tegra210 driver I moved now 
to dfll-fcpu driver pm_ops as all dfll dependencies will be restored 
thru clk_restore_context by then. This will be in V6.

>> Also I don't see Tegra124 CPU Freq driver using flag
>> CPUFREQ_NEED_INITIAL_FREQ_CHECK.
>>
>> Tegra124 CPUFreq driver is not using cpufreq_driver
>>
>>
>> <https://elixir.bootlin.com/linux/v5.2.1/ident/cpufreq_driver>
>>
> T124 driver is a wrapper around cpufreq-dt driver.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17  5:55                                                             ` Sowjanya Komatineni
@ 2019-07-17  6:33                                                               ` Dmitry Osipenko
  2019-07-17  6:36                                                                 ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-17  6:33 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree

В Tue, 16 Jul 2019 22:55:52 -0700
Sowjanya Komatineni <skomatineni@nvidia.com> пишет:

> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
> > В Tue, 16 Jul 2019 22:25:25 -0700
> > Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
> >  
> >> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:  
> >>> В Tue, 16 Jul 2019 19:35:49 -0700
> >>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
> >>>     
> >>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:  
> >>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:  
> >>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:  
> >>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:  
> >>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:  
> >>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:  
> >>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:  
> >>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:  
> >>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:  
> >>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:  
> >>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:  
> >>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:  
> >>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:  
> >>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:  
> >>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:  
> >>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph
> >>>>>>>>>>>>>>>>>>> Lo wrote:  
> >>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...  
> >>>>>>>>>>>>>>>>>>>>>> The other thing that also need attention is
> >>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
> >>>>>>>>>>>>>>>>>>>>>> driver
> >>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed
> >>>>>>>>>>>>>>>>>>>>>> first, which is
> >>>>>>>>>>>>>>>>>>>>>> icky.
> >>>>>>>>>>>>>>>>>>>>>>        
> >>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
> >>>>>>>>>>>>>>>>>>>>> register explicitly in
> >>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
> >>>>>>>>>>>>>>>>>>>>> registers?  
> >>>>>>>>>>>>>>>>>> Probably you should use the "device links". See
> >>>>>>>>>>>>>>>>>> [1][2] for the
> >>>>>>>>>>>>>>>>>> example.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> [2]
> >>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
> >>>>>>>>>>>>>>>>>> device_link_add() fails.
> >>>>>>>>>>>>>>>>>> And
> >>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
> >>>>>>>>>>>>>>>>>> device, see [3].
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> [3]
> >>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>        
> >>>>>>>>>>>>>>>>> Will go thru and add...  
> >>>>>>>>>>>>>>> Looks like I initially confused this case with getting
> >>>>>>>>>>>>>>> orphaned clock.
> >>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the
> >>>>>>>>>>>>>>> clock and then
> >>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until
> >>>>>>>>>>>>>>> DFLL driver is
> >>>>>>>>>>>>>>> probed, hence everything should be fine as-is and
> >>>>>>>>>>>>>>> there is no real
> >>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
> >>>>>>>>>>>>>>>        
> >>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
> >>>>>>>>>>>>>>>>>>>> regarding the DFLL
> >>>>>>>>>>>>>>>>>>>> part.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU
> >>>>>>>>>>>>>>>>>>>> clock sources and
> >>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
> >>>>>>>>>>>>>>>>>>>> regulator. We will not
> >>>>>>>>>>>>>>>>>>>> switch
> >>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to
> >>>>>>>>>>>>>>>>>>>> DFLL. Because the
> >>>>>>>>>>>>>>>>>>>> CPU has
> >>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table
> >>>>>>>>>>>>>>>>>>>> (CVB or OPP
> >>>>>>>>>>>>>>>>>>>> table
> >>>>>>>>>>>>>>>>>>>> you see
> >>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to
> >>>>>>>>>>>>>>>>>>>> other sources with
> >>>>>>>>>>>>>>>>>>>> unknew
> >>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We
> >>>>>>>>>>>>>>>>>>>> allow switching to
> >>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.  
> >>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce
> >>>>>>>>>>>>>>>>>> DFLL freq to
> >>>>>>>>>>>>>>>>>> PLLP's
> >>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to have a
> >>>>>>>>>>>>>>>>>> proper CPU voltage.  
> >>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no
> >>>>>>>>>>>>>>>>> need to enforce
> >>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source
> >>>>>>>>>>>>>>>>> to PLLP during
> >>>>>>>>>>>>>>>>> suspend
> >>>>>>>>>>>>>>>>>        
> >>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
> >>>>>>>>>>>>>>>> suspend, need to change
> >>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop
> >>>>>>>>>>>>>>>> mode first and
> >>>>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>> dfll need to be set to open loop.  
> >>>>>>>>>>>>>>> Okay.
> >>>>>>>>>>>>>>>        
> >>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we need to
> >>>>>>>>>>>>>>>>>>>> switch to PLLP in
> >>>>>>>>>>>>>>>>>>>> CPU
> >>>>>>>>>>>>>>>>>>>> idle
> >>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the
> >>>>>>>>>>>>>>>>>>>> time.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it
> >>>>>>>>>>>>>>>>>>>> the open-loop
> >>>>>>>>>>>>>>>>>>>> mode. That's
> >>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest
> >>>>>>>>>>>>>>>>>>>> of the sequence to
> >>>>>>>>>>>>>>>>>>>> turn off
> >>>>>>>>>>>>>>>>>>>> the CPU power.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the
> >>>>>>>>>>>>>>>>>>>> sequence to
> >>>>>>>>>>>>>>>>>>>> turn on
> >>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave
> >>>>>>>>>>>>>>>>>>>> it on PLL_P.
> >>>>>>>>>>>>>>>>>>>> After
> >>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore
> >>>>>>>>>>>>>>>>>>>> the CPU clock
> >>>>>>>>>>>>>>>>>>>> policy (CPU
> >>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
> >>>>>>>>>>>>>>>>>>>> close-loop mode.  
> >>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL
> >>>>>>>>>>>>>>>>>> parent during of
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
> >>>>>>>>>>>>>>>>>> instead of having
> >>>>>>>>>>>>>>>>>> odd
> >>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a
> >>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
> >>>>>>>>>>>>>>>>>> drivers. In this case
> >>>>>>>>>>>>>>>>>> CPUFreq
> >>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and switches
> >>>>>>>>>>>>>>>>>> CPU to that
> >>>>>>>>>>>>>>>>>> clock
> >>>>>>>>>>>>>>>>>> source, which means that this driver is also should
> >>>>>>>>>>>>>>>>>> be responsible for
> >>>>>>>>>>>>>>>>>> management of the DFLL's state during of
> >>>>>>>>>>>>>>>>>> suspend/resume process. If
> >>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
> >>>>>>>>>>>>>>>>>> re-enables it
> >>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks around
> >>>>>>>>>>>>>>>>>> DFLL are not
> >>>>>>>>>>>>>>>>>> needed.
> >>>>>>>>>>>>>>>>>>        
> >>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the
> >>>>>>>>>>>>>>>>>>>> patch subject to
> >>>>>>>>>>>>>>>>>>>> "Add
> >>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to
> >>>>>>>>>>>>>>>>>>>> me.  
> >>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as
> >>>>>>>>>>>>>>>>>>> follows (assuming
> >>>>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Switch to DFLL:
> >>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
> >>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
> >>>>>>>>>>>>>>>>>>> 2) Enable DFLL
> >>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
> >>>>>>>>>>>>>>>>>>> For OVR regulator:
> >>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
> >>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
> >>>>>>>>>>>>>>>>>>> For I2C regulator:
> >>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
> >>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Switch away from DFLL:
> >>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
> >>>>>>>>>>>>>>>>>>> frequency is ok for
> >>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>> vdd_cpu voltage
> >>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
> >>>>>>>>>>>>>>>>>>>        
> >>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g
> >>>>>>>>>>>>>>>> parent is not
> >>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open loop
> >>>>>>>>>>>>>>>> mode.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Will add this ...  
> >>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during the
> >>>>>>>>>>>>>>> probe, similar
> >>>>>>>>>>>>>>> should be done on suspend.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I'm also wondering if it's always safe to switch to
> >>>>>>>>>>>>>>> PLLP in the probe.
> >>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then some
> >>>>>>>>>>>>>>> other more
> >>>>>>>>>>>>>>> appropriate intermediate parent should be selected.
> >>>>>>>>>>>>>>>        
> >>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always
> >>>>>>>>>>>>>> runs at higher
> >>>>>>>>>>>>>> rate
> >>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to
> >>>>>>>>>>>>>> dfll clock enable
> >>>>>>>>>>>>>> should be safe.  
> >>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
> >>>>>>>>>>>>> divided output of
> >>>>>>>>>>>>> PLLP
> >>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Probably, realistically, CPU is always running off a
> >>>>>>>>>>>>> fast PLLX during
> >>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I
> >>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
> >>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
> >>>>>>>>>>>>> on a reboot, but likely that there are other
> >>>>>>>>>>>>> clock-related problems as
> >>>>>>>>>>>>> well that may break KEXEC and thus it is not very
> >>>>>>>>>>>>> important at the
> >>>>>>>>>>>>> moment.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [snip]  
> >>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source
> >>>>>>>>>>>> above I meant
> >>>>>>>>>>>> PLL_P_OUT4.
> >>>>>>>>>>>>
> >>>>>>>>>>>> As per clock policies, PLL_X is always used for high freq
> >>>>>>>>>>>> like  
> >>>>>>>>>>>>> 800Mhz  
> >>>>>>>>>>>> and for low frequency it will be sourced from PLLP.  
> >>>>>>>>>>> Alright, then please don't forget to pre-initialize
> >>>>>>>>>>> PLLP_OUT4 rate to a
> >>>>>>>>>>> reasonable value using tegra_clk_init_table or
> >>>>>>>>>>> assigned-clocks.  
> >>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to run at
> >>>>>>>>>> 408Mhz because it is below fmax @ Vmin  
> >>>>>>>>> So even 204MHz CVB entries are having the same voltage as
> >>>>>>>>> 408MHz, correct? It's not instantly obvious to me from the
> >>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined, I see
> >>>>>>>>> that there is the min_millivolts
> >>>>>>>>> and frequency entries starting from 204MHZ defined
> >>>>>>>>> per-table.  
> >>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will
> >>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.  
> >>>>>>> Thank you for the clarification. It would be good to have that
> >>>>>>> commented
> >>>>>>> in the code as well.  
> >>>>>> OK, Will add...  
> >>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend
> >>>>> happens very early even before disabling non-boot CPUs and also
> >>>>> need to export clock driver APIs to CPUFreq.
> >>>>>
> >>>>> Was thinking of below way of implementing this...
> >>>>>
> >>>>>
> >>>>> Clock DFLL driver Suspend:
> >>>>>
> >>>>>           - Save CPU clock policy registers, and Perform dfll
> >>>>> suspend which sets in open loop mode
> >>>>>
> >>>>> CPU Freq driver Suspend: does nothing
> >>>>>
> >>>>>
> >>>>> Clock DFLL driver Resume:
> >>>>>
> >>>>>           - Re-init DFLL, Set in Open-Loop mode, restore CPU
> >>>>> Clock policy registers which actually sets source to DFLL along
> >>>>> with other CPU Policy register restore.
> >>>>>
> >>>>> CPU Freq driver Resume:
> >>>>>
> >>>>>           - do clk_prepare_enable which acutally sets DFLL in
> >>>>> Closed loop mode
> >>>>>
> >>>>>
> >>>>> Adding one more note: Switching CPU Clock to PLLP is not needed
> >>>>> as CPU CLock can be from dfll in open-loop mode as DFLL is not
> >>>>> disabled anywhere throught the suspend/resume path and SC7 entry
> >>>>> FW and Warm boot code will switch CPU source to PLLP.  
> >>> Since CPU resumes on PLLP, it will be cleaner to suspend it on
> >>> PLLP as well. And besides, seems that currently disabling DFLL
> >>> clock will disable DFLL completely and then you'd want to re-init
> >>> the DFLL on resume any ways. So better to just disable DFLL
> >>> completely on suspend, which should happen on clk_disable(dfll).  
> >> Will switch to PLLP during CPUFreq suspend. With decision of using
> >> clk_disable during suspend, its mandatory to switch to PLLP as DFLL
> >> is completely disabled.
> >>
> >> My earlier concern was on restoring CPU policy as we can't do that
> >> from CPUFreq driver and need export from clock driver.
> >>
> >> Clear now and will do CPU clock policy restore in after dfll
> >> re-init.  
> > Why the policy can't be saved/restored by the CaR driver as a
> > context of any other clock?  
> 
> restoring cpu clock policy involves programming source and 
> super_cclkg_divider.
> 
> cclk_g is registered as clk_super_mux and it doesn't use frac_div ops
> to do save/restore its divider.

That can be changed of course and I guess it also could be as simple as
saving and restoring of two raw u32 values of the policy/divider
registers.

> Also, during clock context we cant restore cclk_g as cclk_g source
> will be dfll and dfll will not be resumed/re-initialized by the time 
> clk_super_mux save/restore happens.
> 
> we can't use save/restore context for dfll clk_ops because
> dfllCPU_out parent to CCLK_G is first in the clock tree and dfll_ref
> and dfll_soc peripheral clocks are not restored by the time dfll
> restore happens. Also dfll peripheral clock enables need to be
> restored before dfll restore happens which involves programming dfll
> controller for re-initialization.
> 
> So dfll resume/re-init is done in clk-tegra210 at end of all clocks 
> restore in V5 series but instead of in clk-tegra210 driver I moved
> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
> restored thru clk_restore_context by then. This will be in V6.

Since DFLL is now guaranteed to be disabled across CaR suspend/resume
(hence it has nothing to do in regards to CCLK) and given that PLLs
state is restored before the rest of the clocks, I don't see why not to
implement CCLK save/restore in a generic fasion. CPU policy wull be
restored to either PLLP or PLLX (if CPUFreq driver is disabled).


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17  6:33                                                               ` Dmitry Osipenko
@ 2019-07-17  6:36                                                                 ` Sowjanya Komatineni
  2019-07-17 15:17                                                                   ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17  6:36 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree


On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
> В Tue, 16 Jul 2019 22:55:52 -0700
> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>
>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>   
>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>      
>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph
>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need attention is
>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed
>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links". See
>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>> Looks like I initially confused this case with getting
>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the
>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until
>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>> probed, hence everything should be fine as-is and
>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU
>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to
>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table
>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to
>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We
>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce
>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to have a
>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no
>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source
>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop
>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we need to
>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the
>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it
>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest
>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the
>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave
>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore
>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL
>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a
>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and switches
>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>> source, which means that this driver is also should
>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks around
>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the
>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to
>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as
>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g
>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open loop
>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during the
>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to switch to
>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then some
>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always
>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to
>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Probably, realistically, CPU is always running off a
>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I
>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source
>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As per clock policies, PLL_X is always used for high freq
>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to run at
>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>> So even 204MHz CVB entries are having the same voltage as
>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me from the
>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined, I see
>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>> per-table.
>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will
>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>> Thank you for the clarification. It would be good to have that
>>>>>>>>> commented
>>>>>>>>> in the code as well.
>>>>>>>> OK, Will add...
>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend
>>>>>>> happens very early even before disabling non-boot CPUs and also
>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>
>>>>>>> Was thinking of below way of implementing this...
>>>>>>>
>>>>>>>
>>>>>>> Clock DFLL driver Suspend:
>>>>>>>
>>>>>>>            - Save CPU clock policy registers, and Perform dfll
>>>>>>> suspend which sets in open loop mode
>>>>>>>
>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>
>>>>>>>
>>>>>>> Clock DFLL driver Resume:
>>>>>>>
>>>>>>>            - Re-init DFLL, Set in Open-Loop mode, restore CPU
>>>>>>> Clock policy registers which actually sets source to DFLL along
>>>>>>> with other CPU Policy register restore.
>>>>>>>
>>>>>>> CPU Freq driver Resume:
>>>>>>>
>>>>>>>            - do clk_prepare_enable which acutally sets DFLL in
>>>>>>> Closed loop mode
>>>>>>>
>>>>>>>
>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not needed
>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL is not
>>>>>>> disabled anywhere throught the suspend/resume path and SC7 entry
>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend it on
>>>>> PLLP as well. And besides, seems that currently disabling DFLL
>>>>> clock will disable DFLL completely and then you'd want to re-init
>>>>> the DFLL on resume any ways. So better to just disable DFLL
>>>>> completely on suspend, which should happen on clk_disable(dfll).
>>>> Will switch to PLLP during CPUFreq suspend. With decision of using
>>>> clk_disable during suspend, its mandatory to switch to PLLP as DFLL
>>>> is completely disabled.
>>>>
>>>> My earlier concern was on restoring CPU policy as we can't do that
>>>> from CPUFreq driver and need export from clock driver.
>>>>
>>>> Clear now and will do CPU clock policy restore in after dfll
>>>> re-init.
>>> Why the policy can't be saved/restored by the CaR driver as a
>>> context of any other clock?
>> restoring cpu clock policy involves programming source and
>> super_cclkg_divider.
>>
>> cclk_g is registered as clk_super_mux and it doesn't use frac_div ops
>> to do save/restore its divider.
> That can be changed of course and I guess it also could be as simple as
> saving and restoring of two raw u32 values of the policy/divider
> registers.
>
>> Also, during clock context we cant restore cclk_g as cclk_g source
>> will be dfll and dfll will not be resumed/re-initialized by the time
>> clk_super_mux save/restore happens.
>>
>> we can't use save/restore context for dfll clk_ops because
>> dfllCPU_out parent to CCLK_G is first in the clock tree and dfll_ref
>> and dfll_soc peripheral clocks are not restored by the time dfll
>> restore happens. Also dfll peripheral clock enables need to be
>> restored before dfll restore happens which involves programming dfll
>> controller for re-initialization.
>>
>> So dfll resume/re-init is done in clk-tegra210 at end of all clocks
>> restore in V5 series but instead of in clk-tegra210 driver I moved
>> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
>> restored thru clk_restore_context by then. This will be in V6.
> Since DFLL is now guaranteed to be disabled across CaR suspend/resume
> (hence it has nothing to do in regards to CCLK) and given that PLLs
> state is restored before the rest of the clocks, I don't see why not to
> implement CCLK save/restore in a generic fasion. CPU policy wull be
> restored to either PLLP or PLLX (if CPUFreq driver is disabled).
>
CCLK_G save/restore should happen in clk_super_mux ops save/context and 
clk_super_mux save/restore happens very early as cclk_g is first in the 
clock tree and save/restore traverses through the tree top-bottom order.

DFLL enable thru CPUFreq resume happens after all clk_restore_context 
happens. So during clk_restore_context, dfll re-init doesnt happen and 
doing cpu clock policy restore during super_mux clk_ops will crash as 
DFLL is not initialized and its clock is not enabled but CPU clock 
restore sets source to DFLL if we restore during super_clk_mux


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17  6:36                                                                 ` Sowjanya Komatineni
@ 2019-07-17 15:17                                                                   ` Dmitry Osipenko
  2019-07-17 17:29                                                                     ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-17 15:17 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree

17.07.2019 9:36, Sowjanya Komatineni пишет:
> 
> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>> В Tue, 16 Jul 2019 22:55:52 -0700
>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>
>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>  
>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>     
>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph
>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need attention is
>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed
>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links". See
>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with getting
>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the
>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until
>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>> probed, hence everything should be fine as-is and
>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>>>>        
>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU
>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to
>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table
>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to
>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We
>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce
>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to have a
>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no
>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source
>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop
>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>        
>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we need to
>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the
>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it
>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest
>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the
>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave
>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore
>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL
>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a
>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and switches
>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is also should
>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks around
>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>        
>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the
>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to
>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as
>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g
>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open loop
>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during the
>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to switch to
>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then some
>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always
>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to
>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running off a
>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I
>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source
>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used for high freq
>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to run at
>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>> So even 204MHz CVB entries are having the same voltage as
>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me from the
>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined, I see
>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>> per-table.
>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will
>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>> Thank you for the clarification. It would be good to have that
>>>>>>>>>> commented
>>>>>>>>>> in the code as well.
>>>>>>>>> OK, Will add...
>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend
>>>>>>>> happens very early even before disabling non-boot CPUs and also
>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>
>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>
>>>>>>>>
>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>
>>>>>>>>            - Save CPU clock policy registers, and Perform dfll
>>>>>>>> suspend which sets in open loop mode
>>>>>>>>
>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>
>>>>>>>>
>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>
>>>>>>>>            - Re-init DFLL, Set in Open-Loop mode, restore CPU
>>>>>>>> Clock policy registers which actually sets source to DFLL along
>>>>>>>> with other CPU Policy register restore.
>>>>>>>>
>>>>>>>> CPU Freq driver Resume:
>>>>>>>>
>>>>>>>>            - do clk_prepare_enable which acutally sets DFLL in
>>>>>>>> Closed loop mode
>>>>>>>>
>>>>>>>>
>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not needed
>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL is not
>>>>>>>> disabled anywhere throught the suspend/resume path and SC7 entry
>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend it on
>>>>>> PLLP as well. And besides, seems that currently disabling DFLL
>>>>>> clock will disable DFLL completely and then you'd want to re-init
>>>>>> the DFLL on resume any ways. So better to just disable DFLL
>>>>>> completely on suspend, which should happen on clk_disable(dfll).
>>>>> Will switch to PLLP during CPUFreq suspend. With decision of using
>>>>> clk_disable during suspend, its mandatory to switch to PLLP as DFLL
>>>>> is completely disabled.
>>>>>
>>>>> My earlier concern was on restoring CPU policy as we can't do that
>>>>> from CPUFreq driver and need export from clock driver.
>>>>>
>>>>> Clear now and will do CPU clock policy restore in after dfll
>>>>> re-init.
>>>> Why the policy can't be saved/restored by the CaR driver as a
>>>> context of any other clock?
>>> restoring cpu clock policy involves programming source and
>>> super_cclkg_divider.
>>>
>>> cclk_g is registered as clk_super_mux and it doesn't use frac_div ops
>>> to do save/restore its divider.
>> That can be changed of course and I guess it also could be as simple as
>> saving and restoring of two raw u32 values of the policy/divider
>> registers.
>>
>>> Also, during clock context we cant restore cclk_g as cclk_g source
>>> will be dfll and dfll will not be resumed/re-initialized by the time
>>> clk_super_mux save/restore happens.
>>>
>>> we can't use save/restore context for dfll clk_ops because
>>> dfllCPU_out parent to CCLK_G is first in the clock tree and dfll_ref
>>> and dfll_soc peripheral clocks are not restored by the time dfll
>>> restore happens. Also dfll peripheral clock enables need to be
>>> restored before dfll restore happens which involves programming dfll
>>> controller for re-initialization.
>>>
>>> So dfll resume/re-init is done in clk-tegra210 at end of all clocks
>>> restore in V5 series but instead of in clk-tegra210 driver I moved
>>> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
>>> restored thru clk_restore_context by then. This will be in V6.
>> Since DFLL is now guaranteed to be disabled across CaR suspend/resume
>> (hence it has nothing to do in regards to CCLK) and given that PLLs
>> state is restored before the rest of the clocks, I don't see why not to
>> implement CCLK save/restore in a generic fasion. CPU policy wull be
>> restored to either PLLP or PLLX (if CPUFreq driver is disabled).
>>
> CCLK_G save/restore should happen in clk_super_mux ops save/context and
> clk_super_mux save/restore happens very early as cclk_g is first in the
> clock tree and save/restore traverses through the tree top-bottom order.

If CCLK_G is restored before the PLLs, then just change the clocks order
such that it won't happen.

> DFLL enable thru CPUFreq resume happens after all clk_restore_context
> happens. So during clk_restore_context, dfll re-init doesnt happen and
> doing cpu clock policy restore during super_mux clk_ops will crash as
> DFLL is not initialized and its clock is not enabled but CPU clock
> restore sets source to DFLL if we restore during super_clk_mux

If CPU was suspended on PLLP, then it will be restored on PLLP by CaR. I
don't understand what DFLL has to do with the CCLK in that case during
the clocks restore.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 15:17                                                                   ` Dmitry Osipenko
@ 2019-07-17 17:29                                                                     ` Sowjanya Komatineni
  2019-07-17 18:32                                                                       ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17 17:29 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree


On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>
>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>   
>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>      
>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph
>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need attention is
>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed
>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>          
>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links". See
>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>          
>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with getting
>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the
>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until
>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine as-is and
>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU
>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to
>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table
>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to
>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We
>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce
>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to have a
>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no
>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source
>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>          
>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop
>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we need to
>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the
>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it
>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest
>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the
>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave
>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore
>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL
>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a
>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and switches
>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is also should
>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks around
>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the
>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to
>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as
>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>          
>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g
>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open loop
>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during the
>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to switch to
>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then some
>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>>>>>>>>>>>          
>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always
>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to
>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running off a
>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I
>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source
>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used for high freq
>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to run at
>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>> So even 204MHz CVB entries are having the same voltage as
>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me from the
>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined, I see
>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>> per-table.
>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will
>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>> Thank you for the clarification. It would be good to have that
>>>>>>>>>>> commented
>>>>>>>>>>> in the code as well.
>>>>>>>>>> OK, Will add...
>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend
>>>>>>>>> happens very early even before disabling non-boot CPUs and also
>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>
>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>
>>>>>>>>>             - Save CPU clock policy registers, and Perform dfll
>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>
>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>
>>>>>>>>>             - Re-init DFLL, Set in Open-Loop mode, restore CPU
>>>>>>>>> Clock policy registers which actually sets source to DFLL along
>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>
>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>
>>>>>>>>>             - do clk_prepare_enable which acutally sets DFLL in
>>>>>>>>> Closed loop mode
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not needed
>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL is not
>>>>>>>>> disabled anywhere throught the suspend/resume path and SC7 entry
>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend it on
>>>>>>> PLLP as well. And besides, seems that currently disabling DFLL
>>>>>>> clock will disable DFLL completely and then you'd want to re-init
>>>>>>> the DFLL on resume any ways. So better to just disable DFLL
>>>>>>> completely on suspend, which should happen on clk_disable(dfll).
>>>>>> Will switch to PLLP during CPUFreq suspend. With decision of using
>>>>>> clk_disable during suspend, its mandatory to switch to PLLP as DFLL
>>>>>> is completely disabled.
>>>>>>
>>>>>> My earlier concern was on restoring CPU policy as we can't do that
>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>
>>>>>> Clear now and will do CPU clock policy restore in after dfll
>>>>>> re-init.
>>>>> Why the policy can't be saved/restored by the CaR driver as a
>>>>> context of any other clock?
>>>> restoring cpu clock policy involves programming source and
>>>> super_cclkg_divider.
>>>>
>>>> cclk_g is registered as clk_super_mux and it doesn't use frac_div ops
>>>> to do save/restore its divider.
>>> That can be changed of course and I guess it also could be as simple as
>>> saving and restoring of two raw u32 values of the policy/divider
>>> registers.
>>>
>>>> Also, during clock context we cant restore cclk_g as cclk_g source
>>>> will be dfll and dfll will not be resumed/re-initialized by the time
>>>> clk_super_mux save/restore happens.
>>>>
>>>> we can't use save/restore context for dfll clk_ops because
>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and dfll_ref
>>>> and dfll_soc peripheral clocks are not restored by the time dfll
>>>> restore happens. Also dfll peripheral clock enables need to be
>>>> restored before dfll restore happens which involves programming dfll
>>>> controller for re-initialization.
>>>>
>>>> So dfll resume/re-init is done in clk-tegra210 at end of all clocks
>>>> restore in V5 series but instead of in clk-tegra210 driver I moved
>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
>>>> restored thru clk_restore_context by then. This will be in V6.
>>> Since DFLL is now guaranteed to be disabled across CaR suspend/resume
>>> (hence it has nothing to do in regards to CCLK) and given that PLLs
>>> state is restored before the rest of the clocks, I don't see why not to
>>> implement CCLK save/restore in a generic fasion. CPU policy wull be
>>> restored to either PLLP or PLLX (if CPUFreq driver is disabled).
>>>
>> CCLK_G save/restore should happen in clk_super_mux ops save/context and
>> clk_super_mux save/restore happens very early as cclk_g is first in the
>> clock tree and save/restore traverses through the tree top-bottom order.
> If CCLK_G is restored before the PLLs, then just change the clocks order
> such that it won't happen.
>
I dont think we can change clocks order for CCLK_G.

During bootup, cclk_g is registered after all pll's and peripheral 
clocks which is the way we wanted, So cclk_g will be the first one in 
the clk list as clk_register adds new clock first in the list.

When clk_save_context and clk_restore_context APIs iterates over the 
list, cclk_g is the first

>> DFLL enable thru CPUFreq resume happens after all clk_restore_context
>> happens. So during clk_restore_context, dfll re-init doesnt happen and
>> doing cpu clock policy restore during super_mux clk_ops will crash as
>> DFLL is not initialized and its clock is not enabled but CPU clock
>> restore sets source to DFLL if we restore during super_clk_mux
> If CPU was suspended on PLLP, then it will be restored on PLLP by CaR. I
> don't understand what DFLL has to do with the CCLK in that case during
> the clocks restore.

My above comment is in reference to your request of doing save/restore 
for cclk_g in normal fashion thru save/restore context. Because of the 
clk order I mentioned above, we cclk_g will be the first one to go thru 
save/context.

During save_context of cclk_g, source can be from PLLX, dfll.

Issue will be when we do restore during clk_restore_context of cclk_g as 
by that time PLLX/dfll will not be restored.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 17:29                                                                     ` Sowjanya Komatineni
@ 2019-07-17 18:32                                                                       ` Dmitry Osipenko
  2019-07-17 18:51                                                                         ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-17 18:32 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree

17.07.2019 20:29, Sowjanya Komatineni пишет:
> 
> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>
>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>  
>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>     
>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph
>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed
>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>          
>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links". See
>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>          
>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with getting
>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the
>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until
>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine as-is and
>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>>>>>>        
>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU
>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to
>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table
>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to
>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We
>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce
>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to have a
>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no
>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source
>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>          
>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop
>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>        
>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we need to
>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the
>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it
>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest
>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the
>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave
>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore
>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL
>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a
>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and switches
>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is also should
>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks around
>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>        
>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the
>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as
>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>          
>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g
>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open loop
>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during the
>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to switch to
>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then some
>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>>>>>>>>>>>>          
>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always
>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to
>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running off a
>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I
>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source
>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used for high freq
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to run at
>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same voltage as
>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me from the
>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined, I see
>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will
>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>> Thank you for the clarification. It would be good to have that
>>>>>>>>>>>> commented
>>>>>>>>>>>> in the code as well.
>>>>>>>>>>> OK, Will add...
>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend
>>>>>>>>>> happens very early even before disabling non-boot CPUs and also
>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>
>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>
>>>>>>>>>>             - Save CPU clock policy registers, and Perform dfll
>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>
>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>
>>>>>>>>>>             - Re-init DFLL, Set in Open-Loop mode, restore CPU
>>>>>>>>>> Clock policy registers which actually sets source to DFLL along
>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>
>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>
>>>>>>>>>>             - do clk_prepare_enable which acutally sets DFLL in
>>>>>>>>>> Closed loop mode
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not needed
>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL is not
>>>>>>>>>> disabled anywhere throught the suspend/resume path and SC7 entry
>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend it on
>>>>>>>> PLLP as well. And besides, seems that currently disabling DFLL
>>>>>>>> clock will disable DFLL completely and then you'd want to re-init
>>>>>>>> the DFLL on resume any ways. So better to just disable DFLL
>>>>>>>> completely on suspend, which should happen on clk_disable(dfll).
>>>>>>> Will switch to PLLP during CPUFreq suspend. With decision of using
>>>>>>> clk_disable during suspend, its mandatory to switch to PLLP as DFLL
>>>>>>> is completely disabled.
>>>>>>>
>>>>>>> My earlier concern was on restoring CPU policy as we can't do that
>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>
>>>>>>> Clear now and will do CPU clock policy restore in after dfll
>>>>>>> re-init.
>>>>>> Why the policy can't be saved/restored by the CaR driver as a
>>>>>> context of any other clock?
>>>>> restoring cpu clock policy involves programming source and
>>>>> super_cclkg_divider.
>>>>>
>>>>> cclk_g is registered as clk_super_mux and it doesn't use frac_div ops
>>>>> to do save/restore its divider.
>>>> That can be changed of course and I guess it also could be as simple as
>>>> saving and restoring of two raw u32 values of the policy/divider
>>>> registers.
>>>>
>>>>> Also, during clock context we cant restore cclk_g as cclk_g source
>>>>> will be dfll and dfll will not be resumed/re-initialized by the time
>>>>> clk_super_mux save/restore happens.
>>>>>
>>>>> we can't use save/restore context for dfll clk_ops because
>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and dfll_ref
>>>>> and dfll_soc peripheral clocks are not restored by the time dfll
>>>>> restore happens. Also dfll peripheral clock enables need to be
>>>>> restored before dfll restore happens which involves programming dfll
>>>>> controller for re-initialization.
>>>>>
>>>>> So dfll resume/re-init is done in clk-tegra210 at end of all clocks
>>>>> restore in V5 series but instead of in clk-tegra210 driver I moved
>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
>>>>> restored thru clk_restore_context by then. This will be in V6.
>>>> Since DFLL is now guaranteed to be disabled across CaR suspend/resume
>>>> (hence it has nothing to do in regards to CCLK) and given that PLLs
>>>> state is restored before the rest of the clocks, I don't see why not to
>>>> implement CCLK save/restore in a generic fasion. CPU policy wull be
>>>> restored to either PLLP or PLLX (if CPUFreq driver is disabled).
>>>>
>>> CCLK_G save/restore should happen in clk_super_mux ops save/context and
>>> clk_super_mux save/restore happens very early as cclk_g is first in the
>>> clock tree and save/restore traverses through the tree top-bottom order.
>> If CCLK_G is restored before the PLLs, then just change the clocks order
>> such that it won't happen.
>>
> I dont think we can change clocks order for CCLK_G.
> 
> During bootup, cclk_g is registered after all pll's and peripheral
> clocks which is the way we wanted, So cclk_g will be the first one in
> the clk list as clk_register adds new clock first in the list.
> 
> When clk_save_context and clk_restore_context APIs iterates over the
> list, cclk_g is the first

Looking at clk_core_restore_context(), I see that it walks up CLKs list
from parent to children, hence I don't understand how it can ever happen
that CCLK will be restored before the parent. The clocks registration
order doesn't matter at all in that case.

>>> DFLL enable thru CPUFreq resume happens after all clk_restore_context
>>> happens. So during clk_restore_context, dfll re-init doesnt happen and
>>> doing cpu clock policy restore during super_mux clk_ops will crash as
>>> DFLL is not initialized and its clock is not enabled but CPU clock
>>> restore sets source to DFLL if we restore during super_clk_mux
>> If CPU was suspended on PLLP, then it will be restored on PLLP by CaR. I
>> don't understand what DFLL has to do with the CCLK in that case during
>> the clocks restore.
> 
> My above comment is in reference to your request of doing save/restore
> for cclk_g in normal fashion thru save/restore context. Because of the
> clk order I mentioned above, we cclk_g will be the first one to go thru
> save/context.
> 
> During save_context of cclk_g, source can be from PLLX, dfll.
> 
> Issue will be when we do restore during clk_restore_context of cclk_g as
> by that time PLLX/dfll will not be restored.
> 

Seems we already agreed that DFLL will be disabled by the CPUFreq driver
on suspend. Hence CCLK can't be from DFLL if CPU is reparented to PLLP
on CPUFreq driver's suspend, otherwise CPU keeps running from a
boot-state PLLX if CPUFreq driver is disabled.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 18:32                                                                       ` Dmitry Osipenko
@ 2019-07-17 18:51                                                                         ` Sowjanya Komatineni
  2019-07-17 18:54                                                                           ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17 18:51 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree


On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>
>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>   
>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>      
>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph
>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>           
>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links". See
>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>           
>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with getting
>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the
>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until
>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine as-is and
>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to
>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS table
>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to
>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to work. We
>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to enforce
>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to have a
>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. So no
>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G source
>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>           
>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop
>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we need to
>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the
>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function moves it
>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest
>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle the
>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And leave
>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, restore
>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to DFLL
>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to have a
>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and switches
>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is also should
>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks around
>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>         
>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the
>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use are as
>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>           
>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend), cclk_g
>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open loop
>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during the
>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to switch to
>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then some
>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>>>>>>>>>>>>>           
>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always
>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to
>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running off a
>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I
>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source
>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used for high freq
>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to run at
>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same voltage as
>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me from the
>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined, I see
>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that will
>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>> Thank you for the clarification. It would be good to have that
>>>>>>>>>>>>> commented
>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend
>>>>>>>>>>> happens very early even before disabling non-boot CPUs and also
>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>
>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>
>>>>>>>>>>>              - Save CPU clock policy registers, and Perform dfll
>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>
>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>
>>>>>>>>>>>              - Re-init DFLL, Set in Open-Loop mode, restore CPU
>>>>>>>>>>> Clock policy registers which actually sets source to DFLL along
>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>
>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>
>>>>>>>>>>>              - do clk_prepare_enable which acutally sets DFLL in
>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not needed
>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL is not
>>>>>>>>>>> disabled anywhere throught the suspend/resume path and SC7 entry
>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend it on
>>>>>>>>> PLLP as well. And besides, seems that currently disabling DFLL
>>>>>>>>> clock will disable DFLL completely and then you'd want to re-init
>>>>>>>>> the DFLL on resume any ways. So better to just disable DFLL
>>>>>>>>> completely on suspend, which should happen on clk_disable(dfll).
>>>>>>>> Will switch to PLLP during CPUFreq suspend. With decision of using
>>>>>>>> clk_disable during suspend, its mandatory to switch to PLLP as DFLL
>>>>>>>> is completely disabled.
>>>>>>>>
>>>>>>>> My earlier concern was on restoring CPU policy as we can't do that
>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>
>>>>>>>> Clear now and will do CPU clock policy restore in after dfll
>>>>>>>> re-init.
>>>>>>> Why the policy can't be saved/restored by the CaR driver as a
>>>>>>> context of any other clock?
>>>>>> restoring cpu clock policy involves programming source and
>>>>>> super_cclkg_divider.
>>>>>>
>>>>>> cclk_g is registered as clk_super_mux and it doesn't use frac_div ops
>>>>>> to do save/restore its divider.
>>>>> That can be changed of course and I guess it also could be as simple as
>>>>> saving and restoring of two raw u32 values of the policy/divider
>>>>> registers.
>>>>>
>>>>>> Also, during clock context we cant restore cclk_g as cclk_g source
>>>>>> will be dfll and dfll will not be resumed/re-initialized by the time
>>>>>> clk_super_mux save/restore happens.
>>>>>>
>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and dfll_ref
>>>>>> and dfll_soc peripheral clocks are not restored by the time dfll
>>>>>> restore happens. Also dfll peripheral clock enables need to be
>>>>>> restored before dfll restore happens which involves programming dfll
>>>>>> controller for re-initialization.
>>>>>>
>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of all clocks
>>>>>> restore in V5 series but instead of in clk-tegra210 driver I moved
>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
>>>>>> restored thru clk_restore_context by then. This will be in V6.
>>>>> Since DFLL is now guaranteed to be disabled across CaR suspend/resume
>>>>> (hence it has nothing to do in regards to CCLK) and given that PLLs
>>>>> state is restored before the rest of the clocks, I don't see why not to
>>>>> implement CCLK save/restore in a generic fasion. CPU policy wull be
>>>>> restored to either PLLP or PLLX (if CPUFreq driver is disabled).
>>>>>
>>>> CCLK_G save/restore should happen in clk_super_mux ops save/context and
>>>> clk_super_mux save/restore happens very early as cclk_g is first in the
>>>> clock tree and save/restore traverses through the tree top-bottom order.
>>> If CCLK_G is restored before the PLLs, then just change the clocks order
>>> such that it won't happen.
>>>
>> I dont think we can change clocks order for CCLK_G.
>>
>> During bootup, cclk_g is registered after all pll's and peripheral
>> clocks which is the way we wanted, So cclk_g will be the first one in
>> the clk list as clk_register adds new clock first in the list.
>>
>> When clk_save_context and clk_restore_context APIs iterates over the
>> list, cclk_g is the first
> Looking at clk_core_restore_context(), I see that it walks up CLKs list
> from parent to children, hence I don't understand how it can ever happen
> that CCLK will be restored before the parent. The clocks registration
> order doesn't matter at all in that case.

yes from parent to children and dfllCPU_out is the top in the list and 
its child is cclk_g.

the way clocks are registered is the order I see in the clock list and 
looking into clk_register API it adds new node first in the list.

cclkg_g & dfll register happens after all plls and peripheral clocks as 
it need ref, soc and peripheral clocks to be enabled.

So they are the last to get registered and so becomes first in the list.

During save/restore context, it traverses thru this list and first in 
the list is dfllcpu_OUT (parent) and its child (cclk_g)

saving should not be an issue at all but we cant restore cclk_g/dfll in 
normal way thru clk_ops restore as plls and peripherals restore doesn't 
happen by that time.

>>>> DFLL enable thru CPUFreq resume happens after all clk_restore_context
>>>> happens. So during clk_restore_context, dfll re-init doesnt happen and
>>>> doing cpu clock policy restore during super_mux clk_ops will crash as
>>>> DFLL is not initialized and its clock is not enabled but CPU clock
>>>> restore sets source to DFLL if we restore during super_clk_mux
>>> If CPU was suspended on PLLP, then it will be restored on PLLP by CaR. I
>>> don't understand what DFLL has to do with the CCLK in that case during
>>> the clocks restore.
>> My above comment is in reference to your request of doing save/restore
>> for cclk_g in normal fashion thru save/restore context. Because of the
>> clk order I mentioned above, we cclk_g will be the first one to go thru
>> save/context.
>>
>> During save_context of cclk_g, source can be from PLLX, dfll.
>>
>> Issue will be when we do restore during clk_restore_context of cclk_g as
>> by that time PLLX/dfll will not be restored.
>>
> Seems we already agreed that DFLL will be disabled by the CPUFreq driver
> on suspend. Hence CCLK can't be from DFLL if CPU is reparented to PLLP
> on CPUFreq driver's suspend, otherwise CPU keeps running from a
> boot-state PLLX if CPUFreq driver is disabled.

Yes suspend should not be an issue but issue will be during resume where 
if we do cclk_g restore in normal way thru clk_restore_context, cclk_g 
restore happens very early as dfllCPU out is the first one that goes 
thru restore context and plls/peripherals are not resumed by then.

CPU runs from PLLX if dfll clock enable fails during boot. So when it 
gets to suspend, we save CPU running clock source as either PLLX or DFLL 
and then we switch to PLLP.


On resume, CPU runs from PLLP by warm boot code and we need to restore 
back its source to the one it was using from saved source context (which 
can be either PLLX or DFLL)

So PLLs & DFLL resume need to happen before CCLKG restore/resume.


With all above discussions, we do DFLL disable in CPUFreq driver on 
suspend and on CPUFreq resume we enable DFLL back and restore CPU clock 
source it was using during suspend (which will be either PLLX if dfll 
enable fails during probe or it will be using DFLL).

So i was trying to say dfll/cclk_g restore can't be done in normal way 
thru clk_ops save/restore context


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 18:51                                                                         ` Sowjanya Komatineni
@ 2019-07-17 18:54                                                                           ` Sowjanya Komatineni
  2019-07-17 19:43                                                                             ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17 18:54 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree


On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>
> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>
>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links". See
>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383 
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html 
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100 
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with 
>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the
>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until
>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine as-is and
>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the 
>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS 
>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to
>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to 
>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to 
>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to have a
>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage. 
>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G 
>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop
>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we need to
>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the
>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function 
>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle 
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And 
>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL, 
>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to 
>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to 
>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and 
>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is also 
>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks 
>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the
>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more 
>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use 
>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend), 
>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open loop
>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during 
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to switch to
>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then 
>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always
>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to
>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running off a
>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I
>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source
>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used for high 
>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to 
>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same voltage as
>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me from the
>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined, I see
>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that 
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>> Thank you for the clarification. It would be good to have 
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend
>>>>>>>>>>>> happens very early even before disabling non-boot CPUs and 
>>>>>>>>>>>> also
>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>
>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>
>>>>>>>>>>>>              - Save CPU clock policy registers, and Perform 
>>>>>>>>>>>> dfll
>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>
>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>
>>>>>>>>>>>>              - Re-init DFLL, Set in Open-Loop mode, restore 
>>>>>>>>>>>> CPU
>>>>>>>>>>>> Clock policy registers which actually sets source to DFLL 
>>>>>>>>>>>> along
>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>
>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>
>>>>>>>>>>>>              - do clk_prepare_enable which acutally sets 
>>>>>>>>>>>> DFLL in
>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not 
>>>>>>>>>>>> needed
>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL is not
>>>>>>>>>>>> disabled anywhere throught the suspend/resume path and SC7 
>>>>>>>>>>>> entry
>>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend it on
>>>>>>>>>> PLLP as well. And besides, seems that currently disabling DFLL
>>>>>>>>>> clock will disable DFLL completely and then you'd want to 
>>>>>>>>>> re-init
>>>>>>>>>> the DFLL on resume any ways. So better to just disable DFLL
>>>>>>>>>> completely on suspend, which should happen on clk_disable(dfll).
>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With decision of 
>>>>>>>>> using
>>>>>>>>> clk_disable during suspend, its mandatory to switch to PLLP as 
>>>>>>>>> DFLL
>>>>>>>>> is completely disabled.
>>>>>>>>>
>>>>>>>>> My earlier concern was on restoring CPU policy as we can't do 
>>>>>>>>> that
>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>
>>>>>>>>> Clear now and will do CPU clock policy restore in after dfll
>>>>>>>>> re-init.
>>>>>>>> Why the policy can't be saved/restored by the CaR driver as a
>>>>>>>> context of any other clock?
>>>>>>> restoring cpu clock policy involves programming source and
>>>>>>> super_cclkg_divider.
>>>>>>>
>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use 
>>>>>>> frac_div ops
>>>>>>> to do save/restore its divider.
>>>>>> That can be changed of course and I guess it also could be as 
>>>>>> simple as
>>>>>> saving and restoring of two raw u32 values of the policy/divider
>>>>>> registers.
>>>>>>
>>>>>>> Also, during clock context we cant restore cclk_g as cclk_g source
>>>>>>> will be dfll and dfll will not be resumed/re-initialized by the 
>>>>>>> time
>>>>>>> clk_super_mux save/restore happens.
>>>>>>>
>>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and 
>>>>>>> dfll_ref
>>>>>>> and dfll_soc peripheral clocks are not restored by the time dfll
>>>>>>> restore happens. Also dfll peripheral clock enables need to be
>>>>>>> restored before dfll restore happens which involves programming 
>>>>>>> dfll
>>>>>>> controller for re-initialization.
>>>>>>>
>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of all clocks
>>>>>>> restore in V5 series but instead of in clk-tegra210 driver I moved
>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
>>>>>>> restored thru clk_restore_context by then. This will be in V6.
>>>>>> Since DFLL is now guaranteed to be disabled across CaR 
>>>>>> suspend/resume
>>>>>> (hence it has nothing to do in regards to CCLK) and given that PLLs
>>>>>> state is restored before the rest of the clocks, I don't see why 
>>>>>> not to
>>>>>> implement CCLK save/restore in a generic fasion. CPU policy wull be
>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is disabled).
>>>>>>
>>>>> CCLK_G save/restore should happen in clk_super_mux ops 
>>>>> save/context and
>>>>> clk_super_mux save/restore happens very early as cclk_g is first 
>>>>> in the
>>>>> clock tree and save/restore traverses through the tree top-bottom 
>>>>> order.
>>>> If CCLK_G is restored before the PLLs, then just change the clocks 
>>>> order
>>>> such that it won't happen.
>>>>
>>> I dont think we can change clocks order for CCLK_G.
>>>
>>> During bootup, cclk_g is registered after all pll's and peripheral
>>> clocks which is the way we wanted, So cclk_g will be the first one in
>>> the clk list as clk_register adds new clock first in the list.
>>>
>>> When clk_save_context and clk_restore_context APIs iterates over the
>>> list, cclk_g is the first
>> Looking at clk_core_restore_context(), I see that it walks up CLKs list
>> from parent to children, hence I don't understand how it can ever happen
>> that CCLK will be restored before the parent. The clocks registration
>> order doesn't matter at all in that case.
>
> yes from parent to children and dfllCPU_out is the top in the list and 
> its child is cclk_g.
>
> the way clocks are registered is the order I see in the clock list and 
> looking into clk_register API it adds new node first in the list.
>
cclkg_g & dfll register happens after all plls and peripheral clocks as 
it need ref, soc and peripheral clocks to be enabled.
>
> So they are the last to get registered and so becomes first in the list.
>
> During save/restore context, it traverses thru this list and first in 
> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>
> saving should not be an issue at all but we cant restore cclk_g/dfll 
> in normal way thru clk_ops restore as plls and peripherals restore 
> doesn't happen by that time.
>

I was referring to clk_restore_context where it iterates thru root list 
and for each core from the root list clk_core_restore does restore of 
parent and children.

dfllCPU_Out gets first in the list and its child is cclk_g

https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105


>>>>> DFLL enable thru CPUFreq resume happens after all clk_restore_context
>>>>> happens. So during clk_restore_context, dfll re-init doesnt happen 
>>>>> and
>>>>> doing cpu clock policy restore during super_mux clk_ops will crash as
>>>>> DFLL is not initialized and its clock is not enabled but CPU clock
>>>>> restore sets source to DFLL if we restore during super_clk_mux
>>>> If CPU was suspended on PLLP, then it will be restored on PLLP by 
>>>> CaR. I
>>>> don't understand what DFLL has to do with the CCLK in that case during
>>>> the clocks restore.
>>> My above comment is in reference to your request of doing save/restore
>>> for cclk_g in normal fashion thru save/restore context. Because of the
>>> clk order I mentioned above, we cclk_g will be the first one to go thru
>>> save/context.
>>>
>>> During save_context of cclk_g, source can be from PLLX, dfll.
>>>
>>> Issue will be when we do restore during clk_restore_context of 
>>> cclk_g as
>>> by that time PLLX/dfll will not be restored.
>>>
>> Seems we already agreed that DFLL will be disabled by the CPUFreq driver
>> on suspend. Hence CCLK can't be from DFLL if CPU is reparented to PLLP
>> on CPUFreq driver's suspend, otherwise CPU keeps running from a
>> boot-state PLLX if CPUFreq driver is disabled.
>
> Yes suspend should not be an issue but issue will be during resume 
> where if we do cclk_g restore in normal way thru clk_restore_context, 
> cclk_g restore happens very early as dfllCPU out is the first one that 
> goes thru restore context and plls/peripherals are not resumed by then.
>
> CPU runs from PLLX if dfll clock enable fails during boot. So when it 
> gets to suspend, we save CPU running clock source as either PLLX or 
> DFLL and then we switch to PLLP.
>
>
> On resume, CPU runs from PLLP by warm boot code and we need to restore 
> back its source to the one it was using from saved source context 
> (which can be either PLLX or DFLL)
>
> So PLLs & DFLL resume need to happen before CCLKG restore/resume.
>
>
> With all above discussions, we do DFLL disable in CPUFreq driver on 
> suspend and on CPUFreq resume we enable DFLL back and restore CPU 
> clock source it was using during suspend (which will be either PLLX if 
> dfll enable fails during probe or it will be using DFLL).
>
> So i was trying to say dfll/cclk_g restore can't be done in normal way 
> thru clk_ops save/restore context
>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 18:54                                                                           ` Sowjanya Komatineni
@ 2019-07-17 19:43                                                                             ` Dmitry Osipenko
  2019-07-17 20:01                                                                               ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-17 19:43 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree

17.07.2019 21:54, Sowjanya Komatineni пишет:
> 
> On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>>
>> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>
>>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links". See
>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with
>>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the
>>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until
>>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine as-is and
>>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to
>>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to have a
>>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G
>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop
>>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function
>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And
>>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to
>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
>>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to
>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
>>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and
>>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is also
>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
>>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks
>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more
>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use
>>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend),
>>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open loop
>>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to switch to
>>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then
>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always
>>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to
>>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
>>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running off a
>>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I
>>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source
>>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used for high
>>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to
>>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same voltage as
>>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me from the
>>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined, I see
>>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>>> Thank you for the clarification. It would be good to have
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend
>>>>>>>>>>>>> happens very early even before disabling non-boot CPUs and
>>>>>>>>>>>>> also
>>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>>
>>>>>>>>>>>>>              - Save CPU clock policy registers, and Perform
>>>>>>>>>>>>> dfll
>>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>>
>>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>>
>>>>>>>>>>>>>              - Re-init DFLL, Set in Open-Loop mode, restore
>>>>>>>>>>>>> CPU
>>>>>>>>>>>>> Clock policy registers which actually sets source to DFLL
>>>>>>>>>>>>> along
>>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>>
>>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>>
>>>>>>>>>>>>>              - do clk_prepare_enable which acutally sets
>>>>>>>>>>>>> DFLL in
>>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not
>>>>>>>>>>>>> needed
>>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL is not
>>>>>>>>>>>>> disabled anywhere throught the suspend/resume path and SC7
>>>>>>>>>>>>> entry
>>>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend it on
>>>>>>>>>>> PLLP as well. And besides, seems that currently disabling DFLL
>>>>>>>>>>> clock will disable DFLL completely and then you'd want to
>>>>>>>>>>> re-init
>>>>>>>>>>> the DFLL on resume any ways. So better to just disable DFLL
>>>>>>>>>>> completely on suspend, which should happen on clk_disable(dfll).
>>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With decision of
>>>>>>>>>> using
>>>>>>>>>> clk_disable during suspend, its mandatory to switch to PLLP as
>>>>>>>>>> DFLL
>>>>>>>>>> is completely disabled.
>>>>>>>>>>
>>>>>>>>>> My earlier concern was on restoring CPU policy as we can't do
>>>>>>>>>> that
>>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>>
>>>>>>>>>> Clear now and will do CPU clock policy restore in after dfll
>>>>>>>>>> re-init.
>>>>>>>>> Why the policy can't be saved/restored by the CaR driver as a
>>>>>>>>> context of any other clock?
>>>>>>>> restoring cpu clock policy involves programming source and
>>>>>>>> super_cclkg_divider.
>>>>>>>>
>>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use
>>>>>>>> frac_div ops
>>>>>>>> to do save/restore its divider.
>>>>>>> That can be changed of course and I guess it also could be as
>>>>>>> simple as
>>>>>>> saving and restoring of two raw u32 values of the policy/divider
>>>>>>> registers.
>>>>>>>
>>>>>>>> Also, during clock context we cant restore cclk_g as cclk_g source
>>>>>>>> will be dfll and dfll will not be resumed/re-initialized by the
>>>>>>>> time
>>>>>>>> clk_super_mux save/restore happens.
>>>>>>>>
>>>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and
>>>>>>>> dfll_ref
>>>>>>>> and dfll_soc peripheral clocks are not restored by the time dfll
>>>>>>>> restore happens. Also dfll peripheral clock enables need to be
>>>>>>>> restored before dfll restore happens which involves programming
>>>>>>>> dfll
>>>>>>>> controller for re-initialization.
>>>>>>>>
>>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of all clocks
>>>>>>>> restore in V5 series but instead of in clk-tegra210 driver I moved
>>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
>>>>>>>> restored thru clk_restore_context by then. This will be in V6.
>>>>>>> Since DFLL is now guaranteed to be disabled across CaR
>>>>>>> suspend/resume
>>>>>>> (hence it has nothing to do in regards to CCLK) and given that PLLs
>>>>>>> state is restored before the rest of the clocks, I don't see why
>>>>>>> not to
>>>>>>> implement CCLK save/restore in a generic fasion. CPU policy wull be
>>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is disabled).
>>>>>>>
>>>>>> CCLK_G save/restore should happen in clk_super_mux ops
>>>>>> save/context and
>>>>>> clk_super_mux save/restore happens very early as cclk_g is first
>>>>>> in the
>>>>>> clock tree and save/restore traverses through the tree top-bottom
>>>>>> order.
>>>>> If CCLK_G is restored before the PLLs, then just change the clocks
>>>>> order
>>>>> such that it won't happen.
>>>>>
>>>> I dont think we can change clocks order for CCLK_G.
>>>>
>>>> During bootup, cclk_g is registered after all pll's and peripheral
>>>> clocks which is the way we wanted, So cclk_g will be the first one in
>>>> the clk list as clk_register adds new clock first in the list.
>>>>
>>>> When clk_save_context and clk_restore_context APIs iterates over the
>>>> list, cclk_g is the first
>>> Looking at clk_core_restore_context(), I see that it walks up CLKs list
>>> from parent to children, hence I don't understand how it can ever happen
>>> that CCLK will be restored before the parent. The clocks registration
>>> order doesn't matter at all in that case.
>>
>> yes from parent to children and dfllCPU_out is the top in the list and
>> its child is cclk_g.
>>
>> the way clocks are registered is the order I see in the clock list and
>> looking into clk_register API it adds new node first in the list.
>>
> cclkg_g & dfll register happens after all plls and peripheral clocks as
> it need ref, soc and peripheral clocks to be enabled.
>>
>> So they are the last to get registered and so becomes first in the list.
>>
>> During save/restore context, it traverses thru this list and first in
>> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>>
>> saving should not be an issue at all but we cant restore cclk_g/dfll
>> in normal way thru clk_ops restore as plls and peripherals restore
>> doesn't happen by that time.
>>
> 
> I was referring to clk_restore_context where it iterates thru root list
> and for each core from the root list clk_core_restore does restore of
> parent and children.
> 
> dfllCPU_Out gets first in the list and its child is cclk_g
> 
> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105

What list you're talking about? clk_summary? It shows current *active*
clocks configuration, if you'll try to disable CPUFreq driver then the
parent of CCLK_G should be PLLX. Similarly when CPU is reparented to
PLLP on driver's suspend, then PLLP is the parent.

>>>>>> DFLL enable thru CPUFreq resume happens after all clk_restore_context
>>>>>> happens. So during clk_restore_context, dfll re-init doesnt happen
>>>>>> and
>>>>>> doing cpu clock policy restore during super_mux clk_ops will crash as
>>>>>> DFLL is not initialized and its clock is not enabled but CPU clock
>>>>>> restore sets source to DFLL if we restore during super_clk_mux
>>>>> If CPU was suspended on PLLP, then it will be restored on PLLP by
>>>>> CaR. I
>>>>> don't understand what DFLL has to do with the CCLK in that case during
>>>>> the clocks restore.
>>>> My above comment is in reference to your request of doing save/restore
>>>> for cclk_g in normal fashion thru save/restore context. Because of the
>>>> clk order I mentioned above, we cclk_g will be the first one to go thru
>>>> save/context.
>>>>
>>>> During save_context of cclk_g, source can be from PLLX, dfll.
>>>>
>>>> Issue will be when we do restore during clk_restore_context of
>>>> cclk_g as
>>>> by that time PLLX/dfll will not be restored.
>>>>
>>> Seems we already agreed that DFLL will be disabled by the CPUFreq driver
>>> on suspend. Hence CCLK can't be from DFLL if CPU is reparented to PLLP
>>> on CPUFreq driver's suspend, otherwise CPU keeps running from a
>>> boot-state PLLX if CPUFreq driver is disabled.
>>
>> Yes suspend should not be an issue but issue will be during resume
>> where if we do cclk_g restore in normal way thru clk_restore_context,
>> cclk_g restore happens very early as dfllCPU out is the first one that
>> goes thru restore context and plls/peripherals are not resumed by then.
>>
>> CPU runs from PLLX if dfll clock enable fails during boot. So when it
>> gets to suspend, we save CPU running clock source as either PLLX or
>> DFLL and then we switch to PLLP.
>>
>>
>> On resume, CPU runs from PLLP by warm boot code and we need to restore
>> back its source to the one it was using from saved source context
>> (which can be either PLLX or DFLL)
>>
>> So PLLs & DFLL resume need to happen before CCLKG restore/resume.
>>
>>
>> With all above discussions, we do DFLL disable in CPUFreq driver on
>> suspend and on CPUFreq resume we enable DFLL back and restore CPU
>> clock source it was using during suspend (which will be either PLLX if
>> dfll enable fails during probe or it will be using DFLL).

During suspend CPU's parent shall be PLLP and not DFLL (note that it is
disabled) after reparenting to PLLP by the CPUFreq driver.

>> So i was trying to say dfll/cclk_g restore can't be done in normal way
>> thru clk_ops save/restore context

Let's see what happens if CPUFreq is active:

1. CPUFreq driver probe happens
	2. CPU is reparented to PLLP
	3. DFLL inited
	4. CPU is reparented to DFLL

5. CPUFreq driver suspend happens
	6. CPU is reparented to PLLP
	7. DFLL is disabled

8. Car suspend happens
	9. DFLL context saved
	10. PLLP/PLLX context saved
	11. CCLK context saved

12. Car resume happens
	13. DFLL context restored
	14. PLLP/PLLX context restored
	15. CCLK context restored

16. CPUFreq driver resume happens
	17. DFLL re-inited
	18. CPU is reparented to DFLL

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 19:43                                                                             ` Dmitry Osipenko
@ 2019-07-17 20:01                                                                               ` Sowjanya Komatineni
  2019-07-17 20:11                                                                                 ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17 20:01 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree


On 7/17/19 12:43 PM, Dmitry Osipenko wrote:
> 17.07.2019 21:54, Sowjanya Komatineni пишет:
>> On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>>> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>>>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>
>>>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links". See
>>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with
>>>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the
>>>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER until
>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine as-is and
>>>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to
>>>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to have a
>>>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G
>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed loop
>>>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to
>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. Hence
>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to
>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
>>>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and
>>>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is also
>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend and
>>>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks
>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use
>>>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to output
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend),
>>>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open loop
>>>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to switch to
>>>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then
>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be selected.
>>>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always
>>>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe prior to
>>>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
>>>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running off a
>>>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I
>>>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P source
>>>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used for high
>>>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to
>>>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same voltage as
>>>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me from the
>>>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined, I see
>>>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>>>> Thank you for the clarification. It would be good to have
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq suspend
>>>>>>>>>>>>>> happens very early even before disabling non-boot CPUs and
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>               - Save CPU clock policy registers, and Perform
>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>               - Re-init DFLL, Set in Open-Loop mode, restore
>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>> Clock policy registers which actually sets source to DFLL
>>>>>>>>>>>>>> along
>>>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>               - do clk_prepare_enable which acutally sets
>>>>>>>>>>>>>> DFLL in
>>>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not
>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL is not
>>>>>>>>>>>>>> disabled anywhere throught the suspend/resume path and SC7
>>>>>>>>>>>>>> entry
>>>>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend it on
>>>>>>>>>>>> PLLP as well. And besides, seems that currently disabling DFLL
>>>>>>>>>>>> clock will disable DFLL completely and then you'd want to
>>>>>>>>>>>> re-init
>>>>>>>>>>>> the DFLL on resume any ways. So better to just disable DFLL
>>>>>>>>>>>> completely on suspend, which should happen on clk_disable(dfll).
>>>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With decision of
>>>>>>>>>>> using
>>>>>>>>>>> clk_disable during suspend, its mandatory to switch to PLLP as
>>>>>>>>>>> DFLL
>>>>>>>>>>> is completely disabled.
>>>>>>>>>>>
>>>>>>>>>>> My earlier concern was on restoring CPU policy as we can't do
>>>>>>>>>>> that
>>>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>>>
>>>>>>>>>>> Clear now and will do CPU clock policy restore in after dfll
>>>>>>>>>>> re-init.
>>>>>>>>>> Why the policy can't be saved/restored by the CaR driver as a
>>>>>>>>>> context of any other clock?
>>>>>>>>> restoring cpu clock policy involves programming source and
>>>>>>>>> super_cclkg_divider.
>>>>>>>>>
>>>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use
>>>>>>>>> frac_div ops
>>>>>>>>> to do save/restore its divider.
>>>>>>>> That can be changed of course and I guess it also could be as
>>>>>>>> simple as
>>>>>>>> saving and restoring of two raw u32 values of the policy/divider
>>>>>>>> registers.
>>>>>>>>
>>>>>>>>> Also, during clock context we cant restore cclk_g as cclk_g source
>>>>>>>>> will be dfll and dfll will not be resumed/re-initialized by the
>>>>>>>>> time
>>>>>>>>> clk_super_mux save/restore happens.
>>>>>>>>>
>>>>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and
>>>>>>>>> dfll_ref
>>>>>>>>> and dfll_soc peripheral clocks are not restored by the time dfll
>>>>>>>>> restore happens. Also dfll peripheral clock enables need to be
>>>>>>>>> restored before dfll restore happens which involves programming
>>>>>>>>> dfll
>>>>>>>>> controller for re-initialization.
>>>>>>>>>
>>>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of all clocks
>>>>>>>>> restore in V5 series but instead of in clk-tegra210 driver I moved
>>>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
>>>>>>>>> restored thru clk_restore_context by then. This will be in V6.
>>>>>>>> Since DFLL is now guaranteed to be disabled across CaR
>>>>>>>> suspend/resume
>>>>>>>> (hence it has nothing to do in regards to CCLK) and given that PLLs
>>>>>>>> state is restored before the rest of the clocks, I don't see why
>>>>>>>> not to
>>>>>>>> implement CCLK save/restore in a generic fasion. CPU policy wull be
>>>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is disabled).
>>>>>>>>
>>>>>>> CCLK_G save/restore should happen in clk_super_mux ops
>>>>>>> save/context and
>>>>>>> clk_super_mux save/restore happens very early as cclk_g is first
>>>>>>> in the
>>>>>>> clock tree and save/restore traverses through the tree top-bottom
>>>>>>> order.
>>>>>> If CCLK_G is restored before the PLLs, then just change the clocks
>>>>>> order
>>>>>> such that it won't happen.
>>>>>>
>>>>> I dont think we can change clocks order for CCLK_G.
>>>>>
>>>>> During bootup, cclk_g is registered after all pll's and peripheral
>>>>> clocks which is the way we wanted, So cclk_g will be the first one in
>>>>> the clk list as clk_register adds new clock first in the list.
>>>>>
>>>>> When clk_save_context and clk_restore_context APIs iterates over the
>>>>> list, cclk_g is the first
>>>> Looking at clk_core_restore_context(), I see that it walks up CLKs list
>>>> from parent to children, hence I don't understand how it can ever happen
>>>> that CCLK will be restored before the parent. The clocks registration
>>>> order doesn't matter at all in that case.
>>> yes from parent to children and dfllCPU_out is the top in the list and
>>> its child is cclk_g.
>>>
>>> the way clocks are registered is the order I see in the clock list and
>>> looking into clk_register API it adds new node first in the list.
>>>
>> cclkg_g & dfll register happens after all plls and peripheral clocks as
>> it need ref, soc and peripheral clocks to be enabled.
>>> So they are the last to get registered and so becomes first in the list.
>>>
>>> During save/restore context, it traverses thru this list and first in
>>> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>>>
>>> saving should not be an issue at all but we cant restore cclk_g/dfll
>>> in normal way thru clk_ops restore as plls and peripherals restore
>>> doesn't happen by that time.
>>>
>> I was referring to clk_restore_context where it iterates thru root list
>> and for each core from the root list clk_core_restore does restore of
>> parent and children.
>>
>> dfllCPU_Out gets first in the list and its child is cclk_g
>>
>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105
> What list you're talking about? clk_summary? It shows current *active*
> clocks configuration, if you'll try to disable CPUFreq driver then the
> parent of CCLK_G should be PLLX. Similarly when CPU is reparented to
> PLLP on driver's suspend, then PLLP is the parent.
>
>>>>>>> DFLL enable thru CPUFreq resume happens after all clk_restore_context
>>>>>>> happens. So during clk_restore_context, dfll re-init doesnt happen
>>>>>>> and
>>>>>>> doing cpu clock policy restore during super_mux clk_ops will crash as
>>>>>>> DFLL is not initialized and its clock is not enabled but CPU clock
>>>>>>> restore sets source to DFLL if we restore during super_clk_mux
>>>>>> If CPU was suspended on PLLP, then it will be restored on PLLP by
>>>>>> CaR. I
>>>>>> don't understand what DFLL has to do with the CCLK in that case during
>>>>>> the clocks restore.
>>>>> My above comment is in reference to your request of doing save/restore
>>>>> for cclk_g in normal fashion thru save/restore context. Because of the
>>>>> clk order I mentioned above, we cclk_g will be the first one to go thru
>>>>> save/context.
>>>>>
>>>>> During save_context of cclk_g, source can be from PLLX, dfll.
>>>>>
>>>>> Issue will be when we do restore during clk_restore_context of
>>>>> cclk_g as
>>>>> by that time PLLX/dfll will not be restored.
>>>>>
>>>> Seems we already agreed that DFLL will be disabled by the CPUFreq driver
>>>> on suspend. Hence CCLK can't be from DFLL if CPU is reparented to PLLP
>>>> on CPUFreq driver's suspend, otherwise CPU keeps running from a
>>>> boot-state PLLX if CPUFreq driver is disabled.
>>> Yes suspend should not be an issue but issue will be during resume
>>> where if we do cclk_g restore in normal way thru clk_restore_context,
>>> cclk_g restore happens very early as dfllCPU out is the first one that
>>> goes thru restore context and plls/peripherals are not resumed by then.
>>>
>>> CPU runs from PLLX if dfll clock enable fails during boot. So when it
>>> gets to suspend, we save CPU running clock source as either PLLX or
>>> DFLL and then we switch to PLLP.
>>>
>>>
>>> On resume, CPU runs from PLLP by warm boot code and we need to restore
>>> back its source to the one it was using from saved source context
>>> (which can be either PLLX or DFLL)
>>>
>>> So PLLs & DFLL resume need to happen before CCLKG restore/resume.
>>>
>>>
>>> With all above discussions, we do DFLL disable in CPUFreq driver on
>>> suspend and on CPUFreq resume we enable DFLL back and restore CPU
>>> clock source it was using during suspend (which will be either PLLX if
>>> dfll enable fails during probe or it will be using DFLL).
> During suspend CPU's parent shall be PLLP and not DFLL (note that it is
> disabled) after reparenting to PLLP by the CPUFreq driver.
>
CPU source context should be saved before switching to safe source of 
PLLP as on resume we need to restore back to source it was using before 
we switch to safe source during suspend entry.

So saved context for CPU Source will be either dfll or PLLX

>>> So i was trying to say dfll/cclk_g restore can't be done in normal way
>>> thru clk_ops save/restore context
> Let's see what happens if CPUFreq is active:
>
> 1. CPUFreq driver probe happens
> 	2. CPU is reparented to PLLP
> 	3. DFLL inited
> 	4. CPU is reparented to DFLL
>
> 5. CPUFreq driver suspend happens
> 	6. CPU is reparented to PLLP
> 	7. DFLL is disabled
>
> 8. Car suspend happens
> 	9. DFLL context saved
> 	10. PLLP/PLLX context saved
> 	11. CCLK context saved
>
> 12. Car resume happens
> 	13. DFLL context restored
> 	14. PLLP/PLLX context restored
> 	15. CCLK context restored
>
> 16. CPUFreq driver resume happens
> 	17. DFLL re-inited
> 	18. CPU is reparented to DFLL


Below is the order of sequence it should be based on the order of clk 
register.

My comments inline in this sequence.

1. CPUFreq driver probe happens
	2. CPU is reparented to PLLP
	3. DFLL inited
	4. CPU is reparented to DFLL


5. CPUFreq driver suspend happens
	6. Save CPU source which could be either dfll or pllx
	7. CPU is reparented to safe known source PLLP
	8. DFLL is disabled

8. Car suspend happens
	9. DFLL context saved (With DFLL disabled in CPUFreq suspend, nothing to be saved here as last freq req will always be saved).
	10. CCLK context saved (CPU clock source will be saved in CPUFreq driver suspend which could be either dfll or pllx)
	11. PLLP/PLLX context saved
	12. Peripheral Clock saved

12. Car resume happens
	13. DFLL context restored : No DFLL context to be restored and we only need to reinitialize DFLL and re-initialize can't be done here as this is the 1st to get restored and PLL/Peripheral clocks are not restored by this time. So we can't use clk_ops restore for DFLL
	14. CCLK context restored
CCLK cant be restored here as context could be either dfll or pllx which is the source orginally it was actually using before we force switch to safe PLLP for suspend entry. So we can't use clk_ops restore for DFLL

15. PLLP/PLLX context restored
	16. Peripheral context restored

16. CPUFreq driver resume happens
	17. DFLL re-inited (Invoking DFLL re-init in CPUFreq resume need exporting DFLL reinit from Clock driver to CPUFreq driver)
	18. CPU is reparented to DFLL or PLLX based on saved context from step 9.

Note: instead of exporting, we can do DFLL re-init from clock-dfll 
driver itself thru dfll-fcpu pm_ops resume. So dfll will be 
re-initialized by the time CPUFreq driver resumes and switches to use 
DFLL source.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 20:01                                                                               ` Sowjanya Komatineni
@ 2019-07-17 20:11                                                                                 ` Sowjanya Komatineni
  2019-07-17 21:29                                                                                   ` Sowjanya Komatineni
  2019-07-17 21:30                                                                                   ` Dmitry Osipenko
  0 siblings, 2 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17 20:11 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree


On 7/17/19 1:01 PM, Sowjanya Komatineni wrote:
>
> On 7/17/19 12:43 PM, Dmitry Osipenko wrote:
>> 17.07.2019 21:54, Sowjanya Komatineni пишет:
>>> On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>>>> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>>>>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>>>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>
>>>>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links". 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> See
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with
>>>>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the
>>>>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER 
>>>>>>>>>>>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine as-is and
>>>>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G
>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed 
>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver. 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hence
>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
>>>>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is also
>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks
>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend),
>>>>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open 
>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to 
>>>>>>>>>>>>>>>>>>>>>>>>> switch to
>>>>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then
>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be 
>>>>>>>>>>>>>>>>>>>>>>>>> selected.
>>>>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X 
>>>>>>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe 
>>>>>>>>>>>>>>>>>>>>>>>> prior to
>>>>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
>>>>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running 
>>>>>>>>>>>>>>>>>>>>>>> off a
>>>>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I
>>>>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P 
>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used for high
>>>>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to
>>>>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same 
>>>>>>>>>>>>>>>>>>> voltage as
>>>>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me 
>>>>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined, 
>>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that
>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>>>>> Thank you for the clarification. It would be good to have
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq 
>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>> happens very early even before disabling non-boot CPUs and
>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>               - Save CPU clock policy registers, and 
>>>>>>>>>>>>>>> Perform
>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>               - Re-init DFLL, Set in Open-Loop mode, 
>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>> Clock policy registers which actually sets source to DFLL
>>>>>>>>>>>>>>> along
>>>>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>               - do clk_prepare_enable which acutally sets
>>>>>>>>>>>>>>> DFLL in
>>>>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not
>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL 
>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>> disabled anywhere throught the suspend/resume path and SC7
>>>>>>>>>>>>>>> entry
>>>>>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend 
>>>>>>>>>>>>> it on
>>>>>>>>>>>>> PLLP as well. And besides, seems that currently disabling 
>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>> clock will disable DFLL completely and then you'd want to
>>>>>>>>>>>>> re-init
>>>>>>>>>>>>> the DFLL on resume any ways. So better to just disable DFLL
>>>>>>>>>>>>> completely on suspend, which should happen on 
>>>>>>>>>>>>> clk_disable(dfll).
>>>>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With decision of
>>>>>>>>>>>> using
>>>>>>>>>>>> clk_disable during suspend, its mandatory to switch to PLLP as
>>>>>>>>>>>> DFLL
>>>>>>>>>>>> is completely disabled.
>>>>>>>>>>>>
>>>>>>>>>>>> My earlier concern was on restoring CPU policy as we can't do
>>>>>>>>>>>> that
>>>>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>>>>
>>>>>>>>>>>> Clear now and will do CPU clock policy restore in after dfll
>>>>>>>>>>>> re-init.
>>>>>>>>>>> Why the policy can't be saved/restored by the CaR driver as a
>>>>>>>>>>> context of any other clock?
>>>>>>>>>> restoring cpu clock policy involves programming source and
>>>>>>>>>> super_cclkg_divider.
>>>>>>>>>>
>>>>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use
>>>>>>>>>> frac_div ops
>>>>>>>>>> to do save/restore its divider.
>>>>>>>>> That can be changed of course and I guess it also could be as
>>>>>>>>> simple as
>>>>>>>>> saving and restoring of two raw u32 values of the policy/divider
>>>>>>>>> registers.
>>>>>>>>>
>>>>>>>>>> Also, during clock context we cant restore cclk_g as cclk_g 
>>>>>>>>>> source
>>>>>>>>>> will be dfll and dfll will not be resumed/re-initialized by the
>>>>>>>>>> time
>>>>>>>>>> clk_super_mux save/restore happens.
>>>>>>>>>>
>>>>>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and
>>>>>>>>>> dfll_ref
>>>>>>>>>> and dfll_soc peripheral clocks are not restored by the time dfll
>>>>>>>>>> restore happens. Also dfll peripheral clock enables need to be
>>>>>>>>>> restored before dfll restore happens which involves programming
>>>>>>>>>> dfll
>>>>>>>>>> controller for re-initialization.
>>>>>>>>>>
>>>>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of all 
>>>>>>>>>> clocks
>>>>>>>>>> restore in V5 series but instead of in clk-tegra210 driver I 
>>>>>>>>>> moved
>>>>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
>>>>>>>>>> restored thru clk_restore_context by then. This will be in V6.
>>>>>>>>> Since DFLL is now guaranteed to be disabled across CaR
>>>>>>>>> suspend/resume
>>>>>>>>> (hence it has nothing to do in regards to CCLK) and given that 
>>>>>>>>> PLLs
>>>>>>>>> state is restored before the rest of the clocks, I don't see why
>>>>>>>>> not to
>>>>>>>>> implement CCLK save/restore in a generic fasion. CPU policy 
>>>>>>>>> wull be
>>>>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is disabled).
>>>>>>>>>
>>>>>>>> CCLK_G save/restore should happen in clk_super_mux ops
>>>>>>>> save/context and
>>>>>>>> clk_super_mux save/restore happens very early as cclk_g is first
>>>>>>>> in the
>>>>>>>> clock tree and save/restore traverses through the tree top-bottom
>>>>>>>> order.
>>>>>>> If CCLK_G is restored before the PLLs, then just change the clocks
>>>>>>> order
>>>>>>> such that it won't happen.
>>>>>>>
>>>>>> I dont think we can change clocks order for CCLK_G.
>>>>>>
>>>>>> During bootup, cclk_g is registered after all pll's and peripheral
>>>>>> clocks which is the way we wanted, So cclk_g will be the first 
>>>>>> one in
>>>>>> the clk list as clk_register adds new clock first in the list.
>>>>>>
>>>>>> When clk_save_context and clk_restore_context APIs iterates over the
>>>>>> list, cclk_g is the first
>>>>> Looking at clk_core_restore_context(), I see that it walks up CLKs 
>>>>> list
>>>>> from parent to children, hence I don't understand how it can ever 
>>>>> happen
>>>>> that CCLK will be restored before the parent. The clocks registration
>>>>> order doesn't matter at all in that case.
>>>> yes from parent to children and dfllCPU_out is the top in the list and
>>>> its child is cclk_g.
>>>>
>>>> the way clocks are registered is the order I see in the clock list and
>>>> looking into clk_register API it adds new node first in the list.
>>>>
>>> cclkg_g & dfll register happens after all plls and peripheral clocks as
>>> it need ref, soc and peripheral clocks to be enabled.
>>>> So they are the last to get registered and so becomes first in the 
>>>> list.
>>>>
>>>> During save/restore context, it traverses thru this list and first in
>>>> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>>>>
>>>> saving should not be an issue at all but we cant restore cclk_g/dfll
>>>> in normal way thru clk_ops restore as plls and peripherals restore
>>>> doesn't happen by that time.
>>>>
>>> I was referring to clk_restore_context where it iterates thru root list
>>> and for each core from the root list clk_core_restore does restore of
>>> parent and children.
>>>
>>> dfllCPU_Out gets first in the list and its child is cclk_g
>>>
>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105
>> What list you're talking about? clk_summary? It shows current *active*
>> clocks configuration, if you'll try to disable CPUFreq driver then the
>> parent of CCLK_G should be PLLX. Similarly when CPU is reparented to
>> PLLP on driver's suspend, then PLLP is the parent.
>>
>>>>>>>> DFLL enable thru CPUFreq resume happens after all 
>>>>>>>> clk_restore_context
>>>>>>>> happens. So during clk_restore_context, dfll re-init doesnt happen
>>>>>>>> and
>>>>>>>> doing cpu clock policy restore during super_mux clk_ops will 
>>>>>>>> crash as
>>>>>>>> DFLL is not initialized and its clock is not enabled but CPU clock
>>>>>>>> restore sets source to DFLL if we restore during super_clk_mux
>>>>>>> If CPU was suspended on PLLP, then it will be restored on PLLP by
>>>>>>> CaR. I
>>>>>>> don't understand what DFLL has to do with the CCLK in that case 
>>>>>>> during
>>>>>>> the clocks restore.
>>>>>> My above comment is in reference to your request of doing 
>>>>>> save/restore
>>>>>> for cclk_g in normal fashion thru save/restore context. Because 
>>>>>> of the
>>>>>> clk order I mentioned above, we cclk_g will be the first one to 
>>>>>> go thru
>>>>>> save/context.
>>>>>>
>>>>>> During save_context of cclk_g, source can be from PLLX, dfll.
>>>>>>
>>>>>> Issue will be when we do restore during clk_restore_context of
>>>>>> cclk_g as
>>>>>> by that time PLLX/dfll will not be restored.
>>>>>>
>>>>> Seems we already agreed that DFLL will be disabled by the CPUFreq 
>>>>> driver
>>>>> on suspend. Hence CCLK can't be from DFLL if CPU is reparented to 
>>>>> PLLP
>>>>> on CPUFreq driver's suspend, otherwise CPU keeps running from a
>>>>> boot-state PLLX if CPUFreq driver is disabled.
>>>> Yes suspend should not be an issue but issue will be during resume
>>>> where if we do cclk_g restore in normal way thru clk_restore_context,
>>>> cclk_g restore happens very early as dfllCPU out is the first one that
>>>> goes thru restore context and plls/peripherals are not resumed by 
>>>> then.
>>>>
>>>> CPU runs from PLLX if dfll clock enable fails during boot. So when it
>>>> gets to suspend, we save CPU running clock source as either PLLX or
>>>> DFLL and then we switch to PLLP.
>>>>
>>>>
>>>> On resume, CPU runs from PLLP by warm boot code and we need to restore
>>>> back its source to the one it was using from saved source context
>>>> (which can be either PLLX or DFLL)
>>>>
>>>> So PLLs & DFLL resume need to happen before CCLKG restore/resume.
>>>>
>>>>
>>>> With all above discussions, we do DFLL disable in CPUFreq driver on
>>>> suspend and on CPUFreq resume we enable DFLL back and restore CPU
>>>> clock source it was using during suspend (which will be either PLLX if
>>>> dfll enable fails during probe or it will be using DFLL).
>> During suspend CPU's parent shall be PLLP and not DFLL (note that it is
>> disabled) after reparenting to PLLP by the CPUFreq driver.
>>
> CPU source context should be saved before switching to safe source of 
> PLLP as on resume we need to restore back to source it was using 
> before we switch to safe source during suspend entry.
>
> So saved context for CPU Source will be either dfll or PLLX
>
PLLP reparenting is only during suspend/entry to have it as safe source 
but actual CPU source it was running from before suspending is either 
dfll/pllx which should be the one to be restored on CPUFreq resume. 
Resume happens with CPU running from PLLP till it gets to the point of 
restoring its original source (dfll or pllx)
>>>> So i was trying to say dfll/cclk_g restore can't be done in normal way
>>>> thru clk_ops save/restore context
>> Let's see what happens if CPUFreq is active:
>>
>> 1. CPUFreq driver probe happens
>>     2. CPU is reparented to PLLP
>>     3. DFLL inited
>>     4. CPU is reparented to DFLL
>>
>> 5. CPUFreq driver suspend happens
>>     6. CPU is reparented to PLLP
>>     7. DFLL is disabled
>>
>> 8. Car suspend happens
>>     9. DFLL context saved
>>     10. PLLP/PLLX context saved
>>     11. CCLK context saved
>>
>> 12. Car resume happens
>>     13. DFLL context restored
>>     14. PLLP/PLLX context restored
>>     15. CCLK context restored
>>
>> 16. CPUFreq driver resume happens
>>     17. DFLL re-inited
>>     18. CPU is reparented to DFLL
>
>
> Below is the order of sequence it should be based on the order of clk 
> register.
>
> My comments inline in this sequence.
>
> 1. CPUFreq driver probe happens
>     2. CPU is reparented to PLLP
>     3. DFLL inited
>     4. CPU is reparented to DFLL
>
>
> 5. CPUFreq driver suspend happens
>     6. Save CPU source which could be either dfll or pllx
>     7. CPU is reparented to safe known source PLLP
>     8. DFLL is disabled
>
> 8. Car suspend happens
>     9. DFLL context saved (With DFLL disabled in CPUFreq suspend, 
> nothing to be saved here as last freq req will always be saved).
>     10. CCLK context saved (CPU clock source will be saved in CPUFreq 
> driver suspend which could be either dfll or pllx)
>     11. PLLP/PLLX context saved
>     12. Peripheral Clock saved
>
> 12. Car resume happens
>     13. DFLL context restored : No DFLL context to be restored and we 
> only need to reinitialize DFLL and re-initialize can't be done here as 
> this is the 1st to get restored and PLL/Peripheral clocks are not 
> restored by this time. So we can't use clk_ops restore for DFLL
>     14. CCLK context restored
> CCLK cant be restored here as context could be either dfll or pllx 
> which is the source orginally it was actually using before we force 
> switch to safe PLLP for suspend entry. So we can't use clk_ops restore 
> for DFLL
>
> 15. PLLP/PLLX context restored
>     16. Peripheral context restored
>
> 16. CPUFreq driver resume happens
>     17. DFLL re-inited (Invoking DFLL re-init in CPUFreq resume need 
> exporting DFLL reinit from Clock driver to CPUFreq driver)
>     18. CPU is reparented to DFLL or PLLX based on saved context from 
> step 9.
>
> Note: instead of exporting, we can do DFLL re-init from clock-dfll 
> driver itself thru dfll-fcpu pm_ops resume. So dfll will be 
> re-initialized by the time CPUFreq driver resumes and switches to use 
> DFLL source.
>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 20:11                                                                                 ` Sowjanya Komatineni
@ 2019-07-17 21:29                                                                                   ` Sowjanya Komatineni
  2019-07-17 21:30                                                                                   ` Dmitry Osipenko
  1 sibling, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17 21:29 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree


On 7/17/19 1:11 PM, Sowjanya Komatineni wrote:
>
> On 7/17/19 1:01 PM, Sowjanya Komatineni wrote:
>>
>> On 7/17/19 12:43 PM, Dmitry Osipenko wrote:
>>> 17.07.2019 21:54, Sowjanya Komatineni пишет:
>>>> On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>>>>> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>>>>>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>>>>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>>>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>
>>>>>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> links". See
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with
>>>>>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers 
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning 
>>>>>>>>>>>>>>>>>>>>>>>>>> EPROBE_DEFER until
>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine as-is 
>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G
>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed 
>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Hence
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW, 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> done)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend),
>>>>>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open 
>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP 
>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to 
>>>>>>>>>>>>>>>>>>>>>>>>>> switch to
>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, 
>>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be 
>>>>>>>>>>>>>>>>>>>>>>>>>> selected.
>>>>>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X 
>>>>>>>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe 
>>>>>>>>>>>>>>>>>>>>>>>>> prior to
>>>>>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
>>>>>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running 
>>>>>>>>>>>>>>>>>>>>>>>> off a
>>>>>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on 
>>>>>>>>>>>>>>>>>>>>>>>> KEXEC. I
>>>>>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P 
>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used for 
>>>>>>>>>>>>>>>>>>>>>>> high
>>>>>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>>>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to
>>>>>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same 
>>>>>>>>>>>>>>>>>>>> voltage as
>>>>>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me 
>>>>>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is 
>>>>>>>>>>>>>>>>>>>> defined, I see
>>>>>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that
>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>>>>>> Thank you for the clarification. It would be good to 
>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq 
>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>> happens very early even before disabling non-boot CPUs and
>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>               - Save CPU clock policy registers, and 
>>>>>>>>>>>>>>>> Perform
>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>               - Re-init DFLL, Set in Open-Loop mode, 
>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>> Clock policy registers which actually sets source to DFLL
>>>>>>>>>>>>>>>> along
>>>>>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>               - do clk_prepare_enable which acutally sets
>>>>>>>>>>>>>>>> DFLL in
>>>>>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not
>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL 
>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>> disabled anywhere throught the suspend/resume path and SC7
>>>>>>>>>>>>>>>> entry
>>>>>>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend 
>>>>>>>>>>>>>> it on
>>>>>>>>>>>>>> PLLP as well. And besides, seems that currently disabling 
>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>> clock will disable DFLL completely and then you'd want to
>>>>>>>>>>>>>> re-init
>>>>>>>>>>>>>> the DFLL on resume any ways. So better to just disable DFLL
>>>>>>>>>>>>>> completely on suspend, which should happen on 
>>>>>>>>>>>>>> clk_disable(dfll).
>>>>>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With decision of
>>>>>>>>>>>>> using
>>>>>>>>>>>>> clk_disable during suspend, its mandatory to switch to 
>>>>>>>>>>>>> PLLP as
>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>> is completely disabled.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My earlier concern was on restoring CPU policy as we can't do
>>>>>>>>>>>>> that
>>>>>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Clear now and will do CPU clock policy restore in after dfll
>>>>>>>>>>>>> re-init.
>>>>>>>>>>>> Why the policy can't be saved/restored by the CaR driver as a
>>>>>>>>>>>> context of any other clock?
>>>>>>>>>>> restoring cpu clock policy involves programming source and
>>>>>>>>>>> super_cclkg_divider.
>>>>>>>>>>>
>>>>>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use
>>>>>>>>>>> frac_div ops
>>>>>>>>>>> to do save/restore its divider.
>>>>>>>>>> That can be changed of course and I guess it also could be as
>>>>>>>>>> simple as
>>>>>>>>>> saving and restoring of two raw u32 values of the policy/divider
>>>>>>>>>> registers.
>>>>>>>>>>
>>>>>>>>>>> Also, during clock context we cant restore cclk_g as cclk_g 
>>>>>>>>>>> source
>>>>>>>>>>> will be dfll and dfll will not be resumed/re-initialized by the
>>>>>>>>>>> time
>>>>>>>>>>> clk_super_mux save/restore happens.
>>>>>>>>>>>
>>>>>>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and
>>>>>>>>>>> dfll_ref
>>>>>>>>>>> and dfll_soc peripheral clocks are not restored by the time 
>>>>>>>>>>> dfll
>>>>>>>>>>> restore happens. Also dfll peripheral clock enables need to be
>>>>>>>>>>> restored before dfll restore happens which involves programming
>>>>>>>>>>> dfll
>>>>>>>>>>> controller for re-initialization.
>>>>>>>>>>>
>>>>>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of all 
>>>>>>>>>>> clocks
>>>>>>>>>>> restore in V5 series but instead of in clk-tegra210 driver I 
>>>>>>>>>>> moved
>>>>>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
>>>>>>>>>>> restored thru clk_restore_context by then. This will be in V6.
>>>>>>>>>> Since DFLL is now guaranteed to be disabled across CaR
>>>>>>>>>> suspend/resume
>>>>>>>>>> (hence it has nothing to do in regards to CCLK) and given 
>>>>>>>>>> that PLLs
>>>>>>>>>> state is restored before the rest of the clocks, I don't see why
>>>>>>>>>> not to
>>>>>>>>>> implement CCLK save/restore in a generic fasion. CPU policy 
>>>>>>>>>> wull be
>>>>>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is disabled).
>>>>>>>>>>
>>>>>>>>> CCLK_G save/restore should happen in clk_super_mux ops
>>>>>>>>> save/context and
>>>>>>>>> clk_super_mux save/restore happens very early as cclk_g is first
>>>>>>>>> in the
>>>>>>>>> clock tree and save/restore traverses through the tree top-bottom
>>>>>>>>> order.
>>>>>>>> If CCLK_G is restored before the PLLs, then just change the clocks
>>>>>>>> order
>>>>>>>> such that it won't happen.
>>>>>>>>
>>>>>>> I dont think we can change clocks order for CCLK_G.
>>>>>>>
>>>>>>> During bootup, cclk_g is registered after all pll's and peripheral
>>>>>>> clocks which is the way we wanted, So cclk_g will be the first 
>>>>>>> one in
>>>>>>> the clk list as clk_register adds new clock first in the list.
>>>>>>>
>>>>>>> When clk_save_context and clk_restore_context APIs iterates over 
>>>>>>> the
>>>>>>> list, cclk_g is the first
>>>>>> Looking at clk_core_restore_context(), I see that it walks up 
>>>>>> CLKs list
>>>>>> from parent to children, hence I don't understand how it can ever 
>>>>>> happen
>>>>>> that CCLK will be restored before the parent. The clocks 
>>>>>> registration
>>>>>> order doesn't matter at all in that case.
>>>>> yes from parent to children and dfllCPU_out is the top in the list 
>>>>> and
>>>>> its child is cclk_g.
>>>>>
>>>>> the way clocks are registered is the order I see in the clock list 
>>>>> and
>>>>> looking into clk_register API it adds new node first in the list.
>>>>>
>>>> cclkg_g & dfll register happens after all plls and peripheral 
>>>> clocks as
>>>> it need ref, soc and peripheral clocks to be enabled.
>>>>> So they are the last to get registered and so becomes first in the 
>>>>> list.
>>>>>
>>>>> During save/restore context, it traverses thru this list and first in
>>>>> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>>>>>
>>>>> saving should not be an issue at all but we cant restore cclk_g/dfll
>>>>> in normal way thru clk_ops restore as plls and peripherals restore
>>>>> doesn't happen by that time.
>>>>>
>>>> I was referring to clk_restore_context where it iterates thru root 
>>>> list
>>>> and for each core from the root list clk_core_restore does restore of
>>>> parent and children.
>>>>
>>>> dfllCPU_Out gets first in the list and its child is cclk_g
>>>>
>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105
>>> What list you're talking about? clk_summary? It shows current *active*
>>> clocks configuration, if you'll try to disable CPUFreq driver then the
>>> parent of CCLK_G should be PLLX. Similarly when CPU is reparented to
>>> PLLP on driver's suspend, then PLLP is the parent.
>>>
>>>>>>>>> DFLL enable thru CPUFreq resume happens after all 
>>>>>>>>> clk_restore_context
>>>>>>>>> happens. So during clk_restore_context, dfll re-init doesnt 
>>>>>>>>> happen
>>>>>>>>> and
>>>>>>>>> doing cpu clock policy restore during super_mux clk_ops will 
>>>>>>>>> crash as
>>>>>>>>> DFLL is not initialized and its clock is not enabled but CPU 
>>>>>>>>> clock
>>>>>>>>> restore sets source to DFLL if we restore during super_clk_mux
>>>>>>>> If CPU was suspended on PLLP, then it will be restored on PLLP by
>>>>>>>> CaR. I
>>>>>>>> don't understand what DFLL has to do with the CCLK in that case 
>>>>>>>> during
>>>>>>>> the clocks restore.
>>>>>>> My above comment is in reference to your request of doing 
>>>>>>> save/restore
>>>>>>> for cclk_g in normal fashion thru save/restore context. Because 
>>>>>>> of the
>>>>>>> clk order I mentioned above, we cclk_g will be the first one to 
>>>>>>> go thru
>>>>>>> save/context.
>>>>>>>
>>>>>>> During save_context of cclk_g, source can be from PLLX, dfll.
>>>>>>>
>>>>>>> Issue will be when we do restore during clk_restore_context of
>>>>>>> cclk_g as
>>>>>>> by that time PLLX/dfll will not be restored.
>>>>>>>
>>>>>> Seems we already agreed that DFLL will be disabled by the CPUFreq 
>>>>>> driver
>>>>>> on suspend. Hence CCLK can't be from DFLL if CPU is reparented to 
>>>>>> PLLP
>>>>>> on CPUFreq driver's suspend, otherwise CPU keeps running from a
>>>>>> boot-state PLLX if CPUFreq driver is disabled.
>>>>> Yes suspend should not be an issue but issue will be during resume
>>>>> where if we do cclk_g restore in normal way thru clk_restore_context,
>>>>> cclk_g restore happens very early as dfllCPU out is the first one 
>>>>> that
>>>>> goes thru restore context and plls/peripherals are not resumed by 
>>>>> then.
>>>>>
>>>>> CPU runs from PLLX if dfll clock enable fails during boot. So when it
>>>>> gets to suspend, we save CPU running clock source as either PLLX or
>>>>> DFLL and then we switch to PLLP.
>>>>>
>>>>>
>>>>> On resume, CPU runs from PLLP by warm boot code and we need to 
>>>>> restore
>>>>> back its source to the one it was using from saved source context
>>>>> (which can be either PLLX or DFLL)
>>>>>
>>>>> So PLLs & DFLL resume need to happen before CCLKG restore/resume.
>>>>>
>>>>>
>>>>> With all above discussions, we do DFLL disable in CPUFreq driver on
>>>>> suspend and on CPUFreq resume we enable DFLL back and restore CPU
>>>>> clock source it was using during suspend (which will be either 
>>>>> PLLX if
>>>>> dfll enable fails during probe or it will be using DFLL).
>>> During suspend CPU's parent shall be PLLP and not DFLL (note that it is
>>> disabled) after reparenting to PLLP by the CPUFreq driver.
>>>
>> CPU source context should be saved before switching to safe source of 
>> PLLP as on resume we need to restore back to source it was using 
>> before we switch to safe source during suspend entry.
>>
>> So saved context for CPU Source will be either dfll or PLLX
>>
> PLLP reparenting is only during suspend/entry to have it as safe 
> source but actual CPU source it was running from before suspending is 
> either dfll/pllx which should be the one to be restored on CPUFreq 
> resume. Resume happens with CPU running from PLLP till it gets to the 
> point of restoring its original source (dfll or pllx)
>>>>> So i was trying to say dfll/cclk_g restore can't be done in normal 
>>>>> way
>>>>> thru clk_ops save/restore context
>>> Let's see what happens if CPUFreq is active:
>>>
>>> 1. CPUFreq driver probe happens
>>>     2. CPU is reparented to PLLP
>>>     3. DFLL inited
>>>     4. CPU is reparented to DFLL
>>>
>>> 5. CPUFreq driver suspend happens
>>>     6. CPU is reparented to PLLP
>>>     7. DFLL is disabled
>>>
>>> 8. Car suspend happens
>>>     9. DFLL context saved
>>>     10. PLLP/PLLX context saved
>>>     11. CCLK context saved
>>>
>>> 12. Car resume happens
>>>     13. DFLL context restored
>>>     14. PLLP/PLLX context restored
>>>     15. CCLK context restored
>>>
>>> 16. CPUFreq driver resume happens
>>>     17. DFLL re-inited
>>>     18. CPU is reparented to DFLL
>>
>>
>> Below is the order of sequence it should be based on the order of clk 
>> register.
>>
>> My comments inline in this sequence.
>>
>> 1. CPUFreq driver probe happens
>>     2. CPU is reparented to PLLP
>>     3. DFLL inited
>>     4. CPU is reparented to DFLL
>>
>>
>> 5. CPUFreq driver suspend happens
>>     6. Save CPU source which could be either dfll or pllx
>>     7. CPU is reparented to safe known source PLLP
>>     8. DFLL is disabled
>>
>> 8. Car suspend happens
>>     9. DFLL context saved (With DFLL disabled in CPUFreq suspend, 
>> nothing to be saved here as last freq req will always be saved).
>>     10. CCLK context saved (CPU clock source will be saved in CPUFreq 
>> driver suspend which could be either dfll or pllx)
>>     11. PLLP/PLLX context saved
>>     12. Peripheral Clock saved
>>
>> 12. Car resume happens
>>     13. DFLL context restored : No DFLL context to be restored and we 
>> only need to reinitialize DFLL and re-initialize can't be done here 
>> as this is the 1st to get restored and PLL/Peripheral clocks are not 
>> restored by this time. So we can't use clk_ops restore for DFLL
>>     14. CCLK context restored
>> CCLK cant be restored here as context could be either dfll or pllx 
>> which is the source orginally it was actually using before we force 
>> switch to safe PLLP for suspend entry. So we can't use clk_ops 
>> restore for DFLL
>>
>> 15. PLLP/PLLX context restored
>>     16. Peripheral context restored
>>
>> 16. CPUFreq driver resume happens
>>     17. DFLL re-inited (Invoking DFLL re-init in CPUFreq resume need 
>> exporting DFLL reinit from Clock driver to CPUFreq driver)
>>     18. CPU is reparented to DFLL or PLLX based on saved context from 
>> step 9.
>>
>> Note: instead of exporting, we can do DFLL re-init from clock-dfll 
>> driver itself thru dfll-fcpu pm_ops resume. So dfll will be 
>> re-initialized by the time CPUFreq driver resumes and switches to use 
>> DFLL source.
>>
Tested above commented sequence and looks as expected. Will send V6 
after addressing other comments of V5 patch series for further review...



^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 20:11                                                                                 ` Sowjanya Komatineni
  2019-07-17 21:29                                                                                   ` Sowjanya Komatineni
@ 2019-07-17 21:30                                                                                   ` Dmitry Osipenko
  2019-07-17 21:51                                                                                     ` Sowjanya Komatineni
  1 sibling, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-17 21:30 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree

17.07.2019 23:11, Sowjanya Komatineni пишет:
> 
> On 7/17/19 1:01 PM, Sowjanya Komatineni wrote:
>>
>> On 7/17/19 12:43 PM, Dmitry Osipenko wrote:
>>> 17.07.2019 21:54, Sowjanya Komatineni пишет:
>>>> On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>>>>> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>>>>>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>>>>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>>>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>
>>>>>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> See
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with
>>>>>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the
>>>>>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER
>>>>>>>>>>>>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine as-is and
>>>>>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G
>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed
>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hence
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend),
>>>>>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open
>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to
>>>>>>>>>>>>>>>>>>>>>>>>>> switch to
>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then
>>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be
>>>>>>>>>>>>>>>>>>>>>>>>>> selected.
>>>>>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X
>>>>>>>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe
>>>>>>>>>>>>>>>>>>>>>>>>> prior to
>>>>>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
>>>>>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running
>>>>>>>>>>>>>>>>>>>>>>>> off a
>>>>>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I
>>>>>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P
>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used for high
>>>>>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>>>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to
>>>>>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same
>>>>>>>>>>>>>>>>>>>> voltage as
>>>>>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me
>>>>>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined,
>>>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that
>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>>>>>> Thank you for the clarification. It would be good to have
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq
>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>> happens very early even before disabling non-boot CPUs and
>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>               - Save CPU clock policy registers, and
>>>>>>>>>>>>>>>> Perform
>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>               - Re-init DFLL, Set in Open-Loop mode,
>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>> Clock policy registers which actually sets source to DFLL
>>>>>>>>>>>>>>>> along
>>>>>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>               - do clk_prepare_enable which acutally sets
>>>>>>>>>>>>>>>> DFLL in
>>>>>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not
>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL
>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>> disabled anywhere throught the suspend/resume path and SC7
>>>>>>>>>>>>>>>> entry
>>>>>>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend
>>>>>>>>>>>>>> it on
>>>>>>>>>>>>>> PLLP as well. And besides, seems that currently disabling
>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>> clock will disable DFLL completely and then you'd want to
>>>>>>>>>>>>>> re-init
>>>>>>>>>>>>>> the DFLL on resume any ways. So better to just disable DFLL
>>>>>>>>>>>>>> completely on suspend, which should happen on
>>>>>>>>>>>>>> clk_disable(dfll).
>>>>>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With decision of
>>>>>>>>>>>>> using
>>>>>>>>>>>>> clk_disable during suspend, its mandatory to switch to PLLP as
>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>> is completely disabled.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My earlier concern was on restoring CPU policy as we can't do
>>>>>>>>>>>>> that
>>>>>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Clear now and will do CPU clock policy restore in after dfll
>>>>>>>>>>>>> re-init.
>>>>>>>>>>>> Why the policy can't be saved/restored by the CaR driver as a
>>>>>>>>>>>> context of any other clock?
>>>>>>>>>>> restoring cpu clock policy involves programming source and
>>>>>>>>>>> super_cclkg_divider.
>>>>>>>>>>>
>>>>>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use
>>>>>>>>>>> frac_div ops
>>>>>>>>>>> to do save/restore its divider.
>>>>>>>>>> That can be changed of course and I guess it also could be as
>>>>>>>>>> simple as
>>>>>>>>>> saving and restoring of two raw u32 values of the policy/divider
>>>>>>>>>> registers.
>>>>>>>>>>
>>>>>>>>>>> Also, during clock context we cant restore cclk_g as cclk_g
>>>>>>>>>>> source
>>>>>>>>>>> will be dfll and dfll will not be resumed/re-initialized by the
>>>>>>>>>>> time
>>>>>>>>>>> clk_super_mux save/restore happens.
>>>>>>>>>>>
>>>>>>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and
>>>>>>>>>>> dfll_ref
>>>>>>>>>>> and dfll_soc peripheral clocks are not restored by the time dfll
>>>>>>>>>>> restore happens. Also dfll peripheral clock enables need to be
>>>>>>>>>>> restored before dfll restore happens which involves programming
>>>>>>>>>>> dfll
>>>>>>>>>>> controller for re-initialization.
>>>>>>>>>>>
>>>>>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of all
>>>>>>>>>>> clocks
>>>>>>>>>>> restore in V5 series but instead of in clk-tegra210 driver I
>>>>>>>>>>> moved
>>>>>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
>>>>>>>>>>> restored thru clk_restore_context by then. This will be in V6.
>>>>>>>>>> Since DFLL is now guaranteed to be disabled across CaR
>>>>>>>>>> suspend/resume
>>>>>>>>>> (hence it has nothing to do in regards to CCLK) and given that
>>>>>>>>>> PLLs
>>>>>>>>>> state is restored before the rest of the clocks, I don't see why
>>>>>>>>>> not to
>>>>>>>>>> implement CCLK save/restore in a generic fasion. CPU policy
>>>>>>>>>> wull be
>>>>>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is disabled).
>>>>>>>>>>
>>>>>>>>> CCLK_G save/restore should happen in clk_super_mux ops
>>>>>>>>> save/context and
>>>>>>>>> clk_super_mux save/restore happens very early as cclk_g is first
>>>>>>>>> in the
>>>>>>>>> clock tree and save/restore traverses through the tree top-bottom
>>>>>>>>> order.
>>>>>>>> If CCLK_G is restored before the PLLs, then just change the clocks
>>>>>>>> order
>>>>>>>> such that it won't happen.
>>>>>>>>
>>>>>>> I dont think we can change clocks order for CCLK_G.
>>>>>>>
>>>>>>> During bootup, cclk_g is registered after all pll's and peripheral
>>>>>>> clocks which is the way we wanted, So cclk_g will be the first
>>>>>>> one in
>>>>>>> the clk list as clk_register adds new clock first in the list.
>>>>>>>
>>>>>>> When clk_save_context and clk_restore_context APIs iterates over the
>>>>>>> list, cclk_g is the first
>>>>>> Looking at clk_core_restore_context(), I see that it walks up CLKs
>>>>>> list
>>>>>> from parent to children, hence I don't understand how it can ever
>>>>>> happen
>>>>>> that CCLK will be restored before the parent. The clocks registration
>>>>>> order doesn't matter at all in that case.
>>>>> yes from parent to children and dfllCPU_out is the top in the list and
>>>>> its child is cclk_g.
>>>>>
>>>>> the way clocks are registered is the order I see in the clock list and
>>>>> looking into clk_register API it adds new node first in the list.
>>>>>
>>>> cclkg_g & dfll register happens after all plls and peripheral clocks as
>>>> it need ref, soc and peripheral clocks to be enabled.
>>>>> So they are the last to get registered and so becomes first in the
>>>>> list.
>>>>>
>>>>> During save/restore context, it traverses thru this list and first in
>>>>> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>>>>>
>>>>> saving should not be an issue at all but we cant restore cclk_g/dfll
>>>>> in normal way thru clk_ops restore as plls and peripherals restore
>>>>> doesn't happen by that time.
>>>>>
>>>> I was referring to clk_restore_context where it iterates thru root list
>>>> and for each core from the root list clk_core_restore does restore of
>>>> parent and children.
>>>>
>>>> dfllCPU_Out gets first in the list and its child is cclk_g
>>>>
>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105
>>> What list you're talking about? clk_summary? It shows current *active*
>>> clocks configuration, if you'll try to disable CPUFreq driver then the
>>> parent of CCLK_G should be PLLX. Similarly when CPU is reparented to
>>> PLLP on driver's suspend, then PLLP is the parent.
>>>
>>>>>>>>> DFLL enable thru CPUFreq resume happens after all
>>>>>>>>> clk_restore_context
>>>>>>>>> happens. So during clk_restore_context, dfll re-init doesnt happen
>>>>>>>>> and
>>>>>>>>> doing cpu clock policy restore during super_mux clk_ops will
>>>>>>>>> crash as
>>>>>>>>> DFLL is not initialized and its clock is not enabled but CPU clock
>>>>>>>>> restore sets source to DFLL if we restore during super_clk_mux
>>>>>>>> If CPU was suspended on PLLP, then it will be restored on PLLP by
>>>>>>>> CaR. I
>>>>>>>> don't understand what DFLL has to do with the CCLK in that case
>>>>>>>> during
>>>>>>>> the clocks restore.
>>>>>>> My above comment is in reference to your request of doing
>>>>>>> save/restore
>>>>>>> for cclk_g in normal fashion thru save/restore context. Because
>>>>>>> of the
>>>>>>> clk order I mentioned above, we cclk_g will be the first one to
>>>>>>> go thru
>>>>>>> save/context.
>>>>>>>
>>>>>>> During save_context of cclk_g, source can be from PLLX, dfll.
>>>>>>>
>>>>>>> Issue will be when we do restore during clk_restore_context of
>>>>>>> cclk_g as
>>>>>>> by that time PLLX/dfll will not be restored.
>>>>>>>
>>>>>> Seems we already agreed that DFLL will be disabled by the CPUFreq
>>>>>> driver
>>>>>> on suspend. Hence CCLK can't be from DFLL if CPU is reparented to
>>>>>> PLLP
>>>>>> on CPUFreq driver's suspend, otherwise CPU keeps running from a
>>>>>> boot-state PLLX if CPUFreq driver is disabled.
>>>>> Yes suspend should not be an issue but issue will be during resume
>>>>> where if we do cclk_g restore in normal way thru clk_restore_context,
>>>>> cclk_g restore happens very early as dfllCPU out is the first one that
>>>>> goes thru restore context and plls/peripherals are not resumed by
>>>>> then.
>>>>>
>>>>> CPU runs from PLLX if dfll clock enable fails during boot. So when it
>>>>> gets to suspend, we save CPU running clock source as either PLLX or
>>>>> DFLL and then we switch to PLLP.
>>>>>
>>>>>
>>>>> On resume, CPU runs from PLLP by warm boot code and we need to restore
>>>>> back its source to the one it was using from saved source context
>>>>> (which can be either PLLX or DFLL)
>>>>>
>>>>> So PLLs & DFLL resume need to happen before CCLKG restore/resume.
>>>>>
>>>>>
>>>>> With all above discussions, we do DFLL disable in CPUFreq driver on
>>>>> suspend and on CPUFreq resume we enable DFLL back and restore CPU
>>>>> clock source it was using during suspend (which will be either PLLX if
>>>>> dfll enable fails during probe or it will be using DFLL).
>>> During suspend CPU's parent shall be PLLP and not DFLL (note that it is
>>> disabled) after reparenting to PLLP by the CPUFreq driver.
>>>
>> CPU source context should be saved before switching to safe source of
>> PLLP as on resume we need to restore back to source it was using
>> before we switch to safe source during suspend entry.
>>
>> So saved context for CPU Source will be either dfll or PLLX
>>
> PLLP reparenting is only during suspend/entry to have it as safe source
> but actual CPU source it was running from before suspending is either
> dfll/pllx which should be the one to be restored on CPUFreq resume.
> Resume happens with CPU running from PLLP till it gets to the point of
> restoring its original source (dfll or pllx)

CaR should restore CPU to PLLP or PLLX, while CPUFreq driver restores
CPU to DFLL. Please see more comments below.

>>>>> So i was trying to say dfll/cclk_g restore can't be done in normal way
>>>>> thru clk_ops save/restore context
>>> Let's see what happens if CPUFreq is active:
>>>
>>> 1. CPUFreq driver probe happens
>>>     2. CPU is reparented to PLLP
>>>     3. DFLL inited
>>>     4. CPU is reparented to DFLL
>>>
>>> 5. CPUFreq driver suspend happens
>>>     6. CPU is reparented to PLLP
>>>     7. DFLL is disabled
>>>
>>> 8. Car suspend happens
>>>     9. DFLL context saved
>>>     10. PLLP/PLLX context saved
>>>     11. CCLK context saved
>>>
>>> 12. Car resume happens
>>>     13. DFLL context restored
>>>     14. PLLP/PLLX context restored
>>>     15. CCLK context restored
>>>
>>> 16. CPUFreq driver resume happens
>>>     17. DFLL re-inited
>>>     18. CPU is reparented to DFLL
>>
>>
>> Below is the order of sequence it should be based on the order of clk
>> register.
>>
>> My comments inline in this sequence.
>>
>> 1. CPUFreq driver probe happens
>>     2. CPU is reparented to PLLP
>>     3. DFLL inited
>>     4. CPU is reparented to DFLL
>>
>>
>> 5. CPUFreq driver suspend happens
>>     6. Save CPU source which could be either dfll or pllx

Please see my next comment.

>>     7. CPU is reparented to safe known source PLLP
>>     8. DFLL is disabled
>>
>> 8. Car suspend happens
>>     9. DFLL context saved (With DFLL disabled in CPUFreq suspend,
>> nothing to be saved here as last freq req will always be saved).
>>     10. CCLK context saved (CPU clock source will be saved in CPUFreq
>> driver suspend which could be either dfll or pllx)

That I don't understand. The CPU's clock source state should be saved at
the moment of the CaR's suspending (i.e. CCLK policy will be set to PLLP
or PLLX) and then CCLK state should be also restored by the CaR in step 14.

CPUFreq driver should only switch CPU to PLLP and disable DFLL on
suspend in step 5, that's it. On resume CPUFreq driver will restore CPU
to DFLL in step 18.

>>     11. PLLP/PLLX context saved
>>     12. Peripheral Clock saved
>>
>> 12. Car resume happens
>>     13. DFLL context restored : No DFLL context to be restored and we
>> only need to reinitialize DFLL and re-initialize can't be done here as
>> this is the 1st to get restored and PLL/Peripheral clocks are not
>> restored by this time. So we can't use clk_ops restore for DFLL

It looks to me that clk_core_restore_context() should just do
hlist_for_each_entry *_reverse*. Don't you think so?

>>     14. CCLK context restored
>> CCLK cant be restored here as context could be either dfll or pllx
>> which is the source orginally it was actually using before we force
>> switch to safe PLLP for suspend entry. So we can't use clk_ops restore
>> for DFLL

See my comment to step 10. CCLK should be restored to the *CaR's saved*
context, which is either PLLX or PLLP policy.

>> 15. PLLP/PLLX context restored
>>     16. Peripheral context restored
>>
>> 16. CPUFreq driver resume happens
>>     17. DFLL re-inited (Invoking DFLL re-init in CPUFreq resume need
>> exporting DFLL reinit from Clock driver to CPUFreq driver)

Will be nice if clk_enable(dfll) could be enough to re-init DFLL. That
should achievable with my next comment to step 18.

>>     18. CPU is reparented to DFLL or PLLX based on saved context from
>> step 9.
>>
>> Note: instead of exporting, we can do DFLL re-init from clock-dfll
>> driver itself thru dfll-fcpu pm_ops resume. So dfll will be
>> re-initialized by the time CPUFreq driver resumes and switches to use
>> DFLL source.
>>

Yes, I guess DFLL hardware should be fully reset on DFLL's driver resume
to be on a safe side any ways.

But(!) we could probably just fix clk_core_restore_context(), like I
suggested in step 13. Then DFLL clock could use generic save / restore
context and CPUFreq driver won't have to do anything at all because DFLL
clock will be saved first and resumed *after* all of the peripherals by
the CCF. In the end CCLK will be switched to DFLL by the CCF restore as
well.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 21:30                                                                                   ` Dmitry Osipenko
@ 2019-07-17 21:51                                                                                     ` Sowjanya Komatineni
  2019-07-17 21:57                                                                                       ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17 21:51 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree


On 7/17/19 2:30 PM, Dmitry Osipenko wrote:
> 17.07.2019 23:11, Sowjanya Komatineni пишет:
>> On 7/17/19 1:01 PM, Sowjanya Komatineni wrote:
>>> On 7/17/19 12:43 PM, Dmitry Osipenko wrote:
>>>> 17.07.2019 21:54, Sowjanya Komatineni пишет:
>>>>> On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>>>>>> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>>>>>>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>>>>>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>>>>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>>>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>
>>>>>>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> See
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with
>>>>>>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver registers the
>>>>>>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER
>>>>>>>>>>>>>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine as-is and
>>>>>>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the DVFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing CCLK_G
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed
>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU cluster. And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching CCLK to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hence
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much nicer to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the device
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver hacks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been done)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend),
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open
>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to
>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to
>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than PLLP, then
>>>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be
>>>>>>>>>>>>>>>>>>>>>>>>>>> selected.
>>>>>>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X
>>>>>>>>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe
>>>>>>>>>>>>>>>>>>>>>>>>>> prior to
>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is also a
>>>>>>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running
>>>>>>>>>>>>>>>>>>>>>>>>> off a
>>>>>>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on KEXEC. I
>>>>>>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not very
>>>>>>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P
>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used for high
>>>>>>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from PLLP.
>>>>>>>>>>>>>>>>>>>>>>> Alright, then please don't forget to pre-initialize
>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is safe to
>>>>>>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same
>>>>>>>>>>>>>>>>>>>>> voltage as
>>>>>>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me
>>>>>>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined,
>>>>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below that
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>>>>>>> Thank you for the clarification. It would be good to have
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq
>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>> happens very early even before disabling non-boot CPUs and
>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                - Save CPU clock policy registers, and
>>>>>>>>>>>>>>>>> Perform
>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                - Re-init DFLL, Set in Open-Loop mode,
>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>> Clock policy registers which actually sets source to DFLL
>>>>>>>>>>>>>>>>> along
>>>>>>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                - do clk_prepare_enable which acutally sets
>>>>>>>>>>>>>>>>> DFLL in
>>>>>>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not
>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL
>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>> disabled anywhere throught the suspend/resume path and SC7
>>>>>>>>>>>>>>>>> entry
>>>>>>>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend
>>>>>>>>>>>>>>> it on
>>>>>>>>>>>>>>> PLLP as well. And besides, seems that currently disabling
>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>> clock will disable DFLL completely and then you'd want to
>>>>>>>>>>>>>>> re-init
>>>>>>>>>>>>>>> the DFLL on resume any ways. So better to just disable DFLL
>>>>>>>>>>>>>>> completely on suspend, which should happen on
>>>>>>>>>>>>>>> clk_disable(dfll).
>>>>>>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With decision of
>>>>>>>>>>>>>> using
>>>>>>>>>>>>>> clk_disable during suspend, its mandatory to switch to PLLP as
>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>> is completely disabled.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> My earlier concern was on restoring CPU policy as we can't do
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Clear now and will do CPU clock policy restore in after dfll
>>>>>>>>>>>>>> re-init.
>>>>>>>>>>>>> Why the policy can't be saved/restored by the CaR driver as a
>>>>>>>>>>>>> context of any other clock?
>>>>>>>>>>>> restoring cpu clock policy involves programming source and
>>>>>>>>>>>> super_cclkg_divider.
>>>>>>>>>>>>
>>>>>>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use
>>>>>>>>>>>> frac_div ops
>>>>>>>>>>>> to do save/restore its divider.
>>>>>>>>>>> That can be changed of course and I guess it also could be as
>>>>>>>>>>> simple as
>>>>>>>>>>> saving and restoring of two raw u32 values of the policy/divider
>>>>>>>>>>> registers.
>>>>>>>>>>>
>>>>>>>>>>>> Also, during clock context we cant restore cclk_g as cclk_g
>>>>>>>>>>>> source
>>>>>>>>>>>> will be dfll and dfll will not be resumed/re-initialized by the
>>>>>>>>>>>> time
>>>>>>>>>>>> clk_super_mux save/restore happens.
>>>>>>>>>>>>
>>>>>>>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and
>>>>>>>>>>>> dfll_ref
>>>>>>>>>>>> and dfll_soc peripheral clocks are not restored by the time dfll
>>>>>>>>>>>> restore happens. Also dfll peripheral clock enables need to be
>>>>>>>>>>>> restored before dfll restore happens which involves programming
>>>>>>>>>>>> dfll
>>>>>>>>>>>> controller for re-initialization.
>>>>>>>>>>>>
>>>>>>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of all
>>>>>>>>>>>> clocks
>>>>>>>>>>>> restore in V5 series but instead of in clk-tegra210 driver I
>>>>>>>>>>>> moved
>>>>>>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies will be
>>>>>>>>>>>> restored thru clk_restore_context by then. This will be in V6.
>>>>>>>>>>> Since DFLL is now guaranteed to be disabled across CaR
>>>>>>>>>>> suspend/resume
>>>>>>>>>>> (hence it has nothing to do in regards to CCLK) and given that
>>>>>>>>>>> PLLs
>>>>>>>>>>> state is restored before the rest of the clocks, I don't see why
>>>>>>>>>>> not to
>>>>>>>>>>> implement CCLK save/restore in a generic fasion. CPU policy
>>>>>>>>>>> wull be
>>>>>>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is disabled).
>>>>>>>>>>>
>>>>>>>>>> CCLK_G save/restore should happen in clk_super_mux ops
>>>>>>>>>> save/context and
>>>>>>>>>> clk_super_mux save/restore happens very early as cclk_g is first
>>>>>>>>>> in the
>>>>>>>>>> clock tree and save/restore traverses through the tree top-bottom
>>>>>>>>>> order.
>>>>>>>>> If CCLK_G is restored before the PLLs, then just change the clocks
>>>>>>>>> order
>>>>>>>>> such that it won't happen.
>>>>>>>>>
>>>>>>>> I dont think we can change clocks order for CCLK_G.
>>>>>>>>
>>>>>>>> During bootup, cclk_g is registered after all pll's and peripheral
>>>>>>>> clocks which is the way we wanted, So cclk_g will be the first
>>>>>>>> one in
>>>>>>>> the clk list as clk_register adds new clock first in the list.
>>>>>>>>
>>>>>>>> When clk_save_context and clk_restore_context APIs iterates over the
>>>>>>>> list, cclk_g is the first
>>>>>>> Looking at clk_core_restore_context(), I see that it walks up CLKs
>>>>>>> list
>>>>>>> from parent to children, hence I don't understand how it can ever
>>>>>>> happen
>>>>>>> that CCLK will be restored before the parent. The clocks registration
>>>>>>> order doesn't matter at all in that case.
>>>>>> yes from parent to children and dfllCPU_out is the top in the list and
>>>>>> its child is cclk_g.
>>>>>>
>>>>>> the way clocks are registered is the order I see in the clock list and
>>>>>> looking into clk_register API it adds new node first in the list.
>>>>>>
>>>>> cclkg_g & dfll register happens after all plls and peripheral clocks as
>>>>> it need ref, soc and peripheral clocks to be enabled.
>>>>>> So they are the last to get registered and so becomes first in the
>>>>>> list.
>>>>>>
>>>>>> During save/restore context, it traverses thru this list and first in
>>>>>> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>>>>>>
>>>>>> saving should not be an issue at all but we cant restore cclk_g/dfll
>>>>>> in normal way thru clk_ops restore as plls and peripherals restore
>>>>>> doesn't happen by that time.
>>>>>>
>>>>> I was referring to clk_restore_context where it iterates thru root list
>>>>> and for each core from the root list clk_core_restore does restore of
>>>>> parent and children.
>>>>>
>>>>> dfllCPU_Out gets first in the list and its child is cclk_g
>>>>>
>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105
>>>> What list you're talking about? clk_summary? It shows current *active*
>>>> clocks configuration, if you'll try to disable CPUFreq driver then the
>>>> parent of CCLK_G should be PLLX. Similarly when CPU is reparented to
>>>> PLLP on driver's suspend, then PLLP is the parent.
>>>>
>>>>>>>>>> DFLL enable thru CPUFreq resume happens after all
>>>>>>>>>> clk_restore_context
>>>>>>>>>> happens. So during clk_restore_context, dfll re-init doesnt happen
>>>>>>>>>> and
>>>>>>>>>> doing cpu clock policy restore during super_mux clk_ops will
>>>>>>>>>> crash as
>>>>>>>>>> DFLL is not initialized and its clock is not enabled but CPU clock
>>>>>>>>>> restore sets source to DFLL if we restore during super_clk_mux
>>>>>>>>> If CPU was suspended on PLLP, then it will be restored on PLLP by
>>>>>>>>> CaR. I
>>>>>>>>> don't understand what DFLL has to do with the CCLK in that case
>>>>>>>>> during
>>>>>>>>> the clocks restore.
>>>>>>>> My above comment is in reference to your request of doing
>>>>>>>> save/restore
>>>>>>>> for cclk_g in normal fashion thru save/restore context. Because
>>>>>>>> of the
>>>>>>>> clk order I mentioned above, we cclk_g will be the first one to
>>>>>>>> go thru
>>>>>>>> save/context.
>>>>>>>>
>>>>>>>> During save_context of cclk_g, source can be from PLLX, dfll.
>>>>>>>>
>>>>>>>> Issue will be when we do restore during clk_restore_context of
>>>>>>>> cclk_g as
>>>>>>>> by that time PLLX/dfll will not be restored.
>>>>>>>>
>>>>>>> Seems we already agreed that DFLL will be disabled by the CPUFreq
>>>>>>> driver
>>>>>>> on suspend. Hence CCLK can't be from DFLL if CPU is reparented to
>>>>>>> PLLP
>>>>>>> on CPUFreq driver's suspend, otherwise CPU keeps running from a
>>>>>>> boot-state PLLX if CPUFreq driver is disabled.
>>>>>> Yes suspend should not be an issue but issue will be during resume
>>>>>> where if we do cclk_g restore in normal way thru clk_restore_context,
>>>>>> cclk_g restore happens very early as dfllCPU out is the first one that
>>>>>> goes thru restore context and plls/peripherals are not resumed by
>>>>>> then.
>>>>>>
>>>>>> CPU runs from PLLX if dfll clock enable fails during boot. So when it
>>>>>> gets to suspend, we save CPU running clock source as either PLLX or
>>>>>> DFLL and then we switch to PLLP.
>>>>>>
>>>>>>
>>>>>> On resume, CPU runs from PLLP by warm boot code and we need to restore
>>>>>> back its source to the one it was using from saved source context
>>>>>> (which can be either PLLX or DFLL)
>>>>>>
>>>>>> So PLLs & DFLL resume need to happen before CCLKG restore/resume.
>>>>>>
>>>>>>
>>>>>> With all above discussions, we do DFLL disable in CPUFreq driver on
>>>>>> suspend and on CPUFreq resume we enable DFLL back and restore CPU
>>>>>> clock source it was using during suspend (which will be either PLLX if
>>>>>> dfll enable fails during probe or it will be using DFLL).
>>>> During suspend CPU's parent shall be PLLP and not DFLL (note that it is
>>>> disabled) after reparenting to PLLP by the CPUFreq driver.
>>>>
>>> CPU source context should be saved before switching to safe source of
>>> PLLP as on resume we need to restore back to source it was using
>>> before we switch to safe source during suspend entry.
>>>
>>> So saved context for CPU Source will be either dfll or PLLX
>>>
>> PLLP reparenting is only during suspend/entry to have it as safe source
>> but actual CPU source it was running from before suspending is either
>> dfll/pllx which should be the one to be restored on CPUFreq resume.
>> Resume happens with CPU running from PLLP till it gets to the point of
>> restoring its original source (dfll or pllx)
> CaR should restore CPU to PLLP or PLLX, while CPUFreq driver restores
> CPU to DFLL. Please see more comments below.
>
>>>>>> So i was trying to say dfll/cclk_g restore can't be done in normal way
>>>>>> thru clk_ops save/restore context
>>>> Let's see what happens if CPUFreq is active:
>>>>
>>>> 1. CPUFreq driver probe happens
>>>>      2. CPU is reparented to PLLP
>>>>      3. DFLL inited
>>>>      4. CPU is reparented to DFLL
>>>>
>>>> 5. CPUFreq driver suspend happens
>>>>      6. CPU is reparented to PLLP
>>>>      7. DFLL is disabled
>>>>
>>>> 8. Car suspend happens
>>>>      9. DFLL context saved
>>>>      10. PLLP/PLLX context saved
>>>>      11. CCLK context saved
>>>>
>>>> 12. Car resume happens
>>>>      13. DFLL context restored
>>>>      14. PLLP/PLLX context restored
>>>>      15. CCLK context restored
>>>>
>>>> 16. CPUFreq driver resume happens
>>>>      17. DFLL re-inited
>>>>      18. CPU is reparented to DFLL
>>>
>>> Below is the order of sequence it should be based on the order of clk
>>> register.
>>>
>>> My comments inline in this sequence.
>>>
>>> 1. CPUFreq driver probe happens
>>>      2. CPU is reparented to PLLP
>>>      3. DFLL inited
>>>      4. CPU is reparented to DFLL
>>>
>>>
>>> 5. CPUFreq driver suspend happens
>>>      6. Save CPU source which could be either dfll or pllx
> Please see my next comment.
>
>>>      7. CPU is reparented to safe known source PLLP
>>>      8. DFLL is disabled
>>>
>>> 8. Car suspend happens
>>>      9. DFLL context saved (With DFLL disabled in CPUFreq suspend,
>>> nothing to be saved here as last freq req will always be saved).
>>>      10. CCLK context saved (CPU clock source will be saved in CPUFreq
>>> driver suspend which could be either dfll or pllx)
> That I don't understand. The CPU's clock source state should be saved at
> the moment of the CaR's suspending (i.e. CCLK policy will be set to PLLP
> or PLLX) and then CCLK state should be also restored by the CaR in step 14.

CPU clock to be saved and restored should be the source used before we 
switch it to safe PLLP for suspend/resume operation.

This original source could be either PLLX or DFLL which it was using 
before we disable DFLL during CPU Freq suspend.

If we save CPU clock source at moment of CAR suspending, it will be PLLP 
as we switch to safe PLLP in CPUFreq driver suspend.

Infact, we dont need to restore CPU clock source to PLLP anywhere in 
resume as it comes up with PLLP source from warm boot code itself.

But we need to restore CPU source to original source it was using before 
we switch to safe PLLP source for suspend operation. This original 
source could be PLLX/DFLL and this should be re-stored in CPUFreq resume 
as by this time PLLs and peripherals are restored and dfll is 
re-initialized.

So saving actual CPU source before switching to intermediate safe PLLP 
in CPUFreq driver and then restoring back during CPUFreq resume should 
be good as CPUFreq resume happens right after all clocks (plls restore, 
peripherals restore, dfll resume)
> CPUFreq driver should only switch CPU to PLLP and disable DFLL on
> suspend in step 5, that's it. On resume CPUFreq driver will restore CPU
> to DFLL in step 18.
Also I don't think we should hard-code to force CPU source to DFLL on 
CPUFreq resume.

Reason is during CPU Probe when it tries to switch to dfll source, for 
some reason if dfll enable fails it sets CPU to its original source 
which will be PLLX.

So CPU source could be either DFLL or PLLX in CPUFreq 
tegra124_cpu_switch_to_dfll

>>>      11. PLLP/PLLX context saved
>>>      12. Peripheral Clock saved
>>>
>>> 12. Car resume happens
>>>      13. DFLL context restored : No DFLL context to be restored and we
>>> only need to reinitialize DFLL and re-initialize can't be done here as
>>> this is the 1st to get restored and PLL/Peripheral clocks are not
>>> restored by this time. So we can't use clk_ops restore for DFLL
> It looks to me that clk_core_restore_context() should just do
> hlist_for_each_entry *_reverse*. Don't you think so?

Thought of that but this is in core driver and is used by other 
non-tegra clock driver and not sure if that impacts for those.

But with decision of switching CPUFreq with dfll clock enable/disable 
during CPUFreq suspend/resume, we can re-init dfll during dfll-fcpu 
driver resume and we don't need CCLK save/restore.

>>>      14. CCLK context restored
>>> CCLK cant be restored here as context could be either dfll or pllx
>>> which is the source orginally it was actually using before we force
>>> switch to safe PLLP for suspend entry. So we can't use clk_ops restore
>>> for DFLL
> See my comment to step 10. CCLK should be restored to the *CaR's saved*
> context, which is either PLLX or PLLP policy.
>
>>> 15. PLLP/PLLX context restored
>>>      16. Peripheral context restored
>>>
>>> 16. CPUFreq driver resume happens
>>>      17. DFLL re-inited (Invoking DFLL re-init in CPUFreq resume need
>>> exporting DFLL reinit from Clock driver to CPUFreq driver)
> Will be nice if clk_enable(dfll) could be enough to re-init DFLL. That
> should achievable with my next comment to step 18.
>
>>>      18. CPU is reparented to DFLL or PLLX based on saved context from
>>> step 9.
>>>
>>> Note: instead of exporting, we can do DFLL re-init from clock-dfll
>>> driver itself thru dfll-fcpu pm_ops resume. So dfll will be
>>> re-initialized by the time CPUFreq driver resumes and switches to use
>>> DFLL source.
>>>
> Yes, I guess DFLL hardware should be fully reset on DFLL's driver resume
> to be on a safe side any ways.
>
> But(!) we could probably just fix clk_core_restore_context(), like I
> suggested in step 13. Then DFLL clock could use generic save / restore
> context and CPUFreq driver won't have to do anything at all because DFLL
> clock will be saved first and resumed *after* all of the peripherals by
> the CCF. In the end CCLK will be switched to DFLL by the CCF restore as
> well.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 21:51                                                                                     ` Sowjanya Komatineni
@ 2019-07-17 21:57                                                                                       ` Sowjanya Komatineni
  2019-07-17 22:48                                                                                         ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17 21:57 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree


On 7/17/19 2:51 PM, Sowjanya Komatineni wrote:
>
> On 7/17/19 2:30 PM, Dmitry Osipenko wrote:
>> 17.07.2019 23:11, Sowjanya Komatineni пишет:
>>> On 7/17/19 1:01 PM, Sowjanya Komatineni wrote:
>>>> On 7/17/19 12:43 PM, Dmitry Osipenko wrote:
>>>>> 17.07.2019 21:54, Sowjanya Komatineni пишет:
>>>>>> On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>>>>>>> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>>>>>>>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>>>>>>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>>>>>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>>>>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>
>>>>>>>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> See
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with
>>>>>>>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER
>>>>>>>>>>>>>>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> as-is and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DVFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cluster. And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hence
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nicer to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> done)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP, then
>>>>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> selected.
>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X
>>>>>>>>>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe
>>>>>>>>>>>>>>>>>>>>>>>>>>> prior to
>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is 
>>>>>>>>>>>>>>>>>>>>>>>>>> also a
>>>>>>>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running
>>>>>>>>>>>>>>>>>>>>>>>>>> off a
>>>>>>>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on 
>>>>>>>>>>>>>>>>>>>>>>>>>> KEXEC. I
>>>>>>>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not 
>>>>>>>>>>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P
>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used 
>>>>>>>>>>>>>>>>>>>>>>>>> for high
>>>>>>>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from 
>>>>>>>>>>>>>>>>>>>>>>>>> PLLP.
>>>>>>>>>>>>>>>>>>>>>>>> Alright, then please don't forget to 
>>>>>>>>>>>>>>>>>>>>>>>> pre-initialize
>>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is 
>>>>>>>>>>>>>>>>>>>>>>> safe to
>>>>>>>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same
>>>>>>>>>>>>>>>>>>>>>> voltage as
>>>>>>>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me
>>>>>>>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined,
>>>>>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below 
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>>>>>>>> Thank you for the clarification. It would be good 
>>>>>>>>>>>>>>>>>>>> to have
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq
>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>> happens very early even before disabling non-boot 
>>>>>>>>>>>>>>>>>> CPUs and
>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                - Save CPU clock policy registers, and
>>>>>>>>>>>>>>>>>> Perform
>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                - Re-init DFLL, Set in Open-Loop mode,
>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>> Clock policy registers which actually sets source to 
>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>> along
>>>>>>>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                - do clk_prepare_enable which acutally 
>>>>>>>>>>>>>>>>>> sets
>>>>>>>>>>>>>>>>>> DFLL in
>>>>>>>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not
>>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL
>>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>>> disabled anywhere throught the suspend/resume path 
>>>>>>>>>>>>>>>>>> and SC7
>>>>>>>>>>>>>>>>>> entry
>>>>>>>>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend
>>>>>>>>>>>>>>>> it on
>>>>>>>>>>>>>>>> PLLP as well. And besides, seems that currently disabling
>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>> clock will disable DFLL completely and then you'd want to
>>>>>>>>>>>>>>>> re-init
>>>>>>>>>>>>>>>> the DFLL on resume any ways. So better to just disable 
>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>> completely on suspend, which should happen on
>>>>>>>>>>>>>>>> clk_disable(dfll).
>>>>>>>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With 
>>>>>>>>>>>>>>> decision of
>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>> clk_disable during suspend, its mandatory to switch to 
>>>>>>>>>>>>>>> PLLP as
>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>> is completely disabled.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My earlier concern was on restoring CPU policy as we 
>>>>>>>>>>>>>>> can't do
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Clear now and will do CPU clock policy restore in after 
>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>> re-init.
>>>>>>>>>>>>>> Why the policy can't be saved/restored by the CaR driver 
>>>>>>>>>>>>>> as a
>>>>>>>>>>>>>> context of any other clock?
>>>>>>>>>>>>> restoring cpu clock policy involves programming source and
>>>>>>>>>>>>> super_cclkg_divider.
>>>>>>>>>>>>>
>>>>>>>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use
>>>>>>>>>>>>> frac_div ops
>>>>>>>>>>>>> to do save/restore its divider.
>>>>>>>>>>>> That can be changed of course and I guess it also could be as
>>>>>>>>>>>> simple as
>>>>>>>>>>>> saving and restoring of two raw u32 values of the 
>>>>>>>>>>>> policy/divider
>>>>>>>>>>>> registers.
>>>>>>>>>>>>
>>>>>>>>>>>>> Also, during clock context we cant restore cclk_g as cclk_g
>>>>>>>>>>>>> source
>>>>>>>>>>>>> will be dfll and dfll will not be resumed/re-initialized 
>>>>>>>>>>>>> by the
>>>>>>>>>>>>> time
>>>>>>>>>>>>> clk_super_mux save/restore happens.
>>>>>>>>>>>>>
>>>>>>>>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>>>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and
>>>>>>>>>>>>> dfll_ref
>>>>>>>>>>>>> and dfll_soc peripheral clocks are not restored by the 
>>>>>>>>>>>>> time dfll
>>>>>>>>>>>>> restore happens. Also dfll peripheral clock enables need 
>>>>>>>>>>>>> to be
>>>>>>>>>>>>> restored before dfll restore happens which involves 
>>>>>>>>>>>>> programming
>>>>>>>>>>>>> dfll
>>>>>>>>>>>>> controller for re-initialization.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of all
>>>>>>>>>>>>> clocks
>>>>>>>>>>>>> restore in V5 series but instead of in clk-tegra210 driver I
>>>>>>>>>>>>> moved
>>>>>>>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies 
>>>>>>>>>>>>> will be
>>>>>>>>>>>>> restored thru clk_restore_context by then. This will be in 
>>>>>>>>>>>>> V6.
>>>>>>>>>>>> Since DFLL is now guaranteed to be disabled across CaR
>>>>>>>>>>>> suspend/resume
>>>>>>>>>>>> (hence it has nothing to do in regards to CCLK) and given that
>>>>>>>>>>>> PLLs
>>>>>>>>>>>> state is restored before the rest of the clocks, I don't 
>>>>>>>>>>>> see why
>>>>>>>>>>>> not to
>>>>>>>>>>>> implement CCLK save/restore in a generic fasion. CPU policy
>>>>>>>>>>>> wull be
>>>>>>>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is 
>>>>>>>>>>>> disabled).
>>>>>>>>>>>>
>>>>>>>>>>> CCLK_G save/restore should happen in clk_super_mux ops
>>>>>>>>>>> save/context and
>>>>>>>>>>> clk_super_mux save/restore happens very early as cclk_g is 
>>>>>>>>>>> first
>>>>>>>>>>> in the
>>>>>>>>>>> clock tree and save/restore traverses through the tree 
>>>>>>>>>>> top-bottom
>>>>>>>>>>> order.
>>>>>>>>>> If CCLK_G is restored before the PLLs, then just change the 
>>>>>>>>>> clocks
>>>>>>>>>> order
>>>>>>>>>> such that it won't happen.
>>>>>>>>>>
>>>>>>>>> I dont think we can change clocks order for CCLK_G.
>>>>>>>>>
>>>>>>>>> During bootup, cclk_g is registered after all pll's and 
>>>>>>>>> peripheral
>>>>>>>>> clocks which is the way we wanted, So cclk_g will be the first
>>>>>>>>> one in
>>>>>>>>> the clk list as clk_register adds new clock first in the list.
>>>>>>>>>
>>>>>>>>> When clk_save_context and clk_restore_context APIs iterates 
>>>>>>>>> over the
>>>>>>>>> list, cclk_g is the first
>>>>>>>> Looking at clk_core_restore_context(), I see that it walks up CLKs
>>>>>>>> list
>>>>>>>> from parent to children, hence I don't understand how it can ever
>>>>>>>> happen
>>>>>>>> that CCLK will be restored before the parent. The clocks 
>>>>>>>> registration
>>>>>>>> order doesn't matter at all in that case.
>>>>>>> yes from parent to children and dfllCPU_out is the top in the 
>>>>>>> list and
>>>>>>> its child is cclk_g.
>>>>>>>
>>>>>>> the way clocks are registered is the order I see in the clock 
>>>>>>> list and
>>>>>>> looking into clk_register API it adds new node first in the list.
>>>>>>>
>>>>>> cclkg_g & dfll register happens after all plls and peripheral 
>>>>>> clocks as
>>>>>> it need ref, soc and peripheral clocks to be enabled.
>>>>>>> So they are the last to get registered and so becomes first in the
>>>>>>> list.
>>>>>>>
>>>>>>> During save/restore context, it traverses thru this list and 
>>>>>>> first in
>>>>>>> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>>>>>>>
>>>>>>> saving should not be an issue at all but we cant restore 
>>>>>>> cclk_g/dfll
>>>>>>> in normal way thru clk_ops restore as plls and peripherals restore
>>>>>>> doesn't happen by that time.
>>>>>>>
>>>>>> I was referring to clk_restore_context where it iterates thru 
>>>>>> root list
>>>>>> and for each core from the root list clk_core_restore does 
>>>>>> restore of
>>>>>> parent and children.
>>>>>>
>>>>>> dfllCPU_Out gets first in the list and its child is cclk_g
>>>>>>
>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105 
>>>>>>
>>>>> What list you're talking about? clk_summary? It shows current 
>>>>> *active*
>>>>> clocks configuration, if you'll try to disable CPUFreq driver then 
>>>>> the
>>>>> parent of CCLK_G should be PLLX. Similarly when CPU is reparented to
>>>>> PLLP on driver's suspend, then PLLP is the parent.
>>>>>
>>>>>>>>>>> DFLL enable thru CPUFreq resume happens after all
>>>>>>>>>>> clk_restore_context
>>>>>>>>>>> happens. So during clk_restore_context, dfll re-init doesnt 
>>>>>>>>>>> happen
>>>>>>>>>>> and
>>>>>>>>>>> doing cpu clock policy restore during super_mux clk_ops will
>>>>>>>>>>> crash as
>>>>>>>>>>> DFLL is not initialized and its clock is not enabled but CPU 
>>>>>>>>>>> clock
>>>>>>>>>>> restore sets source to DFLL if we restore during super_clk_mux
>>>>>>>>>> If CPU was suspended on PLLP, then it will be restored on 
>>>>>>>>>> PLLP by
>>>>>>>>>> CaR. I
>>>>>>>>>> don't understand what DFLL has to do with the CCLK in that case
>>>>>>>>>> during
>>>>>>>>>> the clocks restore.
>>>>>>>>> My above comment is in reference to your request of doing
>>>>>>>>> save/restore
>>>>>>>>> for cclk_g in normal fashion thru save/restore context. Because
>>>>>>>>> of the
>>>>>>>>> clk order I mentioned above, we cclk_g will be the first one to
>>>>>>>>> go thru
>>>>>>>>> save/context.
>>>>>>>>>
>>>>>>>>> During save_context of cclk_g, source can be from PLLX, dfll.
>>>>>>>>>
>>>>>>>>> Issue will be when we do restore during clk_restore_context of
>>>>>>>>> cclk_g as
>>>>>>>>> by that time PLLX/dfll will not be restored.
>>>>>>>>>
>>>>>>>> Seems we already agreed that DFLL will be disabled by the CPUFreq
>>>>>>>> driver
>>>>>>>> on suspend. Hence CCLK can't be from DFLL if CPU is reparented to
>>>>>>>> PLLP
>>>>>>>> on CPUFreq driver's suspend, otherwise CPU keeps running from a
>>>>>>>> boot-state PLLX if CPUFreq driver is disabled.
>>>>>>> Yes suspend should not be an issue but issue will be during resume
>>>>>>> where if we do cclk_g restore in normal way thru 
>>>>>>> clk_restore_context,
>>>>>>> cclk_g restore happens very early as dfllCPU out is the first 
>>>>>>> one that
>>>>>>> goes thru restore context and plls/peripherals are not resumed by
>>>>>>> then.
>>>>>>>
>>>>>>> CPU runs from PLLX if dfll clock enable fails during boot. So 
>>>>>>> when it
>>>>>>> gets to suspend, we save CPU running clock source as either PLLX or
>>>>>>> DFLL and then we switch to PLLP.
>>>>>>>
>>>>>>>
>>>>>>> On resume, CPU runs from PLLP by warm boot code and we need to 
>>>>>>> restore
>>>>>>> back its source to the one it was using from saved source context
>>>>>>> (which can be either PLLX or DFLL)
>>>>>>>
>>>>>>> So PLLs & DFLL resume need to happen before CCLKG restore/resume.
>>>>>>>
>>>>>>>
>>>>>>> With all above discussions, we do DFLL disable in CPUFreq driver on
>>>>>>> suspend and on CPUFreq resume we enable DFLL back and restore CPU
>>>>>>> clock source it was using during suspend (which will be either 
>>>>>>> PLLX if
>>>>>>> dfll enable fails during probe or it will be using DFLL).
>>>>> During suspend CPU's parent shall be PLLP and not DFLL (note that 
>>>>> it is
>>>>> disabled) after reparenting to PLLP by the CPUFreq driver.
>>>>>
>>>> CPU source context should be saved before switching to safe source of
>>>> PLLP as on resume we need to restore back to source it was using
>>>> before we switch to safe source during suspend entry.
>>>>
>>>> So saved context for CPU Source will be either dfll or PLLX
>>>>
>>> PLLP reparenting is only during suspend/entry to have it as safe source
>>> but actual CPU source it was running from before suspending is either
>>> dfll/pllx which should be the one to be restored on CPUFreq resume.
>>> Resume happens with CPU running from PLLP till it gets to the point of
>>> restoring its original source (dfll or pllx)
>> CaR should restore CPU to PLLP or PLLX, while CPUFreq driver restores
>> CPU to DFLL. Please see more comments below.
>>
>>>>>>> So i was trying to say dfll/cclk_g restore can't be done in 
>>>>>>> normal way
>>>>>>> thru clk_ops save/restore context
>>>>> Let's see what happens if CPUFreq is active:
>>>>>
>>>>> 1. CPUFreq driver probe happens
>>>>>      2. CPU is reparented to PLLP
>>>>>      3. DFLL inited
>>>>>      4. CPU is reparented to DFLL
>>>>>
>>>>> 5. CPUFreq driver suspend happens
>>>>>      6. CPU is reparented to PLLP
>>>>>      7. DFLL is disabled
>>>>>
>>>>> 8. Car suspend happens
>>>>>      9. DFLL context saved
>>>>>      10. PLLP/PLLX context saved
>>>>>      11. CCLK context saved
>>>>>
>>>>> 12. Car resume happens
>>>>>      13. DFLL context restored
>>>>>      14. PLLP/PLLX context restored
>>>>>      15. CCLK context restored
>>>>>
>>>>> 16. CPUFreq driver resume happens
>>>>>      17. DFLL re-inited
>>>>>      18. CPU is reparented to DFLL
>>>>
>>>> Below is the order of sequence it should be based on the order of clk
>>>> register.
>>>>
>>>> My comments inline in this sequence.
>>>>
>>>> 1. CPUFreq driver probe happens
>>>>      2. CPU is reparented to PLLP
>>>>      3. DFLL inited
>>>>      4. CPU is reparented to DFLL
>>>>
>>>>
>>>> 5. CPUFreq driver suspend happens
>>>>      6. Save CPU source which could be either dfll or pllx
>> Please see my next comment.
>>
>>>>      7. CPU is reparented to safe known source PLLP
>>>>      8. DFLL is disabled
>>>>
>>>> 8. Car suspend happens
>>>>      9. DFLL context saved (With DFLL disabled in CPUFreq suspend,
>>>> nothing to be saved here as last freq req will always be saved).
>>>>      10. CCLK context saved (CPU clock source will be saved in CPUFreq
>>>> driver suspend which could be either dfll or pllx)
>> That I don't understand. The CPU's clock source state should be saved at
>> the moment of the CaR's suspending (i.e. CCLK policy will be set to PLLP
>> or PLLX) and then CCLK state should be also restored by the CaR in 
>> step 14.
>
> CPU clock to be saved and restored should be the source used before we 
> switch it to safe PLLP for suspend/resume operation.
>
> This original source could be either PLLX or DFLL which it was using 
> before we disable DFLL during CPU Freq suspend.
>
> If we save CPU clock source at moment of CAR suspending, it will be 
> PLLP as we switch to safe PLLP in CPUFreq driver suspend.
>
> Infact, we dont need to restore CPU clock source to PLLP anywhere in 
> resume as it comes up with PLLP source from warm boot code itself.
>
> But we need to restore CPU source to original source it was using 
> before we switch to safe PLLP source for suspend operation. This 
> original source could be PLLX/DFLL and this should be re-stored in 
> CPUFreq resume as by this time PLLs and peripherals are restored and 
> dfll is re-initialized.
>
> So saving actual CPU source before switching to intermediate safe PLLP 
> in CPUFreq driver and then restoring back during CPUFreq resume should 
> be good as CPUFreq resume happens right after all clocks (plls 
> restore, peripherals restore, dfll resume)
>> CPUFreq driver should only switch CPU to PLLP and disable DFLL on
>> suspend in step 5, that's it. On resume CPUFreq driver will restore CPU
>> to DFLL in step 18.
> Also I don't think we should hard-code to force CPU source to DFLL on 
> CPUFreq resume.
>
> Reason is during CPU Probe when it tries to switch to dfll source, for 
> some reason if dfll enable fails it sets CPU to its original source 
> which will be PLLX.
>
> So CPU source could be either DFLL or PLLX in CPUFreq 
> tegra124_cpu_switch_to_dfll
>
>>>>      11. PLLP/PLLX context saved
>>>>      12. Peripheral Clock saved
>>>>
>>>> 12. Car resume happens
>>>>      13. DFLL context restored : No DFLL context to be restored and we
>>>> only need to reinitialize DFLL and re-initialize can't be done here as
>>>> this is the 1st to get restored and PLL/Peripheral clocks are not
>>>> restored by this time. So we can't use clk_ops restore for DFLL
>> It looks to me that clk_core_restore_context() should just do
>> hlist_for_each_entry *_reverse*. Don't you think so?
>
> Thought of that but this is in core driver and is used by other 
> non-tegra clock driver and not sure if that impacts for those.
>
> But with decision of switching CPUFreq with dfll clock enable/disable 
> during CPUFreq suspend/resume, we can re-init dfll during dfll-fcpu 
> driver resume and we don't need CCLK save/restore.
>
the way of all clocks order is good except cclk_g which has dependency 
on multiple clocks.

reverse list order during restore might not work as all other clocks are 
in proper order no with any ref clocks for plls getting restored prior 
to their clients

>>>>      14. CCLK context restored
>>>> CCLK cant be restored here as context could be either dfll or pllx
>>>> which is the source orginally it was actually using before we force
>>>> switch to safe PLLP for suspend entry. So we can't use clk_ops restore
>>>> for DFLL
>> See my comment to step 10. CCLK should be restored to the *CaR's saved*
>> context, which is either PLLX or PLLP policy.
>>
>>>> 15. PLLP/PLLX context restored
>>>>      16. Peripheral context restored
>>>>
>>>> 16. CPUFreq driver resume happens
>>>>      17. DFLL re-inited (Invoking DFLL re-init in CPUFreq resume need
>>>> exporting DFLL reinit from Clock driver to CPUFreq driver)
>> Will be nice if clk_enable(dfll) could be enough to re-init DFLL. That
>> should achievable with my next comment to step 18.
>>
>>>>      18. CPU is reparented to DFLL or PLLX based on saved context from
>>>> step 9.
>>>>
>>>> Note: instead of exporting, we can do DFLL re-init from clock-dfll
>>>> driver itself thru dfll-fcpu pm_ops resume. So dfll will be
>>>> re-initialized by the time CPUFreq driver resumes and switches to use
>>>> DFLL source.
>>>>
>> Yes, I guess DFLL hardware should be fully reset on DFLL's driver resume
>> to be on a safe side any ways.
>>
>> But(!) we could probably just fix clk_core_restore_context(), like I
>> suggested in step 13. Then DFLL clock could use generic save / restore
>> context and CPUFreq driver won't have to do anything at all because DFLL
>> clock will be saved first and resumed *after* all of the peripherals by
>> the CCF. In the end CCLK will be switched to DFLL by the CCF restore as
>> well.
>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 21:57                                                                                       ` Sowjanya Komatineni
@ 2019-07-17 22:48                                                                                         ` Dmitry Osipenko
  2019-07-17 23:36                                                                                           ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-17 22:48 UTC (permalink / raw)
  To: Sowjanya Komatineni, sboyd, Michael Turquette
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree

18.07.2019 0:57, Sowjanya Komatineni пишет:
> 
> On 7/17/19 2:51 PM, Sowjanya Komatineni wrote:
>>
>> On 7/17/19 2:30 PM, Dmitry Osipenko wrote:
>>> 17.07.2019 23:11, Sowjanya Komatineni пишет:
>>>> On 7/17/19 1:01 PM, Sowjanya Komatineni wrote:
>>>>> On 7/17/19 12:43 PM, Dmitry Osipenko wrote:
>>>>>> 17.07.2019 21:54, Sowjanya Komatineni пишет:
>>>>>>> On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>>>>>>>> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>>>>>>>>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>>>>>>>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>>>>>>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>>>>>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> See
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as-is and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DVFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cluster. And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hence
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nicer to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> done)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP, then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> selected.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X
>>>>>>>>>>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe
>>>>>>>>>>>>>>>>>>>>>>>>>>>> prior to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is
>>>>>>>>>>>>>>>>>>>>>>>>>>> also a
>>>>>>>>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running
>>>>>>>>>>>>>>>>>>>>>>>>>>> off a
>>>>>>>>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on
>>>>>>>>>>>>>>>>>>>>>>>>>>> KEXEC. I
>>>>>>>>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not
>>>>>>>>>>>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P
>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used
>>>>>>>>>>>>>>>>>>>>>>>>>> for high
>>>>>>>>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from
>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP.
>>>>>>>>>>>>>>>>>>>>>>>>> Alright, then please don't forget to
>>>>>>>>>>>>>>>>>>>>>>>>> pre-initialize
>>>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is
>>>>>>>>>>>>>>>>>>>>>>>> safe to
>>>>>>>>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same
>>>>>>>>>>>>>>>>>>>>>>> voltage as
>>>>>>>>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me
>>>>>>>>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined,
>>>>>>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>>>>>>>>> Thank you for the clarification. It would be good
>>>>>>>>>>>>>>>>>>>>> to have
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq
>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>> happens very early even before disabling non-boot
>>>>>>>>>>>>>>>>>>> CPUs and
>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                - Save CPU clock policy registers, and
>>>>>>>>>>>>>>>>>>> Perform
>>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                - Re-init DFLL, Set in Open-Loop mode,
>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>> Clock policy registers which actually sets source to
>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>> along
>>>>>>>>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                - do clk_prepare_enable which acutally
>>>>>>>>>>>>>>>>>>> sets
>>>>>>>>>>>>>>>>>>> DFLL in
>>>>>>>>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not
>>>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL
>>>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>>>> disabled anywhere throught the suspend/resume path
>>>>>>>>>>>>>>>>>>> and SC7
>>>>>>>>>>>>>>>>>>> entry
>>>>>>>>>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend
>>>>>>>>>>>>>>>>> it on
>>>>>>>>>>>>>>>>> PLLP as well. And besides, seems that currently disabling
>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>> clock will disable DFLL completely and then you'd want to
>>>>>>>>>>>>>>>>> re-init
>>>>>>>>>>>>>>>>> the DFLL on resume any ways. So better to just disable
>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>> completely on suspend, which should happen on
>>>>>>>>>>>>>>>>> clk_disable(dfll).
>>>>>>>>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With
>>>>>>>>>>>>>>>> decision of
>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>> clk_disable during suspend, its mandatory to switch to
>>>>>>>>>>>>>>>> PLLP as
>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>> is completely disabled.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My earlier concern was on restoring CPU policy as we
>>>>>>>>>>>>>>>> can't do
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Clear now and will do CPU clock policy restore in after
>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>> re-init.
>>>>>>>>>>>>>>> Why the policy can't be saved/restored by the CaR driver
>>>>>>>>>>>>>>> as a
>>>>>>>>>>>>>>> context of any other clock?
>>>>>>>>>>>>>> restoring cpu clock policy involves programming source and
>>>>>>>>>>>>>> super_cclkg_divider.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use
>>>>>>>>>>>>>> frac_div ops
>>>>>>>>>>>>>> to do save/restore its divider.
>>>>>>>>>>>>> That can be changed of course and I guess it also could be as
>>>>>>>>>>>>> simple as
>>>>>>>>>>>>> saving and restoring of two raw u32 values of the
>>>>>>>>>>>>> policy/divider
>>>>>>>>>>>>> registers.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also, during clock context we cant restore cclk_g as cclk_g
>>>>>>>>>>>>>> source
>>>>>>>>>>>>>> will be dfll and dfll will not be resumed/re-initialized
>>>>>>>>>>>>>> by the
>>>>>>>>>>>>>> time
>>>>>>>>>>>>>> clk_super_mux save/restore happens.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>>>>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and
>>>>>>>>>>>>>> dfll_ref
>>>>>>>>>>>>>> and dfll_soc peripheral clocks are not restored by the
>>>>>>>>>>>>>> time dfll
>>>>>>>>>>>>>> restore happens. Also dfll peripheral clock enables need
>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>> restored before dfll restore happens which involves
>>>>>>>>>>>>>> programming
>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>> controller for re-initialization.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of all
>>>>>>>>>>>>>> clocks
>>>>>>>>>>>>>> restore in V5 series but instead of in clk-tegra210 driver I
>>>>>>>>>>>>>> moved
>>>>>>>>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies
>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>> restored thru clk_restore_context by then. This will be in
>>>>>>>>>>>>>> V6.
>>>>>>>>>>>>> Since DFLL is now guaranteed to be disabled across CaR
>>>>>>>>>>>>> suspend/resume
>>>>>>>>>>>>> (hence it has nothing to do in regards to CCLK) and given that
>>>>>>>>>>>>> PLLs
>>>>>>>>>>>>> state is restored before the rest of the clocks, I don't
>>>>>>>>>>>>> see why
>>>>>>>>>>>>> not to
>>>>>>>>>>>>> implement CCLK save/restore in a generic fasion. CPU policy
>>>>>>>>>>>>> wull be
>>>>>>>>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is
>>>>>>>>>>>>> disabled).
>>>>>>>>>>>>>
>>>>>>>>>>>> CCLK_G save/restore should happen in clk_super_mux ops
>>>>>>>>>>>> save/context and
>>>>>>>>>>>> clk_super_mux save/restore happens very early as cclk_g is
>>>>>>>>>>>> first
>>>>>>>>>>>> in the
>>>>>>>>>>>> clock tree and save/restore traverses through the tree
>>>>>>>>>>>> top-bottom
>>>>>>>>>>>> order.
>>>>>>>>>>> If CCLK_G is restored before the PLLs, then just change the
>>>>>>>>>>> clocks
>>>>>>>>>>> order
>>>>>>>>>>> such that it won't happen.
>>>>>>>>>>>
>>>>>>>>>> I dont think we can change clocks order for CCLK_G.
>>>>>>>>>>
>>>>>>>>>> During bootup, cclk_g is registered after all pll's and
>>>>>>>>>> peripheral
>>>>>>>>>> clocks which is the way we wanted, So cclk_g will be the first
>>>>>>>>>> one in
>>>>>>>>>> the clk list as clk_register adds new clock first in the list.
>>>>>>>>>>
>>>>>>>>>> When clk_save_context and clk_restore_context APIs iterates
>>>>>>>>>> over the
>>>>>>>>>> list, cclk_g is the first
>>>>>>>>> Looking at clk_core_restore_context(), I see that it walks up CLKs
>>>>>>>>> list
>>>>>>>>> from parent to children, hence I don't understand how it can ever
>>>>>>>>> happen
>>>>>>>>> that CCLK will be restored before the parent. The clocks
>>>>>>>>> registration
>>>>>>>>> order doesn't matter at all in that case.
>>>>>>>> yes from parent to children and dfllCPU_out is the top in the
>>>>>>>> list and
>>>>>>>> its child is cclk_g.
>>>>>>>>
>>>>>>>> the way clocks are registered is the order I see in the clock
>>>>>>>> list and
>>>>>>>> looking into clk_register API it adds new node first in the list.
>>>>>>>>
>>>>>>> cclkg_g & dfll register happens after all plls and peripheral
>>>>>>> clocks as
>>>>>>> it need ref, soc and peripheral clocks to be enabled.
>>>>>>>> So they are the last to get registered and so becomes first in the
>>>>>>>> list.
>>>>>>>>
>>>>>>>> During save/restore context, it traverses thru this list and
>>>>>>>> first in
>>>>>>>> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>>>>>>>>
>>>>>>>> saving should not be an issue at all but we cant restore
>>>>>>>> cclk_g/dfll
>>>>>>>> in normal way thru clk_ops restore as plls and peripherals restore
>>>>>>>> doesn't happen by that time.
>>>>>>>>
>>>>>>> I was referring to clk_restore_context where it iterates thru
>>>>>>> root list
>>>>>>> and for each core from the root list clk_core_restore does
>>>>>>> restore of
>>>>>>> parent and children.
>>>>>>>
>>>>>>> dfllCPU_Out gets first in the list and its child is cclk_g
>>>>>>>
>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105
>>>>>>>
>>>>>> What list you're talking about? clk_summary? It shows current
>>>>>> *active*
>>>>>> clocks configuration, if you'll try to disable CPUFreq driver then
>>>>>> the
>>>>>> parent of CCLK_G should be PLLX. Similarly when CPU is reparented to
>>>>>> PLLP on driver's suspend, then PLLP is the parent.
>>>>>>
>>>>>>>>>>>> DFLL enable thru CPUFreq resume happens after all
>>>>>>>>>>>> clk_restore_context
>>>>>>>>>>>> happens. So during clk_restore_context, dfll re-init doesnt
>>>>>>>>>>>> happen
>>>>>>>>>>>> and
>>>>>>>>>>>> doing cpu clock policy restore during super_mux clk_ops will
>>>>>>>>>>>> crash as
>>>>>>>>>>>> DFLL is not initialized and its clock is not enabled but CPU
>>>>>>>>>>>> clock
>>>>>>>>>>>> restore sets source to DFLL if we restore during super_clk_mux
>>>>>>>>>>> If CPU was suspended on PLLP, then it will be restored on
>>>>>>>>>>> PLLP by
>>>>>>>>>>> CaR. I
>>>>>>>>>>> don't understand what DFLL has to do with the CCLK in that case
>>>>>>>>>>> during
>>>>>>>>>>> the clocks restore.
>>>>>>>>>> My above comment is in reference to your request of doing
>>>>>>>>>> save/restore
>>>>>>>>>> for cclk_g in normal fashion thru save/restore context. Because
>>>>>>>>>> of the
>>>>>>>>>> clk order I mentioned above, we cclk_g will be the first one to
>>>>>>>>>> go thru
>>>>>>>>>> save/context.
>>>>>>>>>>
>>>>>>>>>> During save_context of cclk_g, source can be from PLLX, dfll.
>>>>>>>>>>
>>>>>>>>>> Issue will be when we do restore during clk_restore_context of
>>>>>>>>>> cclk_g as
>>>>>>>>>> by that time PLLX/dfll will not be restored.
>>>>>>>>>>
>>>>>>>>> Seems we already agreed that DFLL will be disabled by the CPUFreq
>>>>>>>>> driver
>>>>>>>>> on suspend. Hence CCLK can't be from DFLL if CPU is reparented to
>>>>>>>>> PLLP
>>>>>>>>> on CPUFreq driver's suspend, otherwise CPU keeps running from a
>>>>>>>>> boot-state PLLX if CPUFreq driver is disabled.
>>>>>>>> Yes suspend should not be an issue but issue will be during resume
>>>>>>>> where if we do cclk_g restore in normal way thru
>>>>>>>> clk_restore_context,
>>>>>>>> cclk_g restore happens very early as dfllCPU out is the first
>>>>>>>> one that
>>>>>>>> goes thru restore context and plls/peripherals are not resumed by
>>>>>>>> then.
>>>>>>>>
>>>>>>>> CPU runs from PLLX if dfll clock enable fails during boot. So
>>>>>>>> when it
>>>>>>>> gets to suspend, we save CPU running clock source as either PLLX or
>>>>>>>> DFLL and then we switch to PLLP.
>>>>>>>>
>>>>>>>>
>>>>>>>> On resume, CPU runs from PLLP by warm boot code and we need to
>>>>>>>> restore
>>>>>>>> back its source to the one it was using from saved source context
>>>>>>>> (which can be either PLLX or DFLL)
>>>>>>>>
>>>>>>>> So PLLs & DFLL resume need to happen before CCLKG restore/resume.
>>>>>>>>
>>>>>>>>
>>>>>>>> With all above discussions, we do DFLL disable in CPUFreq driver on
>>>>>>>> suspend and on CPUFreq resume we enable DFLL back and restore CPU
>>>>>>>> clock source it was using during suspend (which will be either
>>>>>>>> PLLX if
>>>>>>>> dfll enable fails during probe or it will be using DFLL).
>>>>>> During suspend CPU's parent shall be PLLP and not DFLL (note that
>>>>>> it is
>>>>>> disabled) after reparenting to PLLP by the CPUFreq driver.
>>>>>>
>>>>> CPU source context should be saved before switching to safe source of
>>>>> PLLP as on resume we need to restore back to source it was using
>>>>> before we switch to safe source during suspend entry.
>>>>>
>>>>> So saved context for CPU Source will be either dfll or PLLX
>>>>>
>>>> PLLP reparenting is only during suspend/entry to have it as safe source
>>>> but actual CPU source it was running from before suspending is either
>>>> dfll/pllx which should be the one to be restored on CPUFreq resume.
>>>> Resume happens with CPU running from PLLP till it gets to the point of
>>>> restoring its original source (dfll or pllx)
>>> CaR should restore CPU to PLLP or PLLX, while CPUFreq driver restores
>>> CPU to DFLL. Please see more comments below.
>>>
>>>>>>>> So i was trying to say dfll/cclk_g restore can't be done in
>>>>>>>> normal way
>>>>>>>> thru clk_ops save/restore context
>>>>>> Let's see what happens if CPUFreq is active:
>>>>>>
>>>>>> 1. CPUFreq driver probe happens
>>>>>>      2. CPU is reparented to PLLP
>>>>>>      3. DFLL inited
>>>>>>      4. CPU is reparented to DFLL
>>>>>>
>>>>>> 5. CPUFreq driver suspend happens
>>>>>>      6. CPU is reparented to PLLP
>>>>>>      7. DFLL is disabled
>>>>>>
>>>>>> 8. Car suspend happens
>>>>>>      9. DFLL context saved
>>>>>>      10. PLLP/PLLX context saved
>>>>>>      11. CCLK context saved
>>>>>>
>>>>>> 12. Car resume happens
>>>>>>      13. DFLL context restored
>>>>>>      14. PLLP/PLLX context restored
>>>>>>      15. CCLK context restored
>>>>>>
>>>>>> 16. CPUFreq driver resume happens
>>>>>>      17. DFLL re-inited
>>>>>>      18. CPU is reparented to DFLL
>>>>>
>>>>> Below is the order of sequence it should be based on the order of clk
>>>>> register.
>>>>>
>>>>> My comments inline in this sequence.
>>>>>
>>>>> 1. CPUFreq driver probe happens
>>>>>      2. CPU is reparented to PLLP
>>>>>      3. DFLL inited
>>>>>      4. CPU is reparented to DFLL
>>>>>
>>>>>
>>>>> 5. CPUFreq driver suspend happens
>>>>>      6. Save CPU source which could be either dfll or pllx
>>> Please see my next comment.
>>>
>>>>>      7. CPU is reparented to safe known source PLLP
>>>>>      8. DFLL is disabled
>>>>>
>>>>> 8. Car suspend happens
>>>>>      9. DFLL context saved (With DFLL disabled in CPUFreq suspend,
>>>>> nothing to be saved here as last freq req will always be saved).
>>>>>      10. CCLK context saved (CPU clock source will be saved in CPUFreq
>>>>> driver suspend which could be either dfll or pllx)
>>> That I don't understand. The CPU's clock source state should be saved at
>>> the moment of the CaR's suspending (i.e. CCLK policy will be set to PLLP
>>> or PLLX) and then CCLK state should be also restored by the CaR in
>>> step 14.
>>
>> CPU clock to be saved and restored should be the source used before we
>> switch it to safe PLLP for suspend/resume operation.
>>
>> This original source could be either PLLX or DFLL which it was using
>> before we disable DFLL during CPU Freq suspend.
>>
>> If we save CPU clock source at moment of CAR suspending, it will be
>> PLLP as we switch to safe PLLP in CPUFreq driver suspend.
>>
>> Infact, we dont need to restore CPU clock source to PLLP anywhere in
>> resume as it comes up with PLLP source from warm boot code itself.

You must always maintain proper suspend/resume encapsulation, otherwise
it's a total mess. It doesn't matter that CCLK is restored to PLLP even
that CPU is already running off PLLP after warmboot.

>> But we need to restore CPU source to original source it was using
>> before we switch to safe PLLP source for suspend operation. This
>> original source could be PLLX/DFLL and this should be re-stored in
>> CPUFreq resume as by this time PLLs and peripherals are restored and
>> dfll is re-initialized.
>>
>> So saving actual CPU source before switching to intermediate safe PLLP
>> in CPUFreq driver and then restoring back during CPUFreq resume should
>> be good as CPUFreq resume happens right after all clocks (plls
>> restore, peripherals restore, dfll resume)>>
>>> CPUFreq driver should only switch CPU to PLLP and disable DFLL on
>>> suspend in step 5, that's it. On resume CPUFreq driver will restore CPU
>>> to DFLL in step 18.
>> Also I don't think we should hard-code to force CPU source to DFLL on
>> CPUFreq resume.
>>
>> Reason is during CPU Probe when it tries to switch to dfll source, for
>> some reason if dfll enable fails it sets CPU to its original source
>> which will be PLLX.

No!

1. CPU voltage could be too low for PLLX
2. PLLX rate can't be changed without manual reparenting CPU to
intermediate clock
3. CPUFreq can't manually manage CPU voltage

DFLL-restoring failure is an extreme case. CPU must be kept on a safe
PLLP in that case and disable_cpufreq() must be invoked as well.

>> So CPU source could be either DFLL or PLLX in CPUFreq
>> tegra124_cpu_switch_to_dfll
>>
>>>>>      11. PLLP/PLLX context saved
>>>>>      12. Peripheral Clock saved
>>>>>
>>>>> 12. Car resume happens
>>>>>      13. DFLL context restored : No DFLL context to be restored and we
>>>>> only need to reinitialize DFLL and re-initialize can't be done here as
>>>>> this is the 1st to get restored and PLL/Peripheral clocks are not
>>>>> restored by this time. So we can't use clk_ops restore for DFLL
>>> It looks to me that clk_core_restore_context() should just do
>>> hlist_for_each_entry *_reverse*. Don't you think so?
>>
>> Thought of that but this is in core driver and is used by other
>> non-tegra clock driver and not sure if that impacts for those.

The reverse ordering should be correct, I think it's just a shortcoming
of the CCF that need to be fixed. But will better to make a more
thorough research, asking Stephen and Michael for the clarification.

>> But with decision of switching CPUFreq with dfll clock enable/disable
>> during CPUFreq suspend/resume, we can re-init dfll during dfll-fcpu
>> driver resume and we don't need CCLK save/restore.
>>
> the way of all clocks order is good except cclk_g which has dependency
> on multiple clocks.

CCLK_G has a single parent at a time. What "multiple clocks" you're
talking about? Please explain.

> reverse list order during restore might not work as all other clocks are
> in proper order no with any ref clocks for plls getting restored prior
> to their clients

Why? The ref clocks should be registered first and be the roots for PLLs
and the rest. If it's not currently the case, then this need to be
fixed. You need to ensure that each clock is modeled properly. If some
child clock really depends on multiple parents, then the parents need to
in the correct order or CCF need to be taught about such multi-dependencies.

If some required feature is missed, then you have to implement it
properly and for all, that's how things are done in upstream. Sometimes
it's quite a lot of extra work that everyone are benefiting from in the end.

[snip]

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 22:48                                                                                         ` Dmitry Osipenko
@ 2019-07-17 23:36                                                                                           ` Sowjanya Komatineni
  2019-07-17 23:44                                                                                             ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-17 23:36 UTC (permalink / raw)
  To: Dmitry Osipenko, sboyd, Michael Turquette
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree


On 7/17/19 3:48 PM, Dmitry Osipenko wrote:
> 18.07.2019 0:57, Sowjanya Komatineni пишет:
>> On 7/17/19 2:51 PM, Sowjanya Komatineni wrote:
>>> On 7/17/19 2:30 PM, Dmitry Osipenko wrote:
>>>> 17.07.2019 23:11, Sowjanya Komatineni пишет:
>>>>> On 7/17/19 1:01 PM, Sowjanya Komatineni wrote:
>>>>>> On 7/17/19 12:43 PM, Dmitry Osipenko wrote:
>>>>>>> 17.07.2019 21:54, Sowjanya Komatineni пишет:
>>>>>>>> On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>>>>>>>>> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>>>>>>>>>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>>>>>>>>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>>>>>>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till dfll
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device links".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> See
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning EPROBE_DEFER
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as-is and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DVFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not guaranteed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will have to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment. During
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in closed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cluster. And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init DFLL,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR driver.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hence
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nicer to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> done)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from tristate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL (suspend),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to open
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP, then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> selected.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prior to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> also a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running
>>>>>>>>>>>>>>>>>>>>>>>>>>>> off a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on
>>>>>>>>>>>>>>>>>>>>>>>>>>>> KEXEC. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P
>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used
>>>>>>>>>>>>>>>>>>>>>>>>>>> for high
>>>>>>>>>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from
>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP.
>>>>>>>>>>>>>>>>>>>>>>>>>> Alright, then please don't forget to
>>>>>>>>>>>>>>>>>>>>>>>>>> pre-initialize
>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is
>>>>>>>>>>>>>>>>>>>>>>>>> safe to
>>>>>>>>>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same
>>>>>>>>>>>>>>>>>>>>>>>> voltage as
>>>>>>>>>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me
>>>>>>>>>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is defined,
>>>>>>>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>>>>>>>>>> Thank you for the clarification. It would be good
>>>>>>>>>>>>>>>>>>>>>> to have
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq
>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>> happens very early even before disabling non-boot
>>>>>>>>>>>>>>>>>>>> CPUs and
>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                 - Save CPU clock policy registers, and
>>>>>>>>>>>>>>>>>>>> Perform
>>>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                 - Re-init DFLL, Set in Open-Loop mode,
>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>> Clock policy registers which actually sets source to
>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>> along
>>>>>>>>>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                 - do clk_prepare_enable which acutally
>>>>>>>>>>>>>>>>>>>> sets
>>>>>>>>>>>>>>>>>>>> DFLL in
>>>>>>>>>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP is not
>>>>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as DFLL
>>>>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>>>>> disabled anywhere throught the suspend/resume path
>>>>>>>>>>>>>>>>>>>> and SC7
>>>>>>>>>>>>>>>>>>>> entry
>>>>>>>>>>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend
>>>>>>>>>>>>>>>>>> it on
>>>>>>>>>>>>>>>>>> PLLP as well. And besides, seems that currently disabling
>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>> clock will disable DFLL completely and then you'd want to
>>>>>>>>>>>>>>>>>> re-init
>>>>>>>>>>>>>>>>>> the DFLL on resume any ways. So better to just disable
>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>> completely on suspend, which should happen on
>>>>>>>>>>>>>>>>>> clk_disable(dfll).
>>>>>>>>>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With
>>>>>>>>>>>>>>>>> decision of
>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>> clk_disable during suspend, its mandatory to switch to
>>>>>>>>>>>>>>>>> PLLP as
>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>> is completely disabled.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> My earlier concern was on restoring CPU policy as we
>>>>>>>>>>>>>>>>> can't do
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Clear now and will do CPU clock policy restore in after
>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>> re-init.
>>>>>>>>>>>>>>>> Why the policy can't be saved/restored by the CaR driver
>>>>>>>>>>>>>>>> as a
>>>>>>>>>>>>>>>> context of any other clock?
>>>>>>>>>>>>>>> restoring cpu clock policy involves programming source and
>>>>>>>>>>>>>>> super_cclkg_divider.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use
>>>>>>>>>>>>>>> frac_div ops
>>>>>>>>>>>>>>> to do save/restore its divider.
>>>>>>>>>>>>>> That can be changed of course and I guess it also could be as
>>>>>>>>>>>>>> simple as
>>>>>>>>>>>>>> saving and restoring of two raw u32 values of the
>>>>>>>>>>>>>> policy/divider
>>>>>>>>>>>>>> registers.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also, during clock context we cant restore cclk_g as cclk_g
>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>> will be dfll and dfll will not be resumed/re-initialized
>>>>>>>>>>>>>>> by the
>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>> clk_super_mux save/restore happens.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>>>>>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and
>>>>>>>>>>>>>>> dfll_ref
>>>>>>>>>>>>>>> and dfll_soc peripheral clocks are not restored by the
>>>>>>>>>>>>>>> time dfll
>>>>>>>>>>>>>>> restore happens. Also dfll peripheral clock enables need
>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>> restored before dfll restore happens which involves
>>>>>>>>>>>>>>> programming
>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>> controller for re-initialization.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of all
>>>>>>>>>>>>>>> clocks
>>>>>>>>>>>>>>> restore in V5 series but instead of in clk-tegra210 driver I
>>>>>>>>>>>>>>> moved
>>>>>>>>>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies
>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>> restored thru clk_restore_context by then. This will be in
>>>>>>>>>>>>>>> V6.
>>>>>>>>>>>>>> Since DFLL is now guaranteed to be disabled across CaR
>>>>>>>>>>>>>> suspend/resume
>>>>>>>>>>>>>> (hence it has nothing to do in regards to CCLK) and given that
>>>>>>>>>>>>>> PLLs
>>>>>>>>>>>>>> state is restored before the rest of the clocks, I don't
>>>>>>>>>>>>>> see why
>>>>>>>>>>>>>> not to
>>>>>>>>>>>>>> implement CCLK save/restore in a generic fasion. CPU policy
>>>>>>>>>>>>>> wull be
>>>>>>>>>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is
>>>>>>>>>>>>>> disabled).
>>>>>>>>>>>>>>
>>>>>>>>>>>>> CCLK_G save/restore should happen in clk_super_mux ops
>>>>>>>>>>>>> save/context and
>>>>>>>>>>>>> clk_super_mux save/restore happens very early as cclk_g is
>>>>>>>>>>>>> first
>>>>>>>>>>>>> in the
>>>>>>>>>>>>> clock tree and save/restore traverses through the tree
>>>>>>>>>>>>> top-bottom
>>>>>>>>>>>>> order.
>>>>>>>>>>>> If CCLK_G is restored before the PLLs, then just change the
>>>>>>>>>>>> clocks
>>>>>>>>>>>> order
>>>>>>>>>>>> such that it won't happen.
>>>>>>>>>>>>
>>>>>>>>>>> I dont think we can change clocks order for CCLK_G.
>>>>>>>>>>>
>>>>>>>>>>> During bootup, cclk_g is registered after all pll's and
>>>>>>>>>>> peripheral
>>>>>>>>>>> clocks which is the way we wanted, So cclk_g will be the first
>>>>>>>>>>> one in
>>>>>>>>>>> the clk list as clk_register adds new clock first in the list.
>>>>>>>>>>>
>>>>>>>>>>> When clk_save_context and clk_restore_context APIs iterates
>>>>>>>>>>> over the
>>>>>>>>>>> list, cclk_g is the first
>>>>>>>>>> Looking at clk_core_restore_context(), I see that it walks up CLKs
>>>>>>>>>> list
>>>>>>>>>> from parent to children, hence I don't understand how it can ever
>>>>>>>>>> happen
>>>>>>>>>> that CCLK will be restored before the parent. The clocks
>>>>>>>>>> registration
>>>>>>>>>> order doesn't matter at all in that case.
>>>>>>>>> yes from parent to children and dfllCPU_out is the top in the
>>>>>>>>> list and
>>>>>>>>> its child is cclk_g.
>>>>>>>>>
>>>>>>>>> the way clocks are registered is the order I see in the clock
>>>>>>>>> list and
>>>>>>>>> looking into clk_register API it adds new node first in the list.
>>>>>>>>>
>>>>>>>> cclkg_g & dfll register happens after all plls and peripheral
>>>>>>>> clocks as
>>>>>>>> it need ref, soc and peripheral clocks to be enabled.
>>>>>>>>> So they are the last to get registered and so becomes first in the
>>>>>>>>> list.
>>>>>>>>>
>>>>>>>>> During save/restore context, it traverses thru this list and
>>>>>>>>> first in
>>>>>>>>> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>>>>>>>>>
>>>>>>>>> saving should not be an issue at all but we cant restore
>>>>>>>>> cclk_g/dfll
>>>>>>>>> in normal way thru clk_ops restore as plls and peripherals restore
>>>>>>>>> doesn't happen by that time.
>>>>>>>>>
>>>>>>>> I was referring to clk_restore_context where it iterates thru
>>>>>>>> root list
>>>>>>>> and for each core from the root list clk_core_restore does
>>>>>>>> restore of
>>>>>>>> parent and children.
>>>>>>>>
>>>>>>>> dfllCPU_Out gets first in the list and its child is cclk_g
>>>>>>>>
>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105
>>>>>>>>
>>>>>>> What list you're talking about? clk_summary? It shows current
>>>>>>> *active*
>>>>>>> clocks configuration, if you'll try to disable CPUFreq driver then
>>>>>>> the
>>>>>>> parent of CCLK_G should be PLLX. Similarly when CPU is reparented to
>>>>>>> PLLP on driver's suspend, then PLLP is the parent.
>>>>>>>
>>>>>>>>>>>>> DFLL enable thru CPUFreq resume happens after all
>>>>>>>>>>>>> clk_restore_context
>>>>>>>>>>>>> happens. So during clk_restore_context, dfll re-init doesnt
>>>>>>>>>>>>> happen
>>>>>>>>>>>>> and
>>>>>>>>>>>>> doing cpu clock policy restore during super_mux clk_ops will
>>>>>>>>>>>>> crash as
>>>>>>>>>>>>> DFLL is not initialized and its clock is not enabled but CPU
>>>>>>>>>>>>> clock
>>>>>>>>>>>>> restore sets source to DFLL if we restore during super_clk_mux
>>>>>>>>>>>> If CPU was suspended on PLLP, then it will be restored on
>>>>>>>>>>>> PLLP by
>>>>>>>>>>>> CaR. I
>>>>>>>>>>>> don't understand what DFLL has to do with the CCLK in that case
>>>>>>>>>>>> during
>>>>>>>>>>>> the clocks restore.
>>>>>>>>>>> My above comment is in reference to your request of doing
>>>>>>>>>>> save/restore
>>>>>>>>>>> for cclk_g in normal fashion thru save/restore context. Because
>>>>>>>>>>> of the
>>>>>>>>>>> clk order I mentioned above, we cclk_g will be the first one to
>>>>>>>>>>> go thru
>>>>>>>>>>> save/context.
>>>>>>>>>>>
>>>>>>>>>>> During save_context of cclk_g, source can be from PLLX, dfll.
>>>>>>>>>>>
>>>>>>>>>>> Issue will be when we do restore during clk_restore_context of
>>>>>>>>>>> cclk_g as
>>>>>>>>>>> by that time PLLX/dfll will not be restored.
>>>>>>>>>>>
>>>>>>>>>> Seems we already agreed that DFLL will be disabled by the CPUFreq
>>>>>>>>>> driver
>>>>>>>>>> on suspend. Hence CCLK can't be from DFLL if CPU is reparented to
>>>>>>>>>> PLLP
>>>>>>>>>> on CPUFreq driver's suspend, otherwise CPU keeps running from a
>>>>>>>>>> boot-state PLLX if CPUFreq driver is disabled.
>>>>>>>>> Yes suspend should not be an issue but issue will be during resume
>>>>>>>>> where if we do cclk_g restore in normal way thru
>>>>>>>>> clk_restore_context,
>>>>>>>>> cclk_g restore happens very early as dfllCPU out is the first
>>>>>>>>> one that
>>>>>>>>> goes thru restore context and plls/peripherals are not resumed by
>>>>>>>>> then.
>>>>>>>>>
>>>>>>>>> CPU runs from PLLX if dfll clock enable fails during boot. So
>>>>>>>>> when it
>>>>>>>>> gets to suspend, we save CPU running clock source as either PLLX or
>>>>>>>>> DFLL and then we switch to PLLP.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On resume, CPU runs from PLLP by warm boot code and we need to
>>>>>>>>> restore
>>>>>>>>> back its source to the one it was using from saved source context
>>>>>>>>> (which can be either PLLX or DFLL)
>>>>>>>>>
>>>>>>>>> So PLLs & DFLL resume need to happen before CCLKG restore/resume.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> With all above discussions, we do DFLL disable in CPUFreq driver on
>>>>>>>>> suspend and on CPUFreq resume we enable DFLL back and restore CPU
>>>>>>>>> clock source it was using during suspend (which will be either
>>>>>>>>> PLLX if
>>>>>>>>> dfll enable fails during probe or it will be using DFLL).
>>>>>>> During suspend CPU's parent shall be PLLP and not DFLL (note that
>>>>>>> it is
>>>>>>> disabled) after reparenting to PLLP by the CPUFreq driver.
>>>>>>>
>>>>>> CPU source context should be saved before switching to safe source of
>>>>>> PLLP as on resume we need to restore back to source it was using
>>>>>> before we switch to safe source during suspend entry.
>>>>>>
>>>>>> So saved context for CPU Source will be either dfll or PLLX
>>>>>>
>>>>> PLLP reparenting is only during suspend/entry to have it as safe source
>>>>> but actual CPU source it was running from before suspending is either
>>>>> dfll/pllx which should be the one to be restored on CPUFreq resume.
>>>>> Resume happens with CPU running from PLLP till it gets to the point of
>>>>> restoring its original source (dfll or pllx)
>>>> CaR should restore CPU to PLLP or PLLX, while CPUFreq driver restores
>>>> CPU to DFLL. Please see more comments below.
>>>>
>>>>>>>>> So i was trying to say dfll/cclk_g restore can't be done in
>>>>>>>>> normal way
>>>>>>>>> thru clk_ops save/restore context
>>>>>>> Let's see what happens if CPUFreq is active:
>>>>>>>
>>>>>>> 1. CPUFreq driver probe happens
>>>>>>>       2. CPU is reparented to PLLP
>>>>>>>       3. DFLL inited
>>>>>>>       4. CPU is reparented to DFLL
>>>>>>>
>>>>>>> 5. CPUFreq driver suspend happens
>>>>>>>       6. CPU is reparented to PLLP
>>>>>>>       7. DFLL is disabled
>>>>>>>
>>>>>>> 8. Car suspend happens
>>>>>>>       9. DFLL context saved
>>>>>>>       10. PLLP/PLLX context saved
>>>>>>>       11. CCLK context saved
>>>>>>>
>>>>>>> 12. Car resume happens
>>>>>>>       13. DFLL context restored
>>>>>>>       14. PLLP/PLLX context restored
>>>>>>>       15. CCLK context restored
>>>>>>>
>>>>>>> 16. CPUFreq driver resume happens
>>>>>>>       17. DFLL re-inited
>>>>>>>       18. CPU is reparented to DFLL
>>>>>> Below is the order of sequence it should be based on the order of clk
>>>>>> register.
>>>>>>
>>>>>> My comments inline in this sequence.
>>>>>>
>>>>>> 1. CPUFreq driver probe happens
>>>>>>       2. CPU is reparented to PLLP
>>>>>>       3. DFLL inited
>>>>>>       4. CPU is reparented to DFLL
>>>>>>
>>>>>>
>>>>>> 5. CPUFreq driver suspend happens
>>>>>>       6. Save CPU source which could be either dfll or pllx
>>>> Please see my next comment.
>>>>
>>>>>>       7. CPU is reparented to safe known source PLLP
>>>>>>       8. DFLL is disabled
>>>>>>
>>>>>> 8. Car suspend happens
>>>>>>       9. DFLL context saved (With DFLL disabled in CPUFreq suspend,
>>>>>> nothing to be saved here as last freq req will always be saved).
>>>>>>       10. CCLK context saved (CPU clock source will be saved in CPUFreq
>>>>>> driver suspend which could be either dfll or pllx)
>>>> That I don't understand. The CPU's clock source state should be saved at
>>>> the moment of the CaR's suspending (i.e. CCLK policy will be set to PLLP
>>>> or PLLX) and then CCLK state should be also restored by the CaR in
>>>> step 14.
>>> CPU clock to be saved and restored should be the source used before we
>>> switch it to safe PLLP for suspend/resume operation.
>>>
>>> This original source could be either PLLX or DFLL which it was using
>>> before we disable DFLL during CPU Freq suspend.
>>>
>>> If we save CPU clock source at moment of CAR suspending, it will be
>>> PLLP as we switch to safe PLLP in CPUFreq driver suspend.
>>>
>>> Infact, we dont need to restore CPU clock source to PLLP anywhere in
>>> resume as it comes up with PLLP source from warm boot code itself.
> You must always maintain proper suspend/resume encapsulation, otherwise
> it's a total mess. It doesn't matter that CCLK is restored to PLLP even
> that CPU is already running off PLLP after warmboot.
>
>>> But we need to restore CPU source to original source it was using
>>> before we switch to safe PLLP source for suspend operation. This
>>> original source could be PLLX/DFLL and this should be re-stored in
>>> CPUFreq resume as by this time PLLs and peripherals are restored and
>>> dfll is re-initialized.
>>>
>>> So saving actual CPU source before switching to intermediate safe PLLP
>>> in CPUFreq driver and then restoring back during CPUFreq resume should
>>> be good as CPUFreq resume happens right after all clocks (plls
>>> restore, peripherals restore, dfll resume)>>
>>>> CPUFreq driver should only switch CPU to PLLP and disable DFLL on
>>>> suspend in step 5, that's it. On resume CPUFreq driver will restore CPU
>>>> to DFLL in step 18.
>>> Also I don't think we should hard-code to force CPU source to DFLL on
>>> CPUFreq resume.
>>>
>>> Reason is during CPU Probe when it tries to switch to dfll source, for
>>> some reason if dfll enable fails it sets CPU to its original source
>>> which will be PLLX.
> No!
>
> 1. CPU voltage could be too low for PLLX
> 2. PLLX rate can't be changed without manual reparenting CPU to
> intermediate clock
> 3. CPUFreq can't manually manage CPU voltage
>
> DFLL-restoring failure is an extreme case. CPU must be kept on a safe
> PLLP in that case and disable_cpufreq() must be invoked as well.

OK, PLLX option was also in my mind. So If we just consider sources as 
DFLL or PLLP, then we can save source in CCLK save context and restore 
in CCLK restore basically it will be PLLP.

Later during CPUFreq resume we can just switch to DFLL and if DFLL 
enable fails we will keep source as PLLP. Yes will invoke 
disable_cpufreq as well in case of dfll enable failure for some reason.

>>> So CPU source could be either DFLL or PLLX in CPUFreq
>>> tegra124_cpu_switch_to_dfll
>>>
>>>>>>       11. PLLP/PLLX context saved
>>>>>>       12. Peripheral Clock saved
>>>>>>
>>>>>> 12. Car resume happens
>>>>>>       13. DFLL context restored : No DFLL context to be restored and we
>>>>>> only need to reinitialize DFLL and re-initialize can't be done here as
>>>>>> this is the 1st to get restored and PLL/Peripheral clocks are not
>>>>>> restored by this time. So we can't use clk_ops restore for DFLL
>>>> It looks to me that clk_core_restore_context() should just do
>>>> hlist_for_each_entry *_reverse*. Don't you think so?
>>> Thought of that but this is in core driver and is used by other
>>> non-tegra clock driver and not sure if that impacts for those.
> The reverse ordering should be correct, I think it's just a shortcoming
> of the CCF that need to be fixed. But will better to make a more
> thorough research, asking Stephen and Michael for the clarification.
>
>>> But with decision of switching CPUFreq with dfll clock enable/disable
>>> during CPUFreq suspend/resume, we can re-init dfll during dfll-fcpu
>>> driver resume and we don't need CCLK save/restore.
>>>
>> the way of all clocks order is good except cclk_g which has dependency
>> on multiple clocks.
> CCLK_G has a single parent at a time. What "multiple clocks" you're
> talking about? Please explain.

dependencies I am referring are dfll_ref, dfll_soc, and DVFS peripheral 
clocks which need to be restored prior to DFLL reinit.


>> reverse list order during restore might not work as all other clocks are
>> in proper order no with any ref clocks for plls getting restored prior
>> to their clients
> Why? The ref clocks should be registered first and be the roots for PLLs
> and the rest. If it's not currently the case, then this need to be
> fixed. You need to ensure that each clock is modeled properly. If some
> child clock really depends on multiple parents, then the parents need to
> in the correct order or CCF need to be taught about such multi-dependencies.
>
> If some required feature is missed, then you have to implement it
> properly and for all, that's how things are done in upstream. Sometimes
> it's quite a lot of extra work that everyone are benefiting from in the end.
>
> [snip]

Yes, we should register ref/parents before their clients.

cclk_g clk is registered last after all pll and peripheral clocks are 
registers during clock init.

dfllCPU_out clk is registered later during dfll-fcpu driver probe and 
gets added to the clock list.

Probably the issue seems to be not linking dfll_ref and dfll_soc 
dependencies for dfllCPU_out thru clock list.

clk-dfll driver during dfll_init_clks gets ref_clk and soc_clk reference 
thru DT.



^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 23:36                                                                                           ` Sowjanya Komatineni
@ 2019-07-17 23:44                                                                                             ` Dmitry Osipenko
  2019-07-18  0:25                                                                                               ` Sowjanya Komatineni
  2019-07-18 19:42                                                                                               ` Peter De Schrijver
  0 siblings, 2 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-17 23:44 UTC (permalink / raw)
  To: Sowjanya Komatineni, sboyd, Michael Turquette
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree

18.07.2019 2:36, Sowjanya Komatineni пишет:
> 
> On 7/17/19 3:48 PM, Dmitry Osipenko wrote:
>> 18.07.2019 0:57, Sowjanya Komatineni пишет:
>>> On 7/17/19 2:51 PM, Sowjanya Komatineni wrote:
>>>> On 7/17/19 2:30 PM, Dmitry Osipenko wrote:
>>>>> 17.07.2019 23:11, Sowjanya Komatineni пишет:
>>>>>> On 7/17/19 1:01 PM, Sowjanya Komatineni wrote:
>>>>>>> On 7/17/19 12:43 PM, Dmitry Osipenko wrote:
>>>>>>>> 17.07.2019 21:54, Sowjanya Komatineni пишет:
>>>>>>>>> On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>>>>>>>>>> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>>>>>>>>>>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>>>>>>>>>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +0800,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> links".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> See
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> EPROBE_DEFER
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as-is and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DVFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> guaranteed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> closed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cluster. And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hence
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nicer to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> done)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tristate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (suspend),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP, then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> selected.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prior to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> off a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KEXEC. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P
>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for high
>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Alright, then please don't forget to
>>>>>>>>>>>>>>>>>>>>>>>>>>> pre-initialize
>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is
>>>>>>>>>>>>>>>>>>>>>>>>>> safe to
>>>>>>>>>>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same
>>>>>>>>>>>>>>>>>>>>>>>>> voltage as
>>>>>>>>>>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me
>>>>>>>>>>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is
>>>>>>>>>>>>>>>>>>>>>>>>> defined,
>>>>>>>>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below
>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>>>>>>>>>>> Thank you for the clarification. It would be good
>>>>>>>>>>>>>>>>>>>>>>> to have
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq
>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>> happens very early even before disabling non-boot
>>>>>>>>>>>>>>>>>>>>> CPUs and
>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                 - Save CPU clock policy registers, and
>>>>>>>>>>>>>>>>>>>>> Perform
>>>>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                 - Re-init DFLL, Set in Open-Loop mode,
>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>> Clock policy registers which actually sets source to
>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>> along
>>>>>>>>>>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                 - do clk_prepare_enable which acutally
>>>>>>>>>>>>>>>>>>>>> sets
>>>>>>>>>>>>>>>>>>>>> DFLL in
>>>>>>>>>>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP
>>>>>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as
>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>>>>>> disabled anywhere throught the suspend/resume path
>>>>>>>>>>>>>>>>>>>>> and SC7
>>>>>>>>>>>>>>>>>>>>> entry
>>>>>>>>>>>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>>>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend
>>>>>>>>>>>>>>>>>>> it on
>>>>>>>>>>>>>>>>>>> PLLP as well. And besides, seems that currently
>>>>>>>>>>>>>>>>>>> disabling
>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>> clock will disable DFLL completely and then you'd
>>>>>>>>>>>>>>>>>>> want to
>>>>>>>>>>>>>>>>>>> re-init
>>>>>>>>>>>>>>>>>>> the DFLL on resume any ways. So better to just disable
>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>> completely on suspend, which should happen on
>>>>>>>>>>>>>>>>>>> clk_disable(dfll).
>>>>>>>>>>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With
>>>>>>>>>>>>>>>>>> decision of
>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>> clk_disable during suspend, its mandatory to switch to
>>>>>>>>>>>>>>>>>> PLLP as
>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>> is completely disabled.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> My earlier concern was on restoring CPU policy as we
>>>>>>>>>>>>>>>>>> can't do
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Clear now and will do CPU clock policy restore in after
>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>> re-init.
>>>>>>>>>>>>>>>>> Why the policy can't be saved/restored by the CaR driver
>>>>>>>>>>>>>>>>> as a
>>>>>>>>>>>>>>>>> context of any other clock?
>>>>>>>>>>>>>>>> restoring cpu clock policy involves programming source and
>>>>>>>>>>>>>>>> super_cclkg_divider.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use
>>>>>>>>>>>>>>>> frac_div ops
>>>>>>>>>>>>>>>> to do save/restore its divider.
>>>>>>>>>>>>>>> That can be changed of course and I guess it also could
>>>>>>>>>>>>>>> be as
>>>>>>>>>>>>>>> simple as
>>>>>>>>>>>>>>> saving and restoring of two raw u32 values of the
>>>>>>>>>>>>>>> policy/divider
>>>>>>>>>>>>>>> registers.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, during clock context we cant restore cclk_g as cclk_g
>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>> will be dfll and dfll will not be resumed/re-initialized
>>>>>>>>>>>>>>>> by the
>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>> clk_super_mux save/restore happens.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>>>>>>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and
>>>>>>>>>>>>>>>> dfll_ref
>>>>>>>>>>>>>>>> and dfll_soc peripheral clocks are not restored by the
>>>>>>>>>>>>>>>> time dfll
>>>>>>>>>>>>>>>> restore happens. Also dfll peripheral clock enables need
>>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>> restored before dfll restore happens which involves
>>>>>>>>>>>>>>>> programming
>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>> controller for re-initialization.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of
>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>> clocks
>>>>>>>>>>>>>>>> restore in V5 series but instead of in clk-tegra210
>>>>>>>>>>>>>>>> driver I
>>>>>>>>>>>>>>>> moved
>>>>>>>>>>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies
>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>> restored thru clk_restore_context by then. This will be in
>>>>>>>>>>>>>>>> V6.
>>>>>>>>>>>>>>> Since DFLL is now guaranteed to be disabled across CaR
>>>>>>>>>>>>>>> suspend/resume
>>>>>>>>>>>>>>> (hence it has nothing to do in regards to CCLK) and given
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> PLLs
>>>>>>>>>>>>>>> state is restored before the rest of the clocks, I don't
>>>>>>>>>>>>>>> see why
>>>>>>>>>>>>>>> not to
>>>>>>>>>>>>>>> implement CCLK save/restore in a generic fasion. CPU policy
>>>>>>>>>>>>>>> wull be
>>>>>>>>>>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is
>>>>>>>>>>>>>>> disabled).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> CCLK_G save/restore should happen in clk_super_mux ops
>>>>>>>>>>>>>> save/context and
>>>>>>>>>>>>>> clk_super_mux save/restore happens very early as cclk_g is
>>>>>>>>>>>>>> first
>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>> clock tree and save/restore traverses through the tree
>>>>>>>>>>>>>> top-bottom
>>>>>>>>>>>>>> order.
>>>>>>>>>>>>> If CCLK_G is restored before the PLLs, then just change the
>>>>>>>>>>>>> clocks
>>>>>>>>>>>>> order
>>>>>>>>>>>>> such that it won't happen.
>>>>>>>>>>>>>
>>>>>>>>>>>> I dont think we can change clocks order for CCLK_G.
>>>>>>>>>>>>
>>>>>>>>>>>> During bootup, cclk_g is registered after all pll's and
>>>>>>>>>>>> peripheral
>>>>>>>>>>>> clocks which is the way we wanted, So cclk_g will be the first
>>>>>>>>>>>> one in
>>>>>>>>>>>> the clk list as clk_register adds new clock first in the list.
>>>>>>>>>>>>
>>>>>>>>>>>> When clk_save_context and clk_restore_context APIs iterates
>>>>>>>>>>>> over the
>>>>>>>>>>>> list, cclk_g is the first
>>>>>>>>>>> Looking at clk_core_restore_context(), I see that it walks up
>>>>>>>>>>> CLKs
>>>>>>>>>>> list
>>>>>>>>>>> from parent to children, hence I don't understand how it can
>>>>>>>>>>> ever
>>>>>>>>>>> happen
>>>>>>>>>>> that CCLK will be restored before the parent. The clocks
>>>>>>>>>>> registration
>>>>>>>>>>> order doesn't matter at all in that case.
>>>>>>>>>> yes from parent to children and dfllCPU_out is the top in the
>>>>>>>>>> list and
>>>>>>>>>> its child is cclk_g.
>>>>>>>>>>
>>>>>>>>>> the way clocks are registered is the order I see in the clock
>>>>>>>>>> list and
>>>>>>>>>> looking into clk_register API it adds new node first in the list.
>>>>>>>>>>
>>>>>>>>> cclkg_g & dfll register happens after all plls and peripheral
>>>>>>>>> clocks as
>>>>>>>>> it need ref, soc and peripheral clocks to be enabled.
>>>>>>>>>> So they are the last to get registered and so becomes first in
>>>>>>>>>> the
>>>>>>>>>> list.
>>>>>>>>>>
>>>>>>>>>> During save/restore context, it traverses thru this list and
>>>>>>>>>> first in
>>>>>>>>>> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>>>>>>>>>>
>>>>>>>>>> saving should not be an issue at all but we cant restore
>>>>>>>>>> cclk_g/dfll
>>>>>>>>>> in normal way thru clk_ops restore as plls and peripherals
>>>>>>>>>> restore
>>>>>>>>>> doesn't happen by that time.
>>>>>>>>>>
>>>>>>>>> I was referring to clk_restore_context where it iterates thru
>>>>>>>>> root list
>>>>>>>>> and for each core from the root list clk_core_restore does
>>>>>>>>> restore of
>>>>>>>>> parent and children.
>>>>>>>>>
>>>>>>>>> dfllCPU_Out gets first in the list and its child is cclk_g
>>>>>>>>>
>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105
>>>>>>>>>
>>>>>>>>>
>>>>>>>> What list you're talking about? clk_summary? It shows current
>>>>>>>> *active*
>>>>>>>> clocks configuration, if you'll try to disable CPUFreq driver then
>>>>>>>> the
>>>>>>>> parent of CCLK_G should be PLLX. Similarly when CPU is
>>>>>>>> reparented to
>>>>>>>> PLLP on driver's suspend, then PLLP is the parent.
>>>>>>>>
>>>>>>>>>>>>>> DFLL enable thru CPUFreq resume happens after all
>>>>>>>>>>>>>> clk_restore_context
>>>>>>>>>>>>>> happens. So during clk_restore_context, dfll re-init doesnt
>>>>>>>>>>>>>> happen
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> doing cpu clock policy restore during super_mux clk_ops will
>>>>>>>>>>>>>> crash as
>>>>>>>>>>>>>> DFLL is not initialized and its clock is not enabled but CPU
>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>> restore sets source to DFLL if we restore during
>>>>>>>>>>>>>> super_clk_mux
>>>>>>>>>>>>> If CPU was suspended on PLLP, then it will be restored on
>>>>>>>>>>>>> PLLP by
>>>>>>>>>>>>> CaR. I
>>>>>>>>>>>>> don't understand what DFLL has to do with the CCLK in that
>>>>>>>>>>>>> case
>>>>>>>>>>>>> during
>>>>>>>>>>>>> the clocks restore.
>>>>>>>>>>>> My above comment is in reference to your request of doing
>>>>>>>>>>>> save/restore
>>>>>>>>>>>> for cclk_g in normal fashion thru save/restore context. Because
>>>>>>>>>>>> of the
>>>>>>>>>>>> clk order I mentioned above, we cclk_g will be the first one to
>>>>>>>>>>>> go thru
>>>>>>>>>>>> save/context.
>>>>>>>>>>>>
>>>>>>>>>>>> During save_context of cclk_g, source can be from PLLX, dfll.
>>>>>>>>>>>>
>>>>>>>>>>>> Issue will be when we do restore during clk_restore_context of
>>>>>>>>>>>> cclk_g as
>>>>>>>>>>>> by that time PLLX/dfll will not be restored.
>>>>>>>>>>>>
>>>>>>>>>>> Seems we already agreed that DFLL will be disabled by the
>>>>>>>>>>> CPUFreq
>>>>>>>>>>> driver
>>>>>>>>>>> on suspend. Hence CCLK can't be from DFLL if CPU is
>>>>>>>>>>> reparented to
>>>>>>>>>>> PLLP
>>>>>>>>>>> on CPUFreq driver's suspend, otherwise CPU keeps running from a
>>>>>>>>>>> boot-state PLLX if CPUFreq driver is disabled.
>>>>>>>>>> Yes suspend should not be an issue but issue will be during
>>>>>>>>>> resume
>>>>>>>>>> where if we do cclk_g restore in normal way thru
>>>>>>>>>> clk_restore_context,
>>>>>>>>>> cclk_g restore happens very early as dfllCPU out is the first
>>>>>>>>>> one that
>>>>>>>>>> goes thru restore context and plls/peripherals are not resumed by
>>>>>>>>>> then.
>>>>>>>>>>
>>>>>>>>>> CPU runs from PLLX if dfll clock enable fails during boot. So
>>>>>>>>>> when it
>>>>>>>>>> gets to suspend, we save CPU running clock source as either
>>>>>>>>>> PLLX or
>>>>>>>>>> DFLL and then we switch to PLLP.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On resume, CPU runs from PLLP by warm boot code and we need to
>>>>>>>>>> restore
>>>>>>>>>> back its source to the one it was using from saved source context
>>>>>>>>>> (which can be either PLLX or DFLL)
>>>>>>>>>>
>>>>>>>>>> So PLLs & DFLL resume need to happen before CCLKG restore/resume.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> With all above discussions, we do DFLL disable in CPUFreq
>>>>>>>>>> driver on
>>>>>>>>>> suspend and on CPUFreq resume we enable DFLL back and restore CPU
>>>>>>>>>> clock source it was using during suspend (which will be either
>>>>>>>>>> PLLX if
>>>>>>>>>> dfll enable fails during probe or it will be using DFLL).
>>>>>>>> During suspend CPU's parent shall be PLLP and not DFLL (note that
>>>>>>>> it is
>>>>>>>> disabled) after reparenting to PLLP by the CPUFreq driver.
>>>>>>>>
>>>>>>> CPU source context should be saved before switching to safe
>>>>>>> source of
>>>>>>> PLLP as on resume we need to restore back to source it was using
>>>>>>> before we switch to safe source during suspend entry.
>>>>>>>
>>>>>>> So saved context for CPU Source will be either dfll or PLLX
>>>>>>>
>>>>>> PLLP reparenting is only during suspend/entry to have it as safe
>>>>>> source
>>>>>> but actual CPU source it was running from before suspending is either
>>>>>> dfll/pllx which should be the one to be restored on CPUFreq resume.
>>>>>> Resume happens with CPU running from PLLP till it gets to the
>>>>>> point of
>>>>>> restoring its original source (dfll or pllx)
>>>>> CaR should restore CPU to PLLP or PLLX, while CPUFreq driver restores
>>>>> CPU to DFLL. Please see more comments below.
>>>>>
>>>>>>>>>> So i was trying to say dfll/cclk_g restore can't be done in
>>>>>>>>>> normal way
>>>>>>>>>> thru clk_ops save/restore context
>>>>>>>> Let's see what happens if CPUFreq is active:
>>>>>>>>
>>>>>>>> 1. CPUFreq driver probe happens
>>>>>>>>       2. CPU is reparented to PLLP
>>>>>>>>       3. DFLL inited
>>>>>>>>       4. CPU is reparented to DFLL
>>>>>>>>
>>>>>>>> 5. CPUFreq driver suspend happens
>>>>>>>>       6. CPU is reparented to PLLP
>>>>>>>>       7. DFLL is disabled
>>>>>>>>
>>>>>>>> 8. Car suspend happens
>>>>>>>>       9. DFLL context saved
>>>>>>>>       10. PLLP/PLLX context saved
>>>>>>>>       11. CCLK context saved
>>>>>>>>
>>>>>>>> 12. Car resume happens
>>>>>>>>       13. DFLL context restored
>>>>>>>>       14. PLLP/PLLX context restored
>>>>>>>>       15. CCLK context restored
>>>>>>>>
>>>>>>>> 16. CPUFreq driver resume happens
>>>>>>>>       17. DFLL re-inited
>>>>>>>>       18. CPU is reparented to DFLL
>>>>>>> Below is the order of sequence it should be based on the order of
>>>>>>> clk
>>>>>>> register.
>>>>>>>
>>>>>>> My comments inline in this sequence.
>>>>>>>
>>>>>>> 1. CPUFreq driver probe happens
>>>>>>>       2. CPU is reparented to PLLP
>>>>>>>       3. DFLL inited
>>>>>>>       4. CPU is reparented to DFLL
>>>>>>>
>>>>>>>
>>>>>>> 5. CPUFreq driver suspend happens
>>>>>>>       6. Save CPU source which could be either dfll or pllx
>>>>> Please see my next comment.
>>>>>
>>>>>>>       7. CPU is reparented to safe known source PLLP
>>>>>>>       8. DFLL is disabled
>>>>>>>
>>>>>>> 8. Car suspend happens
>>>>>>>       9. DFLL context saved (With DFLL disabled in CPUFreq suspend,
>>>>>>> nothing to be saved here as last freq req will always be saved).
>>>>>>>       10. CCLK context saved (CPU clock source will be saved in
>>>>>>> CPUFreq
>>>>>>> driver suspend which could be either dfll or pllx)
>>>>> That I don't understand. The CPU's clock source state should be
>>>>> saved at
>>>>> the moment of the CaR's suspending (i.e. CCLK policy will be set to
>>>>> PLLP
>>>>> or PLLX) and then CCLK state should be also restored by the CaR in
>>>>> step 14.
>>>> CPU clock to be saved and restored should be the source used before we
>>>> switch it to safe PLLP for suspend/resume operation.
>>>>
>>>> This original source could be either PLLX or DFLL which it was using
>>>> before we disable DFLL during CPU Freq suspend.
>>>>
>>>> If we save CPU clock source at moment of CAR suspending, it will be
>>>> PLLP as we switch to safe PLLP in CPUFreq driver suspend.
>>>>
>>>> Infact, we dont need to restore CPU clock source to PLLP anywhere in
>>>> resume as it comes up with PLLP source from warm boot code itself.
>> You must always maintain proper suspend/resume encapsulation, otherwise
>> it's a total mess. It doesn't matter that CCLK is restored to PLLP even
>> that CPU is already running off PLLP after warmboot.
>>
>>>> But we need to restore CPU source to original source it was using
>>>> before we switch to safe PLLP source for suspend operation. This
>>>> original source could be PLLX/DFLL and this should be re-stored in
>>>> CPUFreq resume as by this time PLLs and peripherals are restored and
>>>> dfll is re-initialized.
>>>>
>>>> So saving actual CPU source before switching to intermediate safe PLLP
>>>> in CPUFreq driver and then restoring back during CPUFreq resume should
>>>> be good as CPUFreq resume happens right after all clocks (plls
>>>> restore, peripherals restore, dfll resume)>>
>>>>> CPUFreq driver should only switch CPU to PLLP and disable DFLL on
>>>>> suspend in step 5, that's it. On resume CPUFreq driver will restore
>>>>> CPU
>>>>> to DFLL in step 18.
>>>> Also I don't think we should hard-code to force CPU source to DFLL on
>>>> CPUFreq resume.
>>>>
>>>> Reason is during CPU Probe when it tries to switch to dfll source, for
>>>> some reason if dfll enable fails it sets CPU to its original source
>>>> which will be PLLX.
>> No!
>>
>> 1. CPU voltage could be too low for PLLX
>> 2. PLLX rate can't be changed without manual reparenting CPU to
>> intermediate clock
>> 3. CPUFreq can't manually manage CPU voltage
>>
>> DFLL-restoring failure is an extreme case. CPU must be kept on a safe
>> PLLP in that case and disable_cpufreq() must be invoked as well.
> 
> OK, PLLX option was also in my mind. So If we just consider sources as
> DFLL or PLLP, then we can save source in CCLK save context and restore
> in CCLK restore basically it will be PLLP.
> 
> Later during CPUFreq resume we can just switch to DFLL and if DFLL
> enable fails we will keep source as PLLP. Yes will invoke
> disable_cpufreq as well in case of dfll enable failure for some reason.

Sounds good!

>>>> So CPU source could be either DFLL or PLLX in CPUFreq
>>>> tegra124_cpu_switch_to_dfll
>>>>
>>>>>>>       11. PLLP/PLLX context saved
>>>>>>>       12. Peripheral Clock saved
>>>>>>>
>>>>>>> 12. Car resume happens
>>>>>>>       13. DFLL context restored : No DFLL context to be restored
>>>>>>> and we
>>>>>>> only need to reinitialize DFLL and re-initialize can't be done
>>>>>>> here as
>>>>>>> this is the 1st to get restored and PLL/Peripheral clocks are not
>>>>>>> restored by this time. So we can't use clk_ops restore for DFLL
>>>>> It looks to me that clk_core_restore_context() should just do
>>>>> hlist_for_each_entry *_reverse*. Don't you think so?
>>>> Thought of that but this is in core driver and is used by other
>>>> non-tegra clock driver and not sure if that impacts for those.
>> The reverse ordering should be correct, I think it's just a shortcoming
>> of the CCF that need to be fixed. But will better to make a more
>> thorough research, asking Stephen and Michael for the clarification.
>>
>>>> But with decision of switching CPUFreq with dfll clock enable/disable
>>>> during CPUFreq suspend/resume, we can re-init dfll during dfll-fcpu
>>>> driver resume and we don't need CCLK save/restore.
>>>>

Actually CPUFreq driver should implement suspend/resume regardless of
CaR ability to restore DFLL or whatever, simply to properly handle
possible clock restoring failure on resume as we just found out.

>>> the way of all clocks order is good except cclk_g which has dependency
>>> on multiple clocks.
>> CCLK_G has a single parent at a time. What "multiple clocks" you're
>> talking about? Please explain.
> 
> dependencies I am referring are dfll_ref, dfll_soc, and DVFS peripheral
> clocks which need to be restored prior to DFLL reinit.

Okay, but that shouldn't be a problem if clock dependencies are set up
properly.

>>> reverse list order during restore might not work as all other clocks are
>>> in proper order no with any ref clocks for plls getting restored prior
>>> to their clients
>> Why? The ref clocks should be registered first and be the roots for PLLs
>> and the rest. If it's not currently the case, then this need to be
>> fixed. You need to ensure that each clock is modeled properly. If some
>> child clock really depends on multiple parents, then the parents need to
>> in the correct order or CCF need to be taught about such
>> multi-dependencies.
>>
>> If some required feature is missed, then you have to implement it
>> properly and for all, that's how things are done in upstream. Sometimes
>> it's quite a lot of extra work that everyone are benefiting from in
>> the end.
>>
>> [snip]
> 
> Yes, we should register ref/parents before their clients.
> 
> cclk_g clk is registered last after all pll and peripheral clocks are
> registers during clock init.
> 
> dfllCPU_out clk is registered later during dfll-fcpu driver probe and
> gets added to the clock list.
> 
> Probably the issue seems to be not linking dfll_ref and dfll_soc
> dependencies for dfllCPU_out thru clock list.
> 
> clk-dfll driver during dfll_init_clks gets ref_clk and soc_clk reference
> thru DT.

Please try to fix all missing dependencies and orderings.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 23:44                                                                                             ` Dmitry Osipenko
@ 2019-07-18  0:25                                                                                               ` Sowjanya Komatineni
  2019-07-18  1:15                                                                                                 ` Sowjanya Komatineni
  2019-07-18 19:42                                                                                               ` Peter De Schrijver
  1 sibling, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-18  0:25 UTC (permalink / raw)
  To: Dmitry Osipenko, sboyd, Michael Turquette
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree


On 7/17/19 4:44 PM, Dmitry Osipenko wrote:
> 18.07.2019 2:36, Sowjanya Komatineni пишет:
>> On 7/17/19 3:48 PM, Dmitry Osipenko wrote:
>>> 18.07.2019 0:57, Sowjanya Komatineni пишет:
>>>> On 7/17/19 2:51 PM, Sowjanya Komatineni wrote:
>>>>> On 7/17/19 2:30 PM, Dmitry Osipenko wrote:
>>>>>> 17.07.2019 23:11, Sowjanya Komatineni пишет:
>>>>>>> On 7/17/19 1:01 PM, Sowjanya Komatineni wrote:
>>>>>>>> On 7/17/19 12:43 PM, Dmitry Osipenko wrote:
>>>>>>>>> 17.07.2019 21:54, Sowjanya Komatineni пишет:
>>>>>>>>>> On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>>>>>>>>>>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +0800,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> links".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> See
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> EPROBE_DEFER
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as-is and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DVFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't reparent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> guaranteed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different sources.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in order to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> closed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand why we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cluster. And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hence
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nicer to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this driver is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> done)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tristate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (suspend),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to PLLP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP, then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> selected.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prior to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always running
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> off a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KEXEC. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By PLL_P
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for high
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alright, then please don't forget to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> pre-initialize
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> reasonable value using tegra_clk_init_table or
>>>>>>>>>>>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is
>>>>>>>>>>>>>>>>>>>>>>>>>>> safe to
>>>>>>>>>>>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same
>>>>>>>>>>>>>>>>>>>>>>>>>> voltage as
>>>>>>>>>>>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious to me
>>>>>>>>>>>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is
>>>>>>>>>>>>>>>>>>>>>>>>>> defined,
>>>>>>>>>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ defined
>>>>>>>>>>>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything below
>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the clarification. It would be good
>>>>>>>>>>>>>>>>>>>>>>>> to have
>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, CPUFreq
>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>> happens very early even before disabling non-boot
>>>>>>>>>>>>>>>>>>>>>> CPUs and
>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                  - Save CPU clock policy registers, and
>>>>>>>>>>>>>>>>>>>>>> Perform
>>>>>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                  - Re-init DFLL, Set in Open-Loop mode,
>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>> Clock policy registers which actually sets source to
>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>> along
>>>>>>>>>>>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                  - do clk_prepare_enable which acutally
>>>>>>>>>>>>>>>>>>>>>> sets
>>>>>>>>>>>>>>>>>>>>>> DFLL in
>>>>>>>>>>>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP
>>>>>>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as
>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>>>>>>> disabled anywhere throught the suspend/resume path
>>>>>>>>>>>>>>>>>>>>>> and SC7
>>>>>>>>>>>>>>>>>>>>>> entry
>>>>>>>>>>>>>>>>>>>>>> FW and Warm boot code will switch CPU source to PLLP.
>>>>>>>>>>>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to suspend
>>>>>>>>>>>>>>>>>>>> it on
>>>>>>>>>>>>>>>>>>>> PLLP as well. And besides, seems that currently
>>>>>>>>>>>>>>>>>>>> disabling
>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>> clock will disable DFLL completely and then you'd
>>>>>>>>>>>>>>>>>>>> want to
>>>>>>>>>>>>>>>>>>>> re-init
>>>>>>>>>>>>>>>>>>>> the DFLL on resume any ways. So better to just disable
>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>> completely on suspend, which should happen on
>>>>>>>>>>>>>>>>>>>> clk_disable(dfll).
>>>>>>>>>>>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With
>>>>>>>>>>>>>>>>>>> decision of
>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>> clk_disable during suspend, its mandatory to switch to
>>>>>>>>>>>>>>>>>>> PLLP as
>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>> is completely disabled.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> My earlier concern was on restoring CPU policy as we
>>>>>>>>>>>>>>>>>>> can't do
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Clear now and will do CPU clock policy restore in after
>>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>>> re-init.
>>>>>>>>>>>>>>>>>> Why the policy can't be saved/restored by the CaR driver
>>>>>>>>>>>>>>>>>> as a
>>>>>>>>>>>>>>>>>> context of any other clock?
>>>>>>>>>>>>>>>>> restoring cpu clock policy involves programming source and
>>>>>>>>>>>>>>>>> super_cclkg_divider.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use
>>>>>>>>>>>>>>>>> frac_div ops
>>>>>>>>>>>>>>>>> to do save/restore its divider.
>>>>>>>>>>>>>>>> That can be changed of course and I guess it also could
>>>>>>>>>>>>>>>> be as
>>>>>>>>>>>>>>>> simple as
>>>>>>>>>>>>>>>> saving and restoring of two raw u32 values of the
>>>>>>>>>>>>>>>> policy/divider
>>>>>>>>>>>>>>>> registers.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also, during clock context we cant restore cclk_g as cclk_g
>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>> will be dfll and dfll will not be resumed/re-initialized
>>>>>>>>>>>>>>>>> by the
>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>> clk_super_mux save/restore happens.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> we can't use save/restore context for dfll clk_ops because
>>>>>>>>>>>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock tree and
>>>>>>>>>>>>>>>>> dfll_ref
>>>>>>>>>>>>>>>>> and dfll_soc peripheral clocks are not restored by the
>>>>>>>>>>>>>>>>> time dfll
>>>>>>>>>>>>>>>>> restore happens. Also dfll peripheral clock enables need
>>>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>>> restored before dfll restore happens which involves
>>>>>>>>>>>>>>>>> programming
>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>> controller for re-initialization.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of
>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>> clocks
>>>>>>>>>>>>>>>>> restore in V5 series but instead of in clk-tegra210
>>>>>>>>>>>>>>>>> driver I
>>>>>>>>>>>>>>>>> moved
>>>>>>>>>>>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies
>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>> restored thru clk_restore_context by then. This will be in
>>>>>>>>>>>>>>>>> V6.
>>>>>>>>>>>>>>>> Since DFLL is now guaranteed to be disabled across CaR
>>>>>>>>>>>>>>>> suspend/resume
>>>>>>>>>>>>>>>> (hence it has nothing to do in regards to CCLK) and given
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> PLLs
>>>>>>>>>>>>>>>> state is restored before the rest of the clocks, I don't
>>>>>>>>>>>>>>>> see why
>>>>>>>>>>>>>>>> not to
>>>>>>>>>>>>>>>> implement CCLK save/restore in a generic fasion. CPU policy
>>>>>>>>>>>>>>>> wull be
>>>>>>>>>>>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is
>>>>>>>>>>>>>>>> disabled).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> CCLK_G save/restore should happen in clk_super_mux ops
>>>>>>>>>>>>>>> save/context and
>>>>>>>>>>>>>>> clk_super_mux save/restore happens very early as cclk_g is
>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>> clock tree and save/restore traverses through the tree
>>>>>>>>>>>>>>> top-bottom
>>>>>>>>>>>>>>> order.
>>>>>>>>>>>>>> If CCLK_G is restored before the PLLs, then just change the
>>>>>>>>>>>>>> clocks
>>>>>>>>>>>>>> order
>>>>>>>>>>>>>> such that it won't happen.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> I dont think we can change clocks order for CCLK_G.
>>>>>>>>>>>>>
>>>>>>>>>>>>> During bootup, cclk_g is registered after all pll's and
>>>>>>>>>>>>> peripheral
>>>>>>>>>>>>> clocks which is the way we wanted, So cclk_g will be the first
>>>>>>>>>>>>> one in
>>>>>>>>>>>>> the clk list as clk_register adds new clock first in the list.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When clk_save_context and clk_restore_context APIs iterates
>>>>>>>>>>>>> over the
>>>>>>>>>>>>> list, cclk_g is the first
>>>>>>>>>>>> Looking at clk_core_restore_context(), I see that it walks up
>>>>>>>>>>>> CLKs
>>>>>>>>>>>> list
>>>>>>>>>>>> from parent to children, hence I don't understand how it can
>>>>>>>>>>>> ever
>>>>>>>>>>>> happen
>>>>>>>>>>>> that CCLK will be restored before the parent. The clocks
>>>>>>>>>>>> registration
>>>>>>>>>>>> order doesn't matter at all in that case.
>>>>>>>>>>> yes from parent to children and dfllCPU_out is the top in the
>>>>>>>>>>> list and
>>>>>>>>>>> its child is cclk_g.
>>>>>>>>>>>
>>>>>>>>>>> the way clocks are registered is the order I see in the clock
>>>>>>>>>>> list and
>>>>>>>>>>> looking into clk_register API it adds new node first in the list.
>>>>>>>>>>>
>>>>>>>>>> cclkg_g & dfll register happens after all plls and peripheral
>>>>>>>>>> clocks as
>>>>>>>>>> it need ref, soc and peripheral clocks to be enabled.
>>>>>>>>>>> So they are the last to get registered and so becomes first in
>>>>>>>>>>> the
>>>>>>>>>>> list.
>>>>>>>>>>>
>>>>>>>>>>> During save/restore context, it traverses thru this list and
>>>>>>>>>>> first in
>>>>>>>>>>> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>>>>>>>>>>>
>>>>>>>>>>> saving should not be an issue at all but we cant restore
>>>>>>>>>>> cclk_g/dfll
>>>>>>>>>>> in normal way thru clk_ops restore as plls and peripherals
>>>>>>>>>>> restore
>>>>>>>>>>> doesn't happen by that time.
>>>>>>>>>>>
>>>>>>>>>> I was referring to clk_restore_context where it iterates thru
>>>>>>>>>> root list
>>>>>>>>>> and for each core from the root list clk_core_restore does
>>>>>>>>>> restore of
>>>>>>>>>> parent and children.
>>>>>>>>>>
>>>>>>>>>> dfllCPU_Out gets first in the list and its child is cclk_g
>>>>>>>>>>
>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> What list you're talking about? clk_summary? It shows current
>>>>>>>>> *active*
>>>>>>>>> clocks configuration, if you'll try to disable CPUFreq driver then
>>>>>>>>> the
>>>>>>>>> parent of CCLK_G should be PLLX. Similarly when CPU is
>>>>>>>>> reparented to
>>>>>>>>> PLLP on driver's suspend, then PLLP is the parent.
>>>>>>>>>
>>>>>>>>>>>>>>> DFLL enable thru CPUFreq resume happens after all
>>>>>>>>>>>>>>> clk_restore_context
>>>>>>>>>>>>>>> happens. So during clk_restore_context, dfll re-init doesnt
>>>>>>>>>>>>>>> happen
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> doing cpu clock policy restore during super_mux clk_ops will
>>>>>>>>>>>>>>> crash as
>>>>>>>>>>>>>>> DFLL is not initialized and its clock is not enabled but CPU
>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>> restore sets source to DFLL if we restore during
>>>>>>>>>>>>>>> super_clk_mux
>>>>>>>>>>>>>> If CPU was suspended on PLLP, then it will be restored on
>>>>>>>>>>>>>> PLLP by
>>>>>>>>>>>>>> CaR. I
>>>>>>>>>>>>>> don't understand what DFLL has to do with the CCLK in that
>>>>>>>>>>>>>> case
>>>>>>>>>>>>>> during
>>>>>>>>>>>>>> the clocks restore.
>>>>>>>>>>>>> My above comment is in reference to your request of doing
>>>>>>>>>>>>> save/restore
>>>>>>>>>>>>> for cclk_g in normal fashion thru save/restore context. Because
>>>>>>>>>>>>> of the
>>>>>>>>>>>>> clk order I mentioned above, we cclk_g will be the first one to
>>>>>>>>>>>>> go thru
>>>>>>>>>>>>> save/context.
>>>>>>>>>>>>>
>>>>>>>>>>>>> During save_context of cclk_g, source can be from PLLX, dfll.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Issue will be when we do restore during clk_restore_context of
>>>>>>>>>>>>> cclk_g as
>>>>>>>>>>>>> by that time PLLX/dfll will not be restored.
>>>>>>>>>>>>>
>>>>>>>>>>>> Seems we already agreed that DFLL will be disabled by the
>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>> driver
>>>>>>>>>>>> on suspend. Hence CCLK can't be from DFLL if CPU is
>>>>>>>>>>>> reparented to
>>>>>>>>>>>> PLLP
>>>>>>>>>>>> on CPUFreq driver's suspend, otherwise CPU keeps running from a
>>>>>>>>>>>> boot-state PLLX if CPUFreq driver is disabled.
>>>>>>>>>>> Yes suspend should not be an issue but issue will be during
>>>>>>>>>>> resume
>>>>>>>>>>> where if we do cclk_g restore in normal way thru
>>>>>>>>>>> clk_restore_context,
>>>>>>>>>>> cclk_g restore happens very early as dfllCPU out is the first
>>>>>>>>>>> one that
>>>>>>>>>>> goes thru restore context and plls/peripherals are not resumed by
>>>>>>>>>>> then.
>>>>>>>>>>>
>>>>>>>>>>> CPU runs from PLLX if dfll clock enable fails during boot. So
>>>>>>>>>>> when it
>>>>>>>>>>> gets to suspend, we save CPU running clock source as either
>>>>>>>>>>> PLLX or
>>>>>>>>>>> DFLL and then we switch to PLLP.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On resume, CPU runs from PLLP by warm boot code and we need to
>>>>>>>>>>> restore
>>>>>>>>>>> back its source to the one it was using from saved source context
>>>>>>>>>>> (which can be either PLLX or DFLL)
>>>>>>>>>>>
>>>>>>>>>>> So PLLs & DFLL resume need to happen before CCLKG restore/resume.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> With all above discussions, we do DFLL disable in CPUFreq
>>>>>>>>>>> driver on
>>>>>>>>>>> suspend and on CPUFreq resume we enable DFLL back and restore CPU
>>>>>>>>>>> clock source it was using during suspend (which will be either
>>>>>>>>>>> PLLX if
>>>>>>>>>>> dfll enable fails during probe or it will be using DFLL).
>>>>>>>>> During suspend CPU's parent shall be PLLP and not DFLL (note that
>>>>>>>>> it is
>>>>>>>>> disabled) after reparenting to PLLP by the CPUFreq driver.
>>>>>>>>>
>>>>>>>> CPU source context should be saved before switching to safe
>>>>>>>> source of
>>>>>>>> PLLP as on resume we need to restore back to source it was using
>>>>>>>> before we switch to safe source during suspend entry.
>>>>>>>>
>>>>>>>> So saved context for CPU Source will be either dfll or PLLX
>>>>>>>>
>>>>>>> PLLP reparenting is only during suspend/entry to have it as safe
>>>>>>> source
>>>>>>> but actual CPU source it was running from before suspending is either
>>>>>>> dfll/pllx which should be the one to be restored on CPUFreq resume.
>>>>>>> Resume happens with CPU running from PLLP till it gets to the
>>>>>>> point of
>>>>>>> restoring its original source (dfll or pllx)
>>>>>> CaR should restore CPU to PLLP or PLLX, while CPUFreq driver restores
>>>>>> CPU to DFLL. Please see more comments below.
>>>>>>
>>>>>>>>>>> So i was trying to say dfll/cclk_g restore can't be done in
>>>>>>>>>>> normal way
>>>>>>>>>>> thru clk_ops save/restore context
>>>>>>>>> Let's see what happens if CPUFreq is active:
>>>>>>>>>
>>>>>>>>> 1. CPUFreq driver probe happens
>>>>>>>>>        2. CPU is reparented to PLLP
>>>>>>>>>        3. DFLL inited
>>>>>>>>>        4. CPU is reparented to DFLL
>>>>>>>>>
>>>>>>>>> 5. CPUFreq driver suspend happens
>>>>>>>>>        6. CPU is reparented to PLLP
>>>>>>>>>        7. DFLL is disabled
>>>>>>>>>
>>>>>>>>> 8. Car suspend happens
>>>>>>>>>        9. DFLL context saved
>>>>>>>>>        10. PLLP/PLLX context saved
>>>>>>>>>        11. CCLK context saved
>>>>>>>>>
>>>>>>>>> 12. Car resume happens
>>>>>>>>>        13. DFLL context restored
>>>>>>>>>        14. PLLP/PLLX context restored
>>>>>>>>>        15. CCLK context restored
>>>>>>>>>
>>>>>>>>> 16. CPUFreq driver resume happens
>>>>>>>>>        17. DFLL re-inited
>>>>>>>>>        18. CPU is reparented to DFLL
>>>>>>>> Below is the order of sequence it should be based on the order of
>>>>>>>> clk
>>>>>>>> register.
>>>>>>>>
>>>>>>>> My comments inline in this sequence.
>>>>>>>>
>>>>>>>> 1. CPUFreq driver probe happens
>>>>>>>>        2. CPU is reparented to PLLP
>>>>>>>>        3. DFLL inited
>>>>>>>>        4. CPU is reparented to DFLL
>>>>>>>>
>>>>>>>>
>>>>>>>> 5. CPUFreq driver suspend happens
>>>>>>>>        6. Save CPU source which could be either dfll or pllx
>>>>>> Please see my next comment.
>>>>>>
>>>>>>>>        7. CPU is reparented to safe known source PLLP
>>>>>>>>        8. DFLL is disabled
>>>>>>>>
>>>>>>>> 8. Car suspend happens
>>>>>>>>        9. DFLL context saved (With DFLL disabled in CPUFreq suspend,
>>>>>>>> nothing to be saved here as last freq req will always be saved).
>>>>>>>>        10. CCLK context saved (CPU clock source will be saved in
>>>>>>>> CPUFreq
>>>>>>>> driver suspend which could be either dfll or pllx)
>>>>>> That I don't understand. The CPU's clock source state should be
>>>>>> saved at
>>>>>> the moment of the CaR's suspending (i.e. CCLK policy will be set to
>>>>>> PLLP
>>>>>> or PLLX) and then CCLK state should be also restored by the CaR in
>>>>>> step 14.
>>>>> CPU clock to be saved and restored should be the source used before we
>>>>> switch it to safe PLLP for suspend/resume operation.
>>>>>
>>>>> This original source could be either PLLX or DFLL which it was using
>>>>> before we disable DFLL during CPU Freq suspend.
>>>>>
>>>>> If we save CPU clock source at moment of CAR suspending, it will be
>>>>> PLLP as we switch to safe PLLP in CPUFreq driver suspend.
>>>>>
>>>>> Infact, we dont need to restore CPU clock source to PLLP anywhere in
>>>>> resume as it comes up with PLLP source from warm boot code itself.
>>> You must always maintain proper suspend/resume encapsulation, otherwise
>>> it's a total mess. It doesn't matter that CCLK is restored to PLLP even
>>> that CPU is already running off PLLP after warmboot.
>>>
>>>>> But we need to restore CPU source to original source it was using
>>>>> before we switch to safe PLLP source for suspend operation. This
>>>>> original source could be PLLX/DFLL and this should be re-stored in
>>>>> CPUFreq resume as by this time PLLs and peripherals are restored and
>>>>> dfll is re-initialized.
>>>>>
>>>>> So saving actual CPU source before switching to intermediate safe PLLP
>>>>> in CPUFreq driver and then restoring back during CPUFreq resume should
>>>>> be good as CPUFreq resume happens right after all clocks (plls
>>>>> restore, peripherals restore, dfll resume)>>
>>>>>> CPUFreq driver should only switch CPU to PLLP and disable DFLL on
>>>>>> suspend in step 5, that's it. On resume CPUFreq driver will restore
>>>>>> CPU
>>>>>> to DFLL in step 18.
>>>>> Also I don't think we should hard-code to force CPU source to DFLL on
>>>>> CPUFreq resume.
>>>>>
>>>>> Reason is during CPU Probe when it tries to switch to dfll source, for
>>>>> some reason if dfll enable fails it sets CPU to its original source
>>>>> which will be PLLX.
>>> No!
>>>
>>> 1. CPU voltage could be too low for PLLX
>>> 2. PLLX rate can't be changed without manual reparenting CPU to
>>> intermediate clock
>>> 3. CPUFreq can't manually manage CPU voltage
>>>
>>> DFLL-restoring failure is an extreme case. CPU must be kept on a safe
>>> PLLP in that case and disable_cpufreq() must be invoked as well.
>> OK, PLLX option was also in my mind. So If we just consider sources as
>> DFLL or PLLP, then we can save source in CCLK save context and restore
>> in CCLK restore basically it will be PLLP.
>>
>> Later during CPUFreq resume we can just switch to DFLL and if DFLL
>> enable fails we will keep source as PLLP. Yes will invoke
>> disable_cpufreq as well in case of dfll enable failure for some reason.
> Sounds good!
>
>>>>> So CPU source could be either DFLL or PLLX in CPUFreq
>>>>> tegra124_cpu_switch_to_dfll
>>>>>
>>>>>>>>        11. PLLP/PLLX context saved
>>>>>>>>        12. Peripheral Clock saved
>>>>>>>>
>>>>>>>> 12. Car resume happens
>>>>>>>>        13. DFLL context restored : No DFLL context to be restored
>>>>>>>> and we
>>>>>>>> only need to reinitialize DFLL and re-initialize can't be done
>>>>>>>> here as
>>>>>>>> this is the 1st to get restored and PLL/Peripheral clocks are not
>>>>>>>> restored by this time. So we can't use clk_ops restore for DFLL
>>>>>> It looks to me that clk_core_restore_context() should just do
>>>>>> hlist_for_each_entry *_reverse*. Don't you think so?
>>>>> Thought of that but this is in core driver and is used by other
>>>>> non-tegra clock driver and not sure if that impacts for those.
>>> The reverse ordering should be correct, I think it's just a shortcoming
>>> of the CCF that need to be fixed. But will better to make a more
>>> thorough research, asking Stephen and Michael for the clarification.
>>>
>>>>> But with decision of switching CPUFreq with dfll clock enable/disable
>>>>> during CPUFreq suspend/resume, we can re-init dfll during dfll-fcpu
>>>>> driver resume and we don't need CCLK save/restore.
>>>>>
> Actually CPUFreq driver should implement suspend/resume regardless of
> CaR ability to restore DFLL or whatever, simply to properly handle
> possible clock restoring failure on resume as we just found out.
>
>>>> the way of all clocks order is good except cclk_g which has dependency
>>>> on multiple clocks.
>>> CCLK_G has a single parent at a time. What "multiple clocks" you're
>>> talking about? Please explain.
>> dependencies I am referring are dfll_ref, dfll_soc, and DVFS peripheral
>> clocks which need to be restored prior to DFLL reinit.
> Okay, but that shouldn't be a problem if clock dependencies are set up
> properly.
>
>>>> reverse list order during restore might not work as all other clocks are
>>>> in proper order no with any ref clocks for plls getting restored prior
>>>> to their clients
>>> Why? The ref clocks should be registered first and be the roots for PLLs
>>> and the rest. If it's not currently the case, then this need to be
>>> fixed. You need to ensure that each clock is modeled properly. If some
>>> child clock really depends on multiple parents, then the parents need to
>>> in the correct order or CCF need to be taught about such
>>> multi-dependencies.
>>>
>>> If some required feature is missed, then you have to implement it
>>> properly and for all, that's how things are done in upstream. Sometimes
>>> it's quite a lot of extra work that everyone are benefiting from in
>>> the end.
>>>
>>> [snip]
>> Yes, we should register ref/parents before their clients.
>>
>> cclk_g clk is registered last after all pll and peripheral clocks are
>> registers during clock init.
>>
>> dfllCPU_out clk is registered later during dfll-fcpu driver probe and
>> gets added to the clock list.
>>
>> Probably the issue seems to be not linking dfll_ref and dfll_soc
>> dependencies for dfllCPU_out thru clock list.
>>
>> clk-dfll driver during dfll_init_clks gets ref_clk and soc_clk reference
>> thru DT.
> Please try to fix all missing dependencies and orderings.

Peter,

dfllCPU_OUT is the first one to go thru restore when clk_restore_context 
traverses thru the list.

dfllCPU_OUT has dependency on DFLL_ref and DFLL_SOC but this dependency 
is unknown to clock-tree.

We can add DFLL_REF and DFLL_SOC as parents to dfllCPU_OUT during 
register so dfllCPU_OUT save/restore happens after their parents are 
restored.

But DFLL needs both of these to be restored before DFLLCPU_Out and as 
DFLL_SOC restore always happens after the REF, thinking to add DFLL_SOC 
as parent to dfllCPU_OUT so save/restore follows after their dependencies.

Please comment.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18  0:25                                                                                               ` Sowjanya Komatineni
@ 2019-07-18  1:15                                                                                                 ` Sowjanya Komatineni
  2019-07-18 16:34                                                                                                   ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-18  1:15 UTC (permalink / raw)
  To: Dmitry Osipenko, sboyd, Michael Turquette
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree


On 7/17/19 5:25 PM, Sowjanya Komatineni wrote:
>
> On 7/17/19 4:44 PM, Dmitry Osipenko wrote:
>> 18.07.2019 2:36, Sowjanya Komatineni пишет:
>>> On 7/17/19 3:48 PM, Dmitry Osipenko wrote:
>>>> 18.07.2019 0:57, Sowjanya Komatineni пишет:
>>>>> On 7/17/19 2:51 PM, Sowjanya Komatineni wrote:
>>>>>> On 7/17/19 2:30 PM, Dmitry Osipenko wrote:
>>>>>>> 17.07.2019 23:11, Sowjanya Komatineni пишет:
>>>>>>>> On 7/17/19 1:01 PM, Sowjanya Komatineni wrote:
>>>>>>>>> On 7/17/19 12:43 PM, Dmitry Osipenko wrote:
>>>>>>>>>> 17.07.2019 21:54, Sowjanya Komatineni пишет:
>>>>>>>>>>> On 7/17/19 11:51 AM, Sowjanya Komatineni wrote:
>>>>>>>>>>>> On 7/17/19 11:32 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>> 17.07.2019 20:29, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>> On 7/17/19 8:17 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>> 17.07.2019 9:36, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>> On 7/16/19 11:33 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 22:55:52 -0700
>>>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 7/16/19 10:42 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 22:25:25 -0700
>>>>>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:11 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>> В Tue, 16 Jul 2019 19:35:49 -0700
>>>>>>>>>>>>>>>>>>>>> Sowjanya Komatineni <skomatineni@nvidia.com> пишет:
>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 7:18 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 3:06 PM, Sowjanya Komatineni wrote:
>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 3:00 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:35, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 2:21 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 17.07.2019 0:12, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 1:47 PM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 22:26, Sowjanya Komatineni пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:43 AM, Dmitry Osipenko wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:30, Sowjanya Komatineni 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 11:25 AM, Dmitry Osipenko 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 21:19, Sowjanya Komatineni 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 9:50 AM, Sowjanya Komatineni
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/16/19 8:00 AM, Dmitry Osipenko 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 16.07.2019 11:06, Peter De Schrijver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> пишет:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 16, 2019 at 03:24:26PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +0800,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Joseph
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lo wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add to CPUFreq driver...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The other thing that also need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attention is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that T124 CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicitly relies on DFLL driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> first, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> icky.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Should I add check for successful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register explicitly in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver probe and defer till
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably you should use the "device
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> links".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> See
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2] for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.kernel.org/doc/html/latest/driver-api/device_link.html 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Return EPROBE_DEFER instead of 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> EINVAL if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device_link_add() fails.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use of_find_device_by_node() to get 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device, see [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will go thru and add...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looks like I initially confused this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> orphaned clock.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm now seeing that the DFLL driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registers the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock and then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clk_get(dfll) should be returning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> EPROBE_DEFER
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> until
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probed, hence everything should be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as-is and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there is no real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the 'device link'. Sorry for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> confusion!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I didn't follow the mail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. Just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As you know it, the DFLL clock is 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock sources and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrated with DVFS control logic
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator. We will not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to other clock sources once we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switched to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL. Because the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been regulated by the DFLL HW 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DVFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (CVB or OPP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the driver.). We shouldn't 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reparent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other sources with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unknew
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq/volt pair. That's not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> guaranteed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. We
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow switching to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open-loop mode but different 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sources.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay, then the CPUFreq driver will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate before switching to PLLP in 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> order to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper CPU voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP freq is safe to work for any CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> voltage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to enforce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL freq to PLLP rate before changing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to PLLP during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, please ignore my above comment.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend, need to change
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK_G source to PLLP when dfll is in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> closed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode first and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll need to be set to open loop.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Okay.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And I don't exactly understand 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> why we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to PLLP in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver. Just keep it on CL-DVFS mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 entry, the dfll suspend 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the open-loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode. That's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all. The sc7-entryfirmware will 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the rest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn off
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU power.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In SC7 resume, the warmboot code 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sequence to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turn on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regulator and power up the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cluster. And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it on PLL_P.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resuming to the kernel, we re-init
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy (CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs on DFLL open-loop mode) and 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moving to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close-loop mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL is re-inited after switching
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CCLK to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> early clocks-state restoring by CaR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hence
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks in the CaR driver, it is much
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nicer to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proper suspend-resume sequencing of 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> device
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drivers. In this case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver is the driver that enables DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switches
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU to that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source, which means that this 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> driver is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be responsible for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> management of the DFLL's state 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend/resume process. If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPUFreq driver disables DFLL during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-enables it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resume, then looks like the CaR driver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hacks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL are not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The DFLL part looks good to me. BTW,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch subject to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suspend-resume support" seems more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To clarify this, the sequences for 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows (assuming
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required DFLL hw configuration has 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> done)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch to DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Save current parent and frequency
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enable DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Change cclk_g parent to DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For OVR regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Change PWM output pin from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tristate to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) Enable DFLL PWM output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For I2C regulator:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) Enable DFLL I2C output
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6) Program DFLL to closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Switch away from DFLL:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0) Change cclk_g parent to PLLP so
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CPU
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> frequency is ok for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> vdd_cpu voltage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Program DFLL to open loop mode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I see during switch away from DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (suspend),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parent is not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> changed to PLLP before changing dfll to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loop
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will add this ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The CPUFreq driver switches parent to 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probe, similar
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be done on suspend.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm also wondering if it's always safe to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> switch to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP in the probe.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If CPU is running on a lower freq than
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP, then
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate intermediate parent should be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> selected.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU parents are PLL_X, PLL_P, and dfll. 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLL_X
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs at higher
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so switching to PLL_P during CPUFreq probe
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prior to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfll clock enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be safe.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AFAIK, PLLX could run at ~200MHz. There is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> divided output of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which CCLKG supports, the PLLP_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Probably, realistically, CPU is always 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> off a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fast PLLX during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> boot, but I'm wondering what may happen on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KEXEC. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> guess ideally CPUFreq driver should also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'shutdown' callback to teardown DFLL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on a reboot, but likely that there are 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clock-related problems as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> well that may break KEXEC and thus it is 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important at the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moment.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [snip]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During bootup CPUG sources from PLL_X. By 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLL_P
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> above I meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLL_P_OUT4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As per clock policies, PLL_X is always used
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for high
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 800Mhz
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and for low frequency it will be sourced 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alright, then please don't forget to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pre-initialize
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate to a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reasonable value using 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tegra_clk_init_table or
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> assigned-clocks.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PLLP_OUT4 rate update is not needed as it is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> safe to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> run at
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 408Mhz because it is below fmax @ Vmin
>>>>>>>>>>>>>>>>>>>>>>>>>>> So even 204MHz CVB entries are having the same
>>>>>>>>>>>>>>>>>>>>>>>>>>> voltage as
>>>>>>>>>>>>>>>>>>>>>>>>>>> 408MHz, correct? It's not instantly obvious 
>>>>>>>>>>>>>>>>>>>>>>>>>>> to me
>>>>>>>>>>>>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>>>>>>>>>>>> DFLL driver's code where the fmax @ Vmin is
>>>>>>>>>>>>>>>>>>>>>>>>>>> defined,
>>>>>>>>>>>>>>>>>>>>>>>>>>> I see
>>>>>>>>>>>>>>>>>>>>>>>>>>> that there is the min_millivolts
>>>>>>>>>>>>>>>>>>>>>>>>>>> and frequency entries starting from 204MHZ 
>>>>>>>>>>>>>>>>>>>>>>>>>>> defined
>>>>>>>>>>>>>>>>>>>>>>>>>>> per-table.
>>>>>>>>>>>>>>>>>>>>>>>>>> Yes at Vmin CPU Fmax is ~800Mhz. So anything 
>>>>>>>>>>>>>>>>>>>>>>>>>> below
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>> work at Vmin voltage and PLLP max is 408Mhz.
>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the clarification. It would be good
>>>>>>>>>>>>>>>>>>>>>>>>> to have
>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> commented
>>>>>>>>>>>>>>>>>>>>>>>>> in the code as well.
>>>>>>>>>>>>>>>>>>>>>>>> OK, Will add...
>>>>>>>>>>>>>>>>>>>>>>> Regarding, adding suspend/resume to CPUFreq, 
>>>>>>>>>>>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>>>> happens very early even before disabling non-boot
>>>>>>>>>>>>>>>>>>>>>>> CPUs and
>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>> need to export clock driver APIs to CPUFreq.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Was thinking of below way of implementing this...
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Clock DFLL driver Suspend:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                  - Save CPU clock policy 
>>>>>>>>>>>>>>>>>>>>>>> registers, and
>>>>>>>>>>>>>>>>>>>>>>> Perform
>>>>>>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>>>>>>> suspend which sets in open loop mode
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> CPU Freq driver Suspend: does nothing
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Clock DFLL driver Resume:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                  - Re-init DFLL, Set in 
>>>>>>>>>>>>>>>>>>>>>>> Open-Loop mode,
>>>>>>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>>>>>> Clock policy registers which actually sets 
>>>>>>>>>>>>>>>>>>>>>>> source to
>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>> along
>>>>>>>>>>>>>>>>>>>>>>> with other CPU Policy register restore.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> CPU Freq driver Resume:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                  - do clk_prepare_enable which 
>>>>>>>>>>>>>>>>>>>>>>> acutally
>>>>>>>>>>>>>>>>>>>>>>> sets
>>>>>>>>>>>>>>>>>>>>>>> DFLL in
>>>>>>>>>>>>>>>>>>>>>>> Closed loop mode
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Adding one more note: Switching CPU Clock to PLLP
>>>>>>>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>>>>>>>> as CPU CLock can be from dfll in open-loop mode as
>>>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>>>>>>>> disabled anywhere throught the suspend/resume path
>>>>>>>>>>>>>>>>>>>>>>> and SC7
>>>>>>>>>>>>>>>>>>>>>>> entry
>>>>>>>>>>>>>>>>>>>>>>> FW and Warm boot code will switch CPU source to 
>>>>>>>>>>>>>>>>>>>>>>> PLLP.
>>>>>>>>>>>>>>>>>>>>> Since CPU resumes on PLLP, it will be cleaner to 
>>>>>>>>>>>>>>>>>>>>> suspend
>>>>>>>>>>>>>>>>>>>>> it on
>>>>>>>>>>>>>>>>>>>>> PLLP as well. And besides, seems that currently
>>>>>>>>>>>>>>>>>>>>> disabling
>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>> clock will disable DFLL completely and then you'd
>>>>>>>>>>>>>>>>>>>>> want to
>>>>>>>>>>>>>>>>>>>>> re-init
>>>>>>>>>>>>>>>>>>>>> the DFLL on resume any ways. So better to just 
>>>>>>>>>>>>>>>>>>>>> disable
>>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>>> completely on suspend, which should happen on
>>>>>>>>>>>>>>>>>>>>> clk_disable(dfll).
>>>>>>>>>>>>>>>>>>>> Will switch to PLLP during CPUFreq suspend. With
>>>>>>>>>>>>>>>>>>>> decision of
>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>> clk_disable during suspend, its mandatory to switch to
>>>>>>>>>>>>>>>>>>>> PLLP as
>>>>>>>>>>>>>>>>>>>> DFLL
>>>>>>>>>>>>>>>>>>>> is completely disabled.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> My earlier concern was on restoring CPU policy as we
>>>>>>>>>>>>>>>>>>>> can't do
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> from CPUFreq driver and need export from clock driver.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Clear now and will do CPU clock policy restore in 
>>>>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>>>> re-init.
>>>>>>>>>>>>>>>>>>> Why the policy can't be saved/restored by the CaR 
>>>>>>>>>>>>>>>>>>> driver
>>>>>>>>>>>>>>>>>>> as a
>>>>>>>>>>>>>>>>>>> context of any other clock?
>>>>>>>>>>>>>>>>>> restoring cpu clock policy involves programming 
>>>>>>>>>>>>>>>>>> source and
>>>>>>>>>>>>>>>>>> super_cclkg_divider.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> cclk_g is registered as clk_super_mux and it doesn't use
>>>>>>>>>>>>>>>>>> frac_div ops
>>>>>>>>>>>>>>>>>> to do save/restore its divider.
>>>>>>>>>>>>>>>>> That can be changed of course and I guess it also could
>>>>>>>>>>>>>>>>> be as
>>>>>>>>>>>>>>>>> simple as
>>>>>>>>>>>>>>>>> saving and restoring of two raw u32 values of the
>>>>>>>>>>>>>>>>> policy/divider
>>>>>>>>>>>>>>>>> registers.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Also, during clock context we cant restore cclk_g as 
>>>>>>>>>>>>>>>>>> cclk_g
>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>> will be dfll and dfll will not be resumed/re-initialized
>>>>>>>>>>>>>>>>>> by the
>>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>> clk_super_mux save/restore happens.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> we can't use save/restore context for dfll clk_ops 
>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>> dfllCPU_out parent to CCLK_G is first in the clock 
>>>>>>>>>>>>>>>>>> tree and
>>>>>>>>>>>>>>>>>> dfll_ref
>>>>>>>>>>>>>>>>>> and dfll_soc peripheral clocks are not restored by the
>>>>>>>>>>>>>>>>>> time dfll
>>>>>>>>>>>>>>>>>> restore happens. Also dfll peripheral clock enables need
>>>>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>>>> restored before dfll restore happens which involves
>>>>>>>>>>>>>>>>>> programming
>>>>>>>>>>>>>>>>>> dfll
>>>>>>>>>>>>>>>>>> controller for re-initialization.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So dfll resume/re-init is done in clk-tegra210 at end of
>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>> clocks
>>>>>>>>>>>>>>>>>> restore in V5 series but instead of in clk-tegra210
>>>>>>>>>>>>>>>>>> driver I
>>>>>>>>>>>>>>>>>> moved
>>>>>>>>>>>>>>>>>> now to dfll-fcpu driver pm_ops as all dfll dependencies
>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>> restored thru clk_restore_context by then. This will 
>>>>>>>>>>>>>>>>>> be in
>>>>>>>>>>>>>>>>>> V6.
>>>>>>>>>>>>>>>>> Since DFLL is now guaranteed to be disabled across CaR
>>>>>>>>>>>>>>>>> suspend/resume
>>>>>>>>>>>>>>>>> (hence it has nothing to do in regards to CCLK) and given
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> PLLs
>>>>>>>>>>>>>>>>> state is restored before the rest of the clocks, I don't
>>>>>>>>>>>>>>>>> see why
>>>>>>>>>>>>>>>>> not to
>>>>>>>>>>>>>>>>> implement CCLK save/restore in a generic fasion. CPU 
>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>> wull be
>>>>>>>>>>>>>>>>> restored to either PLLP or PLLX (if CPUFreq driver is
>>>>>>>>>>>>>>>>> disabled).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CCLK_G save/restore should happen in clk_super_mux ops
>>>>>>>>>>>>>>>> save/context and
>>>>>>>>>>>>>>>> clk_super_mux save/restore happens very early as cclk_g is
>>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>>> clock tree and save/restore traverses through the tree
>>>>>>>>>>>>>>>> top-bottom
>>>>>>>>>>>>>>>> order.
>>>>>>>>>>>>>>> If CCLK_G is restored before the PLLs, then just change the
>>>>>>>>>>>>>>> clocks
>>>>>>>>>>>>>>> order
>>>>>>>>>>>>>>> such that it won't happen.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I dont think we can change clocks order for CCLK_G.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> During bootup, cclk_g is registered after all pll's and
>>>>>>>>>>>>>> peripheral
>>>>>>>>>>>>>> clocks which is the way we wanted, So cclk_g will be the 
>>>>>>>>>>>>>> first
>>>>>>>>>>>>>> one in
>>>>>>>>>>>>>> the clk list as clk_register adds new clock first in the 
>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When clk_save_context and clk_restore_context APIs iterates
>>>>>>>>>>>>>> over the
>>>>>>>>>>>>>> list, cclk_g is the first
>>>>>>>>>>>>> Looking at clk_core_restore_context(), I see that it walks up
>>>>>>>>>>>>> CLKs
>>>>>>>>>>>>> list
>>>>>>>>>>>>> from parent to children, hence I don't understand how it can
>>>>>>>>>>>>> ever
>>>>>>>>>>>>> happen
>>>>>>>>>>>>> that CCLK will be restored before the parent. The clocks
>>>>>>>>>>>>> registration
>>>>>>>>>>>>> order doesn't matter at all in that case.
>>>>>>>>>>>> yes from parent to children and dfllCPU_out is the top in the
>>>>>>>>>>>> list and
>>>>>>>>>>>> its child is cclk_g.
>>>>>>>>>>>>
>>>>>>>>>>>> the way clocks are registered is the order I see in the clock
>>>>>>>>>>>> list and
>>>>>>>>>>>> looking into clk_register API it adds new node first in the 
>>>>>>>>>>>> list.
>>>>>>>>>>>>
>>>>>>>>>>> cclkg_g & dfll register happens after all plls and peripheral
>>>>>>>>>>> clocks as
>>>>>>>>>>> it need ref, soc and peripheral clocks to be enabled.
>>>>>>>>>>>> So they are the last to get registered and so becomes first in
>>>>>>>>>>>> the
>>>>>>>>>>>> list.
>>>>>>>>>>>>
>>>>>>>>>>>> During save/restore context, it traverses thru this list and
>>>>>>>>>>>> first in
>>>>>>>>>>>> the list is dfllcpu_OUT (parent) and its child (cclk_g)
>>>>>>>>>>>>
>>>>>>>>>>>> saving should not be an issue at all but we cant restore
>>>>>>>>>>>> cclk_g/dfll
>>>>>>>>>>>> in normal way thru clk_ops restore as plls and peripherals
>>>>>>>>>>>> restore
>>>>>>>>>>>> doesn't happen by that time.
>>>>>>>>>>>>
>>>>>>>>>>> I was referring to clk_restore_context where it iterates thru
>>>>>>>>>>> root list
>>>>>>>>>>> and for each core from the root list clk_core_restore does
>>>>>>>>>>> restore of
>>>>>>>>>>> parent and children.
>>>>>>>>>>>
>>>>>>>>>>> dfllCPU_Out gets first in the list and its child is cclk_g
>>>>>>>>>>>
>>>>>>>>>>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/clk/clk.c#L1105 
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> What list you're talking about? clk_summary? It shows current
>>>>>>>>>> *active*
>>>>>>>>>> clocks configuration, if you'll try to disable CPUFreq driver 
>>>>>>>>>> then
>>>>>>>>>> the
>>>>>>>>>> parent of CCLK_G should be PLLX. Similarly when CPU is
>>>>>>>>>> reparented to
>>>>>>>>>> PLLP on driver's suspend, then PLLP is the parent.
>>>>>>>>>>
>>>>>>>>>>>>>>>> DFLL enable thru CPUFreq resume happens after all
>>>>>>>>>>>>>>>> clk_restore_context
>>>>>>>>>>>>>>>> happens. So during clk_restore_context, dfll re-init 
>>>>>>>>>>>>>>>> doesnt
>>>>>>>>>>>>>>>> happen
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> doing cpu clock policy restore during super_mux clk_ops 
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> crash as
>>>>>>>>>>>>>>>> DFLL is not initialized and its clock is not enabled 
>>>>>>>>>>>>>>>> but CPU
>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>> restore sets source to DFLL if we restore during
>>>>>>>>>>>>>>>> super_clk_mux
>>>>>>>>>>>>>>> If CPU was suspended on PLLP, then it will be restored on
>>>>>>>>>>>>>>> PLLP by
>>>>>>>>>>>>>>> CaR. I
>>>>>>>>>>>>>>> don't understand what DFLL has to do with the CCLK in that
>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>> the clocks restore.
>>>>>>>>>>>>>> My above comment is in reference to your request of doing
>>>>>>>>>>>>>> save/restore
>>>>>>>>>>>>>> for cclk_g in normal fashion thru save/restore context. 
>>>>>>>>>>>>>> Because
>>>>>>>>>>>>>> of the
>>>>>>>>>>>>>> clk order I mentioned above, we cclk_g will be the first 
>>>>>>>>>>>>>> one to
>>>>>>>>>>>>>> go thru
>>>>>>>>>>>>>> save/context.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> During save_context of cclk_g, source can be from PLLX, 
>>>>>>>>>>>>>> dfll.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Issue will be when we do restore during 
>>>>>>>>>>>>>> clk_restore_context of
>>>>>>>>>>>>>> cclk_g as
>>>>>>>>>>>>>> by that time PLLX/dfll will not be restored.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Seems we already agreed that DFLL will be disabled by the
>>>>>>>>>>>>> CPUFreq
>>>>>>>>>>>>> driver
>>>>>>>>>>>>> on suspend. Hence CCLK can't be from DFLL if CPU is
>>>>>>>>>>>>> reparented to
>>>>>>>>>>>>> PLLP
>>>>>>>>>>>>> on CPUFreq driver's suspend, otherwise CPU keeps running 
>>>>>>>>>>>>> from a
>>>>>>>>>>>>> boot-state PLLX if CPUFreq driver is disabled.
>>>>>>>>>>>> Yes suspend should not be an issue but issue will be during
>>>>>>>>>>>> resume
>>>>>>>>>>>> where if we do cclk_g restore in normal way thru
>>>>>>>>>>>> clk_restore_context,
>>>>>>>>>>>> cclk_g restore happens very early as dfllCPU out is the first
>>>>>>>>>>>> one that
>>>>>>>>>>>> goes thru restore context and plls/peripherals are not 
>>>>>>>>>>>> resumed by
>>>>>>>>>>>> then.
>>>>>>>>>>>>
>>>>>>>>>>>> CPU runs from PLLX if dfll clock enable fails during boot. So
>>>>>>>>>>>> when it
>>>>>>>>>>>> gets to suspend, we save CPU running clock source as either
>>>>>>>>>>>> PLLX or
>>>>>>>>>>>> DFLL and then we switch to PLLP.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On resume, CPU runs from PLLP by warm boot code and we need to
>>>>>>>>>>>> restore
>>>>>>>>>>>> back its source to the one it was using from saved source 
>>>>>>>>>>>> context
>>>>>>>>>>>> (which can be either PLLX or DFLL)
>>>>>>>>>>>>
>>>>>>>>>>>> So PLLs & DFLL resume need to happen before CCLKG 
>>>>>>>>>>>> restore/resume.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> With all above discussions, we do DFLL disable in CPUFreq
>>>>>>>>>>>> driver on
>>>>>>>>>>>> suspend and on CPUFreq resume we enable DFLL back and 
>>>>>>>>>>>> restore CPU
>>>>>>>>>>>> clock source it was using during suspend (which will be either
>>>>>>>>>>>> PLLX if
>>>>>>>>>>>> dfll enable fails during probe or it will be using DFLL).
>>>>>>>>>> During suspend CPU's parent shall be PLLP and not DFLL (note 
>>>>>>>>>> that
>>>>>>>>>> it is
>>>>>>>>>> disabled) after reparenting to PLLP by the CPUFreq driver.
>>>>>>>>>>
>>>>>>>>> CPU source context should be saved before switching to safe
>>>>>>>>> source of
>>>>>>>>> PLLP as on resume we need to restore back to source it was using
>>>>>>>>> before we switch to safe source during suspend entry.
>>>>>>>>>
>>>>>>>>> So saved context for CPU Source will be either dfll or PLLX
>>>>>>>>>
>>>>>>>> PLLP reparenting is only during suspend/entry to have it as safe
>>>>>>>> source
>>>>>>>> but actual CPU source it was running from before suspending is 
>>>>>>>> either
>>>>>>>> dfll/pllx which should be the one to be restored on CPUFreq 
>>>>>>>> resume.
>>>>>>>> Resume happens with CPU running from PLLP till it gets to the
>>>>>>>> point of
>>>>>>>> restoring its original source (dfll or pllx)
>>>>>>> CaR should restore CPU to PLLP or PLLX, while CPUFreq driver 
>>>>>>> restores
>>>>>>> CPU to DFLL. Please see more comments below.
>>>>>>>
>>>>>>>>>>>> So i was trying to say dfll/cclk_g restore can't be done in
>>>>>>>>>>>> normal way
>>>>>>>>>>>> thru clk_ops save/restore context
>>>>>>>>>> Let's see what happens if CPUFreq is active:
>>>>>>>>>>
>>>>>>>>>> 1. CPUFreq driver probe happens
>>>>>>>>>>        2. CPU is reparented to PLLP
>>>>>>>>>>        3. DFLL inited
>>>>>>>>>>        4. CPU is reparented to DFLL
>>>>>>>>>>
>>>>>>>>>> 5. CPUFreq driver suspend happens
>>>>>>>>>>        6. CPU is reparented to PLLP
>>>>>>>>>>        7. DFLL is disabled
>>>>>>>>>>
>>>>>>>>>> 8. Car suspend happens
>>>>>>>>>>        9. DFLL context saved
>>>>>>>>>>        10. PLLP/PLLX context saved
>>>>>>>>>>        11. CCLK context saved
>>>>>>>>>>
>>>>>>>>>> 12. Car resume happens
>>>>>>>>>>        13. DFLL context restored
>>>>>>>>>>        14. PLLP/PLLX context restored
>>>>>>>>>>        15. CCLK context restored
>>>>>>>>>>
>>>>>>>>>> 16. CPUFreq driver resume happens
>>>>>>>>>>        17. DFLL re-inited
>>>>>>>>>>        18. CPU is reparented to DFLL
>>>>>>>>> Below is the order of sequence it should be based on the order of
>>>>>>>>> clk
>>>>>>>>> register.
>>>>>>>>>
>>>>>>>>> My comments inline in this sequence.
>>>>>>>>>
>>>>>>>>> 1. CPUFreq driver probe happens
>>>>>>>>>        2. CPU is reparented to PLLP
>>>>>>>>>        3. DFLL inited
>>>>>>>>>        4. CPU is reparented to DFLL
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 5. CPUFreq driver suspend happens
>>>>>>>>>        6. Save CPU source which could be either dfll or pllx
>>>>>>> Please see my next comment.
>>>>>>>
>>>>>>>>>        7. CPU is reparented to safe known source PLLP
>>>>>>>>>        8. DFLL is disabled
>>>>>>>>>
>>>>>>>>> 8. Car suspend happens
>>>>>>>>>        9. DFLL context saved (With DFLL disabled in CPUFreq 
>>>>>>>>> suspend,
>>>>>>>>> nothing to be saved here as last freq req will always be saved).
>>>>>>>>>        10. CCLK context saved (CPU clock source will be saved in
>>>>>>>>> CPUFreq
>>>>>>>>> driver suspend which could be either dfll or pllx)
>>>>>>> That I don't understand. The CPU's clock source state should be
>>>>>>> saved at
>>>>>>> the moment of the CaR's suspending (i.e. CCLK policy will be set to
>>>>>>> PLLP
>>>>>>> or PLLX) and then CCLK state should be also restored by the CaR in
>>>>>>> step 14.
>>>>>> CPU clock to be saved and restored should be the source used 
>>>>>> before we
>>>>>> switch it to safe PLLP for suspend/resume operation.
>>>>>>
>>>>>> This original source could be either PLLX or DFLL which it was using
>>>>>> before we disable DFLL during CPU Freq suspend.
>>>>>>
>>>>>> If we save CPU clock source at moment of CAR suspending, it will be
>>>>>> PLLP as we switch to safe PLLP in CPUFreq driver suspend.
>>>>>>
>>>>>> Infact, we dont need to restore CPU clock source to PLLP anywhere in
>>>>>> resume as it comes up with PLLP source from warm boot code itself.
>>>> You must always maintain proper suspend/resume encapsulation, 
>>>> otherwise
>>>> it's a total mess. It doesn't matter that CCLK is restored to PLLP 
>>>> even
>>>> that CPU is already running off PLLP after warmboot.
>>>>
>>>>>> But we need to restore CPU source to original source it was using
>>>>>> before we switch to safe PLLP source for suspend operation. This
>>>>>> original source could be PLLX/DFLL and this should be re-stored in
>>>>>> CPUFreq resume as by this time PLLs and peripherals are restored and
>>>>>> dfll is re-initialized.
>>>>>>
>>>>>> So saving actual CPU source before switching to intermediate safe 
>>>>>> PLLP
>>>>>> in CPUFreq driver and then restoring back during CPUFreq resume 
>>>>>> should
>>>>>> be good as CPUFreq resume happens right after all clocks (plls
>>>>>> restore, peripherals restore, dfll resume)>>
>>>>>>> CPUFreq driver should only switch CPU to PLLP and disable DFLL on
>>>>>>> suspend in step 5, that's it. On resume CPUFreq driver will restore
>>>>>>> CPU
>>>>>>> to DFLL in step 18.
>>>>>> Also I don't think we should hard-code to force CPU source to 
>>>>>> DFLL on
>>>>>> CPUFreq resume.
>>>>>>
>>>>>> Reason is during CPU Probe when it tries to switch to dfll 
>>>>>> source, for
>>>>>> some reason if dfll enable fails it sets CPU to its original source
>>>>>> which will be PLLX.
>>>> No!
>>>>
>>>> 1. CPU voltage could be too low for PLLX
>>>> 2. PLLX rate can't be changed without manual reparenting CPU to
>>>> intermediate clock
>>>> 3. CPUFreq can't manually manage CPU voltage
>>>>
>>>> DFLL-restoring failure is an extreme case. CPU must be kept on a safe
>>>> PLLP in that case and disable_cpufreq() must be invoked as well.
>>> OK, PLLX option was also in my mind. So If we just consider sources as
>>> DFLL or PLLP, then we can save source in CCLK save context and restore
>>> in CCLK restore basically it will be PLLP.
>>>
>>> Later during CPUFreq resume we can just switch to DFLL and if DFLL
>>> enable fails we will keep source as PLLP. Yes will invoke
>>> disable_cpufreq as well in case of dfll enable failure for some reason.
>> Sounds good!
>>
>>>>>> So CPU source could be either DFLL or PLLX in CPUFreq
>>>>>> tegra124_cpu_switch_to_dfll
>>>>>>
>>>>>>>>>        11. PLLP/PLLX context saved
>>>>>>>>>        12. Peripheral Clock saved
>>>>>>>>>
>>>>>>>>> 12. Car resume happens
>>>>>>>>>        13. DFLL context restored : No DFLL context to be restored
>>>>>>>>> and we
>>>>>>>>> only need to reinitialize DFLL and re-initialize can't be done
>>>>>>>>> here as
>>>>>>>>> this is the 1st to get restored and PLL/Peripheral clocks are not
>>>>>>>>> restored by this time. So we can't use clk_ops restore for DFLL
>>>>>>> It looks to me that clk_core_restore_context() should just do
>>>>>>> hlist_for_each_entry *_reverse*. Don't you think so?
>>>>>> Thought of that but this is in core driver and is used by other
>>>>>> non-tegra clock driver and not sure if that impacts for those.
>>>> The reverse ordering should be correct, I think it's just a 
>>>> shortcoming
>>>> of the CCF that need to be fixed. But will better to make a more
>>>> thorough research, asking Stephen and Michael for the clarification.
>>>>
>>>>>> But with decision of switching CPUFreq with dfll clock 
>>>>>> enable/disable
>>>>>> during CPUFreq suspend/resume, we can re-init dfll during dfll-fcpu
>>>>>> driver resume and we don't need CCLK save/restore.
>>>>>>
>> Actually CPUFreq driver should implement suspend/resume regardless of
>> CaR ability to restore DFLL or whatever, simply to properly handle
>> possible clock restoring failure on resume as we just found out.
>>
>>>>> the way of all clocks order is good except cclk_g which has 
>>>>> dependency
>>>>> on multiple clocks.
>>>> CCLK_G has a single parent at a time. What "multiple clocks" you're
>>>> talking about? Please explain.
>>> dependencies I am referring are dfll_ref, dfll_soc, and DVFS peripheral
>>> clocks which need to be restored prior to DFLL reinit.
>> Okay, but that shouldn't be a problem if clock dependencies are set up
>> properly.
>>
>>>>> reverse list order during restore might not work as all other 
>>>>> clocks are
>>>>> in proper order no with any ref clocks for plls getting restored 
>>>>> prior
>>>>> to their clients
>>>> Why? The ref clocks should be registered first and be the roots for 
>>>> PLLs
>>>> and the rest. If it's not currently the case, then this need to be
>>>> fixed. You need to ensure that each clock is modeled properly. If some
>>>> child clock really depends on multiple parents, then the parents 
>>>> need to
>>>> in the correct order or CCF need to be taught about such
>>>> multi-dependencies.
>>>>
>>>> If some required feature is missed, then you have to implement it
>>>> properly and for all, that's how things are done in upstream. 
>>>> Sometimes
>>>> it's quite a lot of extra work that everyone are benefiting from in
>>>> the end.
>>>>
>>>> [snip]
>>> Yes, we should register ref/parents before their clients.
>>>
>>> cclk_g clk is registered last after all pll and peripheral clocks are
>>> registers during clock init.
>>>
>>> dfllCPU_out clk is registered later during dfll-fcpu driver probe and
>>> gets added to the clock list.
>>>
>>> Probably the issue seems to be not linking dfll_ref and dfll_soc
>>> dependencies for dfllCPU_out thru clock list.
>>>
>>> clk-dfll driver during dfll_init_clks gets ref_clk and soc_clk 
>>> reference
>>> thru DT.
>> Please try to fix all missing dependencies and orderings.
>
> Peter,
>
> dfllCPU_OUT is the first one to go thru restore when 
> clk_restore_context traverses thru the list.
>
> dfllCPU_OUT has dependency on DFLL_ref and DFLL_SOC but this 
> dependency is unknown to clock-tree.
>
> We can add DFLL_REF and DFLL_SOC as parents to dfllCPU_OUT during 
> register so dfllCPU_OUT save/restore happens after their parents are 
> restored.
>
> But DFLL needs both of these to be restored before DFLLCPU_Out and as 
> DFLL_SOC restore always happens after the REF, thinking to add 
> DFLL_SOC as parent to dfllCPU_OUT so save/restore follows after their 
> dependencies.
>
> Please comment.
>
Did quick try and I see by adding dfll-soc as parent to dfllCPU_OUT, its 
in proper order after all its dependencies.

Can now add dfll save/restore to do dfll reinit during restore..


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18  1:15                                                                                                 ` Sowjanya Komatineni
@ 2019-07-18 16:34                                                                                                   ` Dmitry Osipenko
  2019-07-18 17:22                                                                                                     ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-18 16:34 UTC (permalink / raw)
  To: Sowjanya Komatineni, sboyd, Michael Turquette
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree

18.07.2019 4:15, Sowjanya Komatineni пишет:

[snip]

>>> Please try to fix all missing dependencies and orderings.
>>
>> Peter,
>>
>> dfllCPU_OUT is the first one to go thru restore when
>> clk_restore_context traverses thru the list.
>>
>> dfllCPU_OUT has dependency on DFLL_ref and DFLL_SOC but this
>> dependency is unknown to clock-tree.
>>
>> We can add DFLL_REF and DFLL_SOC as parents to dfllCPU_OUT during
>> register so dfllCPU_OUT save/restore happens after their parents are
>> restored.
>>
>> But DFLL needs both of these to be restored before DFLLCPU_Out and as
>> DFLL_SOC restore always happens after the REF, thinking to add
>> DFLL_SOC as parent to dfllCPU_OUT so save/restore follows after their
>> dependencies.
>>
>> Please comment.
>>
> Did quick try and I see by adding dfll-soc as parent to dfllCPU_OUT, its
> in proper order after all its dependencies.
> 
> Can now add dfll save/restore to do dfll reinit during restore..
> 

If dfllCPU_OUT can work properly with dfll-soc being disabled, then this
kind of dependency isn't very correct and just papers over the real
problem, which is that there should be a way for CCF to specify multiple
dependencies for the clock or the reverse ordering should be used for
the restoring.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18 16:34                                                                                                   ` Dmitry Osipenko
@ 2019-07-18 17:22                                                                                                     ` Sowjanya Komatineni
  2019-07-18 17:41                                                                                                       ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-18 17:22 UTC (permalink / raw)
  To: Dmitry Osipenko, sboyd, Michael Turquette
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree


On 7/18/19 9:34 AM, Dmitry Osipenko wrote:
> 18.07.2019 4:15, Sowjanya Komatineni пишет:
>
> [snip]
>
>>>> Please try to fix all missing dependencies and orderings.
>>> Peter,
>>>
>>> dfllCPU_OUT is the first one to go thru restore when
>>> clk_restore_context traverses thru the list.
>>>
>>> dfllCPU_OUT has dependency on DFLL_ref and DFLL_SOC but this
>>> dependency is unknown to clock-tree.
>>>
>>> We can add DFLL_REF and DFLL_SOC as parents to dfllCPU_OUT during
>>> register so dfllCPU_OUT save/restore happens after their parents are
>>> restored.
>>>
>>> But DFLL needs both of these to be restored before DFLLCPU_Out and as
>>> DFLL_SOC restore always happens after the REF, thinking to add
>>> DFLL_SOC as parent to dfllCPU_OUT so save/restore follows after their
>>> dependencies.
>>>
>>> Please comment.
>>>
>> Did quick try and I see by adding dfll-soc as parent to dfllCPU_OUT, its
>> in proper order after all its dependencies.
>>
>> Can now add dfll save/restore to do dfll reinit during restore..
>>
> If dfllCPU_OUT can work properly with dfll-soc being disabled, then this
> kind of dependency isn't very correct and just papers over the real
> problem, which is that there should be a way for CCF to specify multiple
> dependencies for the clock or the reverse ordering should be used for
> the restoring.

dfll will not work without dfll-soc enabled.

CLDVFS control logic is split into 2 clock domains. dvfs_ref_clk and 
dvfs_soc_clk.

Majority of the control logic is clocked from dvfs_soc_clk for 
interfacing control registers.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18 17:22                                                                                                     ` Sowjanya Komatineni
@ 2019-07-18 17:41                                                                                                       ` Sowjanya Komatineni
  2019-07-18 18:29                                                                                                         ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-18 17:41 UTC (permalink / raw)
  To: Dmitry Osipenko, sboyd, Michael Turquette
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree


On 7/18/19 10:22 AM, Sowjanya Komatineni wrote:
>
> On 7/18/19 9:34 AM, Dmitry Osipenko wrote:
>> 18.07.2019 4:15, Sowjanya Komatineni пишет:
>>
>> [snip]
>>
>>>>> Please try to fix all missing dependencies and orderings.
>>>> Peter,
>>>>
>>>> dfllCPU_OUT is the first one to go thru restore when
>>>> clk_restore_context traverses thru the list.
>>>>
>>>> dfllCPU_OUT has dependency on DFLL_ref and DFLL_SOC but this
>>>> dependency is unknown to clock-tree.
>>>>
>>>> We can add DFLL_REF and DFLL_SOC as parents to dfllCPU_OUT during
>>>> register so dfllCPU_OUT save/restore happens after their parents are
>>>> restored.
>>>>
>>>> But DFLL needs both of these to be restored before DFLLCPU_Out and as
>>>> DFLL_SOC restore always happens after the REF, thinking to add
>>>> DFLL_SOC as parent to dfllCPU_OUT so save/restore follows after their
>>>> dependencies.
>>>>
>>>> Please comment.
>>>>
>>> Did quick try and I see by adding dfll-soc as parent to dfllCPU_OUT, 
>>> its
>>> in proper order after all its dependencies.
>>>
>>> Can now add dfll save/restore to do dfll reinit during restore..
>>>
>> If dfllCPU_OUT can work properly with dfll-soc being disabled, then this
>> kind of dependency isn't very correct and just papers over the real
>> problem, which is that there should be a way for CCF to specify multiple
>> dependencies for the clock or the reverse ordering should be used for
>> the restoring.
>
> dfll will not work without dfll-soc enabled.
>
> CLDVFS control logic is split into 2 clock domains. dvfs_ref_clk and 
> dvfs_soc_clk.
>
> Majority of the control logic is clocked from dvfs_soc_clk for 
> interfacing control registers.
>
Note on reverse ordering for restore. Currently restore order goes thru 
clock list and for each root goes thru parent -> child restore.

this order is correct and also all clocks are parented properly so they 
follow proper order.

dfllCPU is the only one where current driver doesn't take care of 
dependency in dfll_soc which gets enabled only after dfll_ref.


Based on dfllCPU control logic module design, dfll_ref and dfll_soc 
should be enabled prior to dfll init/enable.

So parenting dfll_soc to dfllCPU keeps proper order.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18 17:41                                                                                                       ` Sowjanya Komatineni
@ 2019-07-18 18:29                                                                                                         ` Sowjanya Komatineni
  2019-07-18 19:11                                                                                                           ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-18 18:29 UTC (permalink / raw)
  To: Dmitry Osipenko, sboyd, Michael Turquette
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree


On 7/18/19 10:41 AM, Sowjanya Komatineni wrote:
>
> On 7/18/19 10:22 AM, Sowjanya Komatineni wrote:
>>
>> On 7/18/19 9:34 AM, Dmitry Osipenko wrote:
>>> 18.07.2019 4:15, Sowjanya Komatineni пишет:
>>>
>>> [snip]
>>>
>>>>>> Please try to fix all missing dependencies and orderings.
>>>>> Peter,
>>>>>
>>>>> dfllCPU_OUT is the first one to go thru restore when
>>>>> clk_restore_context traverses thru the list.
>>>>>
>>>>> dfllCPU_OUT has dependency on DFLL_ref and DFLL_SOC but this
>>>>> dependency is unknown to clock-tree.
>>>>>
>>>>> We can add DFLL_REF and DFLL_SOC as parents to dfllCPU_OUT during
>>>>> register so dfllCPU_OUT save/restore happens after their parents are
>>>>> restored.
>>>>>
>>>>> But DFLL needs both of these to be restored before DFLLCPU_Out and as
>>>>> DFLL_SOC restore always happens after the REF, thinking to add
>>>>> DFLL_SOC as parent to dfllCPU_OUT so save/restore follows after their
>>>>> dependencies.
>>>>>
>>>>> Please comment.
>>>>>
>>>> Did quick try and I see by adding dfll-soc as parent to 
>>>> dfllCPU_OUT, its
>>>> in proper order after all its dependencies.
>>>>
>>>> Can now add dfll save/restore to do dfll reinit during restore..
>>>>
>>> If dfllCPU_OUT can work properly with dfll-soc being disabled, then 
>>> this
>>> kind of dependency isn't very correct and just papers over the real
>>> problem, which is that there should be a way for CCF to specify 
>>> multiple
>>> dependencies for the clock or the reverse ordering should be used for
>>> the restoring.
>>
>> dfll will not work without dfll-soc enabled.
>>
>> CLDVFS control logic is split into 2 clock domains. dvfs_ref_clk and 
>> dvfs_soc_clk.
>>
>> Majority of the control logic is clocked from dvfs_soc_clk for 
>> interfacing control registers.
>>
> Note on reverse ordering for restore. Currently restore order goes 
> thru clock list and for each root goes thru parent -> child restore.
>
> this order is correct and also all clocks are parented properly so 
> they follow proper order.
>
> dfllCPU is the only one where current driver doesn't take care of 
> dependency in dfll_soc which gets enabled only after dfll_ref.
>
>
> Based on dfllCPU control logic module design, dfll_ref and dfll_soc 
> should be enabled prior to dfll init/enable.
>
> So parenting dfll_soc to dfllCPU keeps proper order.
>

1. With dfllCPU parenting to dfll_soc, its keeps it in expected order 
and we don't define any parent clk_ops anyway for this, so should be OK?

OR

2. Any suggestion on how to define/specify dependencies for clock other 
than parenting to follow proper order in clock tree as clk_save_context 
and clk_restore_context strictly goes thru clock tree order and all 
other clocks are parented properly except for dfllCPU where there is no 
parent. Techinically dfll_ref & dfll_soc are not parents but they need 
to be configured prior to dfll reinit.

OR

3. I don't see way to override clk_save_context/clk_restore_context APIs 
to change the way of traversal so I can modify to traverse in expected 
order without dfllCPU parenting.

OR

4. dfll re-init can be done in dfll-fcpu driver pm_ops which actually 
registers dfll or at the end of tegra210_clock resume


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18 18:29                                                                                                         ` Sowjanya Komatineni
@ 2019-07-18 19:11                                                                                                           ` Sowjanya Komatineni
  0 siblings, 0 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-18 19:11 UTC (permalink / raw)
  To: Dmitry Osipenko, sboyd, Michael Turquette
  Cc: Peter De Schrijver, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree


On 7/18/19 11:29 AM, Sowjanya Komatineni wrote:
>
> On 7/18/19 10:41 AM, Sowjanya Komatineni wrote:
>>
>> On 7/18/19 10:22 AM, Sowjanya Komatineni wrote:
>>>
>>> On 7/18/19 9:34 AM, Dmitry Osipenko wrote:
>>>> 18.07.2019 4:15, Sowjanya Komatineni пишет:
>>>>
>>>> [snip]
>>>>
>>>>>>> Please try to fix all missing dependencies and orderings.
>>>>>> Peter,
>>>>>>
>>>>>> dfllCPU_OUT is the first one to go thru restore when
>>>>>> clk_restore_context traverses thru the list.
>>>>>>
>>>>>> dfllCPU_OUT has dependency on DFLL_ref and DFLL_SOC but this
>>>>>> dependency is unknown to clock-tree.
>>>>>>
>>>>>> We can add DFLL_REF and DFLL_SOC as parents to dfllCPU_OUT during
>>>>>> register so dfllCPU_OUT save/restore happens after their parents are
>>>>>> restored.
>>>>>>
>>>>>> But DFLL needs both of these to be restored before DFLLCPU_Out 
>>>>>> and as
>>>>>> DFLL_SOC restore always happens after the REF, thinking to add
>>>>>> DFLL_SOC as parent to dfllCPU_OUT so save/restore follows after 
>>>>>> their
>>>>>> dependencies.
>>>>>>
>>>>>> Please comment.
>>>>>>
>>>>> Did quick try and I see by adding dfll-soc as parent to 
>>>>> dfllCPU_OUT, its
>>>>> in proper order after all its dependencies.
>>>>>
>>>>> Can now add dfll save/restore to do dfll reinit during restore..
>>>>>
>>>> If dfllCPU_OUT can work properly with dfll-soc being disabled, then 
>>>> this
>>>> kind of dependency isn't very correct and just papers over the real
>>>> problem, which is that there should be a way for CCF to specify 
>>>> multiple
>>>> dependencies for the clock or the reverse ordering should be used for
>>>> the restoring.
>>>
>>> dfll will not work without dfll-soc enabled.
>>>
>>> CLDVFS control logic is split into 2 clock domains. dvfs_ref_clk and 
>>> dvfs_soc_clk.
>>>
>>> Majority of the control logic is clocked from dvfs_soc_clk for 
>>> interfacing control registers.
>>>
>> Note on reverse ordering for restore. Currently restore order goes 
>> thru clock list and for each root goes thru parent -> child restore.
>>
>> this order is correct and also all clocks are parented properly so 
>> they follow proper order.
>>
>> dfllCPU is the only one where current driver doesn't take care of 
>> dependency in dfll_soc which gets enabled only after dfll_ref.
>>
>>
>> Based on dfllCPU control logic module design, dfll_ref and dfll_soc 
>> should be enabled prior to dfll init/enable.
>>
>> So parenting dfll_soc to dfllCPU keeps proper order.
>>
>
> 1. With dfllCPU parenting to dfll_soc, its keeps it in expected order 
> and we don't define any parent clk_ops anyway for this, so should be OK?
>
> OR
>
> 2. Any suggestion on how to define/specify dependencies for clock 
> other than parenting to follow proper order in clock tree as 
> clk_save_context and clk_restore_context strictly goes thru clock tree 
> order and all other clocks are parented properly except for dfllCPU 
> where there is no parent. Techinically dfll_ref & dfll_soc are not 
> parents but they need to be configured prior to dfll reinit.
>
> OR
>
> 3. I don't see way to override clk_save_context/clk_restore_context 
> APIs to change the way of traversal so I can modify to traverse in 
> expected order without dfllCPU parenting.

instead of using core API of save/restore context, probably can change 
traversing to skip the 1st root in clock tree during initial traversing 
and at the end invoke restore for 1st node.

>
> OR
>
> 4. dfll re-init can be done in dfll-fcpu driver pm_ops which actually 
> registers dfll or at the end of tegra210_clock resume
>
Please suggest if you agree with either 1/3/4.



^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 18:25                               ` Dmitry Osipenko
  2019-07-16 18:30                                 ` Sowjanya Komatineni
@ 2019-07-18 19:15                                 ` Peter De Schrijver
  1 sibling, 0 replies; 111+ messages in thread
From: Peter De Schrijver @ 2019-07-18 19:15 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Sowjanya Komatineni, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree

On Tue, Jul 16, 2019 at 09:25:43PM +0300, Dmitry Osipenko wrote:
> 16.07.2019 21:19, Sowjanya Komatineni пишет:
> > 
> > On 7/16/19 9:50 AM, Sowjanya Komatineni wrote:
> >>
> >> On 7/16/19 8:00 AM, Dmitry Osipenko wrote:
> >>> 16.07.2019 11:06, Peter De Schrijver пишет:
> >>>> On Tue, Jul 16, 2019 at 03:24:26PM +0800, Joseph Lo wrote:
> >>>>>> OK, Will add to CPUFreq driver...
> >>>>>>> The other thing that also need attention is that T124 CPUFreq driver
> >>>>>>> implicitly relies on DFLL driver to be probed first, which is icky.
> >>>>>>>
> >>>>>> Should I add check for successful dfll clk register explicitly in
> >>>>>> CPUFreq driver probe and defer till dfll clk registers?
> >>> Probably you should use the "device links". See [1][2] for the example.
> >>>
> >>> [1]
> >>> https://elixir.bootlin.com/linux/v5.2.1/source/drivers/gpu/drm/tegra/dc.c#L2383
> >>>
> >>>
> >>> [2] https://www.kernel.org/doc/html/latest/driver-api/device_link.html
> >>>
> >>> Return EPROBE_DEFER instead of EINVAL if device_link_add() fails. And
> >>> use of_find_device_by_node() to get the DFLL's device, see [3].
> >>>
> >>> [3]
> >>> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/devfreq/tegra20-devfreq.c#n100
> >>>
> >> Will go thru and add...
> 
> Looks like I initially confused this case with getting orphaned clock.
> I'm now seeing that the DFLL driver registers the clock and then
> clk_get(dfll) should be returning EPROBE_DEFER until DFLL driver is
> probed, hence everything should be fine as-is and there is no real need
> for the 'device link'. Sorry for the confusion!
> 
> >>>>> Sorry, I didn't follow the mail thread. Just regarding the DFLL part.
> >>>>>
> >>>>> As you know it, the DFLL clock is one of the CPU clock sources and
> >>>>> integrated with DVFS control logic with the regulator. We will not
> >>>>> switch
> >>>>> CPU to other clock sources once we switched to DFLL. Because the
> >>>>> CPU has
> >>>>> been regulated by the DFLL HW with the DVFS table (CVB or OPP table
> >>>>> you see
> >>>>> in the driver.). We shouldn't reparent it to other sources with unknew
> >>>>> freq/volt pair. That's not guaranteed to work. We allow switching to
> >>>>> open-loop mode but different sources.
> >>> Okay, then the CPUFreq driver will have to enforce DFLL freq to PLLP's
> >>> rate before switching to PLLP in order to have a proper CPU voltage.
> >>
> >> PLLP freq is safe to work for any CPU voltage. So no need to enforce
> >> DFLL freq to PLLP rate before changing CCLK_G source to PLLP during
> >> suspend
> >>
> > Sorry, please ignore my above comment. During suspend, need to change
> > CCLK_G source to PLLP when dfll is in closed loop mode first and then
> > dfll need to be set to open loop.
> 
> Okay.
> 
> >>>>> And I don't exactly understand why we need to switch to PLLP in CPU
> >>>>> idle
> >>>>> driver. Just keep it on CL-DVFS mode all the time.
> >>>>>
> >>>>> In SC7 entry, the dfll suspend function moves it the open-loop
> >>>>> mode. That's
> >>>>> all. The sc7-entryfirmware will handle the rest of the sequence to
> >>>>> turn off
> >>>>> the CPU power.
> >>>>>
> >>>>> In SC7 resume, the warmboot code will handle the sequence to turn on
> >>>>> regulator and power up the CPU cluster. And leave it on PLL_P. After
> >>>>> resuming to the kernel, we re-init DFLL, restore the CPU clock
> >>>>> policy (CPU
> >>>>> runs on DFLL open-loop mode) and then moving to close-loop mode.
> >>> The DFLL is re-inited after switching CCLK to DFLL parent during of the
> >>> early clocks-state restoring by CaR driver. Hence instead of having odd
> >>> hacks in the CaR driver, it is much nicer to have a proper
> >>> suspend-resume sequencing of the device drivers. In this case CPUFreq
> >>> driver is the driver that enables DFLL and switches CPU to that clock
> >>> source, which means that this driver is also should be responsible for
> >>> management of the DFLL's state during of suspend/resume process. If
> >>> CPUFreq driver disables DFLL during suspend and re-enables it during
> >>> resume, then looks like the CaR driver hacks around DFLL are not needed.
> >>>
> >>>>> The DFLL part looks good to me. BTW, change the patch subject to "Add
> >>>>> suspend-resume support" seems more appropriate to me.
> >>>>>
> >>>> To clarify this, the sequences for DFLL use are as follows (assuming
> >>>> all
> >>>> required DFLL hw configuration has been done)
> >>>>
> >>>> Switch to DFLL:
> >>>> 0) Save current parent and frequency
> >>>> 1) Program DFLL to open loop mode
> >>>> 2) Enable DFLL
> >>>> 3) Change cclk_g parent to DFLL
> >>>> For OVR regulator:
> >>>> 4) Change PWM output pin from tristate to output
> >>>> 5) Enable DFLL PWM output
> >>>> For I2C regulator:
> >>>> 4) Enable DFLL I2C output
> >>>> 6) Program DFLL to closed loop mode
> >>>>
> >>>> Switch away from DFLL:
> >>>> 0) Change cclk_g parent to PLLP so the CPU frequency is ok for any
> >>>> vdd_cpu voltage
> >>>> 1) Program DFLL to open loop mode
> >>>>
> > I see during switch away from DFLL (suspend), cclk_g parent is not
> > changed to PLLP before changing dfll to open loop mode.
> > 
> > Will add this ...
> 
> The CPUFreq driver switches parent to PLLP during the probe, similar
> should be done on suspend.
> 
> I'm also wondering if it's always safe to switch to PLLP in the probe.
> If CPU is running on a lower freq than PLLP, then some other more
> appropriate intermediate parent should be selected.
> 

Yes it is always safe because 408MHz is lower than fmax @ Vmin.

Peter.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-16 18:43                                   ` Dmitry Osipenko
  2019-07-16 19:26                                     ` Sowjanya Komatineni
@ 2019-07-18 19:18                                     ` Peter De Schrijver
  2019-07-18 19:24                                       ` Sowjanya Komatineni
  1 sibling, 1 reply; 111+ messages in thread
From: Peter De Schrijver @ 2019-07-18 19:18 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Sowjanya Komatineni, Joseph Lo, thierry.reding, jonathanh, tglx,
	jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, sboyd, linux-clk, linux-gpio, jckuo, talho,
	linux-tegra, linux-kernel, mperttunen, spatra, robh+dt,
	devicetree

On Tue, Jul 16, 2019 at 09:43:16PM +0300, Dmitry Osipenko wrote:
> > CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher rate
> > so switching to PLL_P during CPUFreq probe prior to dfll clock enable
> > should be safe.
> 
> AFAIK, PLLX could run at ~200MHz. There is also a divided output of PLLP
> which CCLKG supports, the PLLP_OUT4.
> 
> Probably, realistically, CPU is always running off a fast PLLX during
> boot, but I'm wondering what may happen on KEXEC. I guess ideally
> CPUFreq driver should also have a 'shutdown' callback to teardown DFLL
> on a reboot, but likely that there are other clock-related problems as
> well that may break KEXEC and thus it is not very important at the moment.
> 

If you turn off the DFLL, you have to be aware that the voltage margins
for DFLL use are lower than for PLL use. So you either need to be sure
to switch to a frequency below fmax @ Vmin or you program the boot
voltage and then you can use PLLX as setup by the bootloader. For OVR
regulators you can't program a voltage without the DFLL, so you have to
tristate the PWM output which will give you a hardwired boot voltage.

Peter.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18 19:18                                     ` Peter De Schrijver
@ 2019-07-18 19:24                                       ` Sowjanya Komatineni
  2019-07-18 20:11                                         ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-18 19:24 UTC (permalink / raw)
  To: Peter De Schrijver, Dmitry Osipenko
  Cc: Joseph Lo, thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree


On 7/18/19 12:18 PM, Peter De Schrijver wrote:
> On Tue, Jul 16, 2019 at 09:43:16PM +0300, Dmitry Osipenko wrote:
>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher rate
>>> so switching to PLL_P during CPUFreq probe prior to dfll clock enable
>>> should be safe.
>> AFAIK, PLLX could run at ~200MHz. There is also a divided output of PLLP
>> which CCLKG supports, the PLLP_OUT4.
>>
>> Probably, realistically, CPU is always running off a fast PLLX during
>> boot, but I'm wondering what may happen on KEXEC. I guess ideally
>> CPUFreq driver should also have a 'shutdown' callback to teardown DFLL
>> on a reboot, but likely that there are other clock-related problems as
>> well that may break KEXEC and thus it is not very important at the moment.
>>
> If you turn off the DFLL, you have to be aware that the voltage margins
> for DFLL use are lower than for PLL use. So you either need to be sure
> to switch to a frequency below fmax @ Vmin or you program the boot
> voltage and then you can use PLLX as setup by the bootloader. For OVR
> regulators you can't program a voltage without the DFLL, so you have to
> tristate the PWM output which will give you a hardwired boot voltage.
>
> Peter.

Yes, we switch CPU to PLLP and then disable DFLL during suspend.



^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-17 23:44                                                                                             ` Dmitry Osipenko
  2019-07-18  0:25                                                                                               ` Sowjanya Komatineni
@ 2019-07-18 19:42                                                                                               ` Peter De Schrijver
  2019-07-18 20:26                                                                                                 ` Dmitry Osipenko
  1 sibling, 1 reply; 111+ messages in thread
From: Peter De Schrijver @ 2019-07-18 19:42 UTC (permalink / raw)
  To: Dmitry Osipenko
  Cc: Sowjanya Komatineni, sboyd, Michael Turquette, Joseph Lo,
	thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree

On Thu, Jul 18, 2019 at 02:44:56AM +0300, Dmitry Osipenko wrote:
> > 
> > dependencies I am referring are dfll_ref, dfll_soc, and DVFS peripheral
> > clocks which need to be restored prior to DFLL reinit.
> 
> Okay, but that shouldn't be a problem if clock dependencies are set up
> properly.
> 
> >>> reverse list order during restore might not work as all other clocks are
> >>> in proper order no with any ref clocks for plls getting restored prior
> >>> to their clients
> >> Why? The ref clocks should be registered first and be the roots for PLLs
> >> and the rest. If it's not currently the case, then this need to be
> >> fixed. You need to ensure that each clock is modeled properly. If some
> >> child clock really depends on multiple parents, then the parents need to
> >> in the correct order or CCF need to be taught about such
> >> multi-dependencies.
> >>
> >> If some required feature is missed, then you have to implement it
> >> properly and for all, that's how things are done in upstream. Sometimes
> >> it's quite a lot of extra work that everyone are benefiting from in
> >> the end.
> >>
> >> [snip]
> > 
> > Yes, we should register ref/parents before their clients.
> > 
> > cclk_g clk is registered last after all pll and peripheral clocks are
> > registers during clock init.
> > 
> > dfllCPU_out clk is registered later during dfll-fcpu driver probe and
> > gets added to the clock list.
> > 
> > Probably the issue seems to be not linking dfll_ref and dfll_soc
> > dependencies for dfllCPU_out thru clock list.
> > 
> > clk-dfll driver during dfll_init_clks gets ref_clk and soc_clk reference
> > thru DT.

The dfll does not have any parents. It has some clocks which are needed
for the logic part of the dfll to function, but there's no parent clock
as such unlike for peripheral clocks or PLLs where the parent is at
least used as a reference. The I2C controller of the DFLL shares the
lines with a normal I2C controller using some arbitration logic. That
logic only works if the clock for the normal I2C controller is enabled.
So you need probably 3 clocks enabled to initialize the dfll in that
case. I don't think it makes sense to add complicated logic to the clock
core to deal with this rather strange case. To me it makes more sense to
use pmops and open code the sequence there.

Peter.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18 19:24                                       ` Sowjanya Komatineni
@ 2019-07-18 20:11                                         ` Dmitry Osipenko
  2019-07-18 20:32                                           ` Dmitry Osipenko
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-18 20:11 UTC (permalink / raw)
  To: Sowjanya Komatineni, Peter De Schrijver
  Cc: Joseph Lo, thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree

18.07.2019 22:24, Sowjanya Komatineni пишет:
> 
> On 7/18/19 12:18 PM, Peter De Schrijver wrote:
>> On Tue, Jul 16, 2019 at 09:43:16PM +0300, Dmitry Osipenko wrote:
>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher
>>>> rate
>>>> so switching to PLL_P during CPUFreq probe prior to dfll clock enable
>>>> should be safe.
>>> AFAIK, PLLX could run at ~200MHz. There is also a divided output of PLLP
>>> which CCLKG supports, the PLLP_OUT4.
>>>
>>> Probably, realistically, CPU is always running off a fast PLLX during
>>> boot, but I'm wondering what may happen on KEXEC. I guess ideally
>>> CPUFreq driver should also have a 'shutdown' callback to teardown DFLL
>>> on a reboot, but likely that there are other clock-related problems as
>>> well that may break KEXEC and thus it is not very important at the
>>> moment.
>>>
>> If you turn off the DFLL, you have to be aware that the voltage margins
>> for DFLL use are lower than for PLL use. So you either need to be sure
>> to switch to a frequency below fmax @ Vmin or you program the boot
>> voltage and then you can use PLLX as setup by the bootloader. For OVR
>> regulators you can't program a voltage without the DFLL, so you have to
>> tristate the PWM output which will give you a hardwired boot voltage.
>>
>> Peter.
> 
> Yes, we switch CPU to PLLP and then disable DFLL during suspend.

I'm wondering what happens to T124 on resume from suspend, given that it
switches CPU to PLLX [1]. I imagine that CPU voltage could be lower than
needed if suspend happened on DFLL. I'm also now vaguely recalling that
CPUFreq driver was disabled for T124 because of some problems.

Or maybe warmboot code is actually touching the voltage regulators?

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/arch/arm/mach-tegra/sleep-tegra30.S#n389

That is also should be a problem for T30 if voltage scaling is happening
and I have some patches in works that are switching CPU to PLLP instead
of PLLX on suspend/resume.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18 19:42                                                                                               ` Peter De Schrijver
@ 2019-07-18 20:26                                                                                                 ` Dmitry Osipenko
  2019-07-18 20:36                                                                                                   ` Sowjanya Komatineni
  0 siblings, 1 reply; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-18 20:26 UTC (permalink / raw)
  To: Peter De Schrijver
  Cc: Sowjanya Komatineni, sboyd, Michael Turquette, Joseph Lo,
	thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree

18.07.2019 22:42, Peter De Schrijver пишет:
> On Thu, Jul 18, 2019 at 02:44:56AM +0300, Dmitry Osipenko wrote:
>>>
>>> dependencies I am referring are dfll_ref, dfll_soc, and DVFS peripheral
>>> clocks which need to be restored prior to DFLL reinit.
>>
>> Okay, but that shouldn't be a problem if clock dependencies are set up
>> properly.
>>
>>>>> reverse list order during restore might not work as all other clocks are
>>>>> in proper order no with any ref clocks for plls getting restored prior
>>>>> to their clients
>>>> Why? The ref clocks should be registered first and be the roots for PLLs
>>>> and the rest. If it's not currently the case, then this need to be
>>>> fixed. You need to ensure that each clock is modeled properly. If some
>>>> child clock really depends on multiple parents, then the parents need to
>>>> in the correct order or CCF need to be taught about such
>>>> multi-dependencies.
>>>>
>>>> If some required feature is missed, then you have to implement it
>>>> properly and for all, that's how things are done in upstream. Sometimes
>>>> it's quite a lot of extra work that everyone are benefiting from in
>>>> the end.
>>>>
>>>> [snip]
>>>
>>> Yes, we should register ref/parents before their clients.
>>>
>>> cclk_g clk is registered last after all pll and peripheral clocks are
>>> registers during clock init.
>>>
>>> dfllCPU_out clk is registered later during dfll-fcpu driver probe and
>>> gets added to the clock list.
>>>
>>> Probably the issue seems to be not linking dfll_ref and dfll_soc
>>> dependencies for dfllCPU_out thru clock list.
>>>
>>> clk-dfll driver during dfll_init_clks gets ref_clk and soc_clk reference
>>> thru DT.
> 
> The dfll does not have any parents. It has some clocks which are needed
> for the logic part of the dfll to function, but there's no parent clock
> as such unlike for peripheral clocks or PLLs where the parent is at
> least used as a reference. The I2C controller of the DFLL shares the
> lines with a normal I2C controller using some arbitration logic. That
> logic only works if the clock for the normal I2C controller is enabled.
> So you need probably 3 clocks enabled to initialize the dfll in that
> case. I don't think it makes sense to add complicated logic to the clock
> core to deal with this rather strange case. To me it makes more sense to
> use pmops and open code the sequence there.

It looks to me that dfllCPU is a PLL and dfll_ref is its reference
parent, while dfll_soc clocks the logic that dynamically reconfigures
dfllCPU in background. I see that PLLP is defined as a parent for
dfll_ref and dfll_soc in the code. Hence seems dfll_ref should be set as
a parent for dfllCPU, no?

Either way is good to me, given that DFLL will be disabled during
suspend. Resetting DFLL on DFLL's driver resume using PM ops should be
good. And then it also will be better to error out if DFLL is active
during suspend on the DFLL's driver suspend.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18 20:11                                         ` Dmitry Osipenko
@ 2019-07-18 20:32                                           ` Dmitry Osipenko
  0 siblings, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-18 20:32 UTC (permalink / raw)
  To: Sowjanya Komatineni, Peter De Schrijver
  Cc: Joseph Lo, thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, sboyd, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree

18.07.2019 23:11, Dmitry Osipenko пишет:
> 18.07.2019 22:24, Sowjanya Komatineni пишет:
>>
>> On 7/18/19 12:18 PM, Peter De Schrijver wrote:
>>> On Tue, Jul 16, 2019 at 09:43:16PM +0300, Dmitry Osipenko wrote:
>>>>> CPU parents are PLL_X, PLL_P, and dfll. PLL_X always runs at higher
>>>>> rate
>>>>> so switching to PLL_P during CPUFreq probe prior to dfll clock enable
>>>>> should be safe.
>>>> AFAIK, PLLX could run at ~200MHz. There is also a divided output of PLLP
>>>> which CCLKG supports, the PLLP_OUT4.
>>>>
>>>> Probably, realistically, CPU is always running off a fast PLLX during
>>>> boot, but I'm wondering what may happen on KEXEC. I guess ideally
>>>> CPUFreq driver should also have a 'shutdown' callback to teardown DFLL
>>>> on a reboot, but likely that there are other clock-related problems as
>>>> well that may break KEXEC and thus it is not very important at the
>>>> moment.
>>>>
>>> If you turn off the DFLL, you have to be aware that the voltage margins
>>> for DFLL use are lower than for PLL use. So you either need to be sure
>>> to switch to a frequency below fmax @ Vmin or you program the boot
>>> voltage and then you can use PLLX as setup by the bootloader. For OVR
>>> regulators you can't program a voltage without the DFLL, so you have to
>>> tristate the PWM output which will give you a hardwired boot voltage.
>>>
>>> Peter.
>>
>> Yes, we switch CPU to PLLP and then disable DFLL during suspend.
> 
> I'm wondering what happens to T124 on resume from suspend, given that it
> switches CPU to PLLX [1]. I imagine that CPU voltage could be lower than
> needed if suspend happened on DFLL. I'm also now vaguely recalling that
> CPUFreq driver was disabled for T124 because of some problems.
> 
> Or maybe warmboot code is actually touching the voltage regulators?

Oh wait, warmboot is even irrelevant LP1.

> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/arch/arm/mach-tegra/sleep-tegra30.S#n389
> 
> That is also should be a problem for T30 if voltage scaling is happening
> and I have some patches in works that are switching CPU to PLLP instead
> of PLLX on suspend/resume.
> 


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18 20:26                                                                                                 ` Dmitry Osipenko
@ 2019-07-18 20:36                                                                                                   ` Sowjanya Komatineni
  2019-07-18 22:52                                                                                                     ` Dmitry Osipenko
  2019-07-18 23:08                                                                                                     ` Sowjanya Komatineni
  0 siblings, 2 replies; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-18 20:36 UTC (permalink / raw)
  To: Dmitry Osipenko, Peter De Schrijver
  Cc: sboyd, Michael Turquette, Joseph Lo, thierry.reding, jonathanh,
	tglx, jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree


On 7/18/19 1:26 PM, Dmitry Osipenko wrote:
> 18.07.2019 22:42, Peter De Schrijver пишет:
>> On Thu, Jul 18, 2019 at 02:44:56AM +0300, Dmitry Osipenko wrote:
>>>> dependencies I am referring are dfll_ref, dfll_soc, and DVFS peripheral
>>>> clocks which need to be restored prior to DFLL reinit.
>>> Okay, but that shouldn't be a problem if clock dependencies are set up
>>> properly.
>>>
>>>>>> reverse list order during restore might not work as all other clocks are
>>>>>> in proper order no with any ref clocks for plls getting restored prior
>>>>>> to their clients
>>>>> Why? The ref clocks should be registered first and be the roots for PLLs
>>>>> and the rest. If it's not currently the case, then this need to be
>>>>> fixed. You need to ensure that each clock is modeled properly. If some
>>>>> child clock really depends on multiple parents, then the parents need to
>>>>> in the correct order or CCF need to be taught about such
>>>>> multi-dependencies.
>>>>>
>>>>> If some required feature is missed, then you have to implement it
>>>>> properly and for all, that's how things are done in upstream. Sometimes
>>>>> it's quite a lot of extra work that everyone are benefiting from in
>>>>> the end.
>>>>>
>>>>> [snip]
>>>> Yes, we should register ref/parents before their clients.
>>>>
>>>> cclk_g clk is registered last after all pll and peripheral clocks are
>>>> registers during clock init.
>>>>
>>>> dfllCPU_out clk is registered later during dfll-fcpu driver probe and
>>>> gets added to the clock list.
>>>>
>>>> Probably the issue seems to be not linking dfll_ref and dfll_soc
>>>> dependencies for dfllCPU_out thru clock list.
>>>>
>>>> clk-dfll driver during dfll_init_clks gets ref_clk and soc_clk reference
>>>> thru DT.
>> The dfll does not have any parents. It has some clocks which are needed
>> for the logic part of the dfll to function, but there's no parent clock
>> as such unlike for peripheral clocks or PLLs where the parent is at
>> least used as a reference. The I2C controller of the DFLL shares the
>> lines with a normal I2C controller using some arbitration logic. That
>> logic only works if the clock for the normal I2C controller is enabled.
>> So you need probably 3 clocks enabled to initialize the dfll in that
>> case. I don't think it makes sense to add complicated logic to the clock
>> core to deal with this rather strange case. To me it makes more sense to
>> use pmops and open code the sequence there.
> It looks to me that dfllCPU is a PLL and dfll_ref is its reference
> parent, while dfll_soc clocks the logic that dynamically reconfigures
> dfllCPU in background. I see that PLLP is defined as a parent for
> dfll_ref and dfll_soc in the code. Hence seems dfll_ref should be set as
> a parent for dfllCPU, no?

dfll_soc will not be restored by the time dfllCPU resume happens after 
dfll_ref.

without dfll_soc, dfllCPU cannot be resumed either. So if we decide to 
use parent we should use dfll_soc.

> Either way is good to me, given that DFLL will be disabled during
> suspend. Resetting DFLL on DFLL's driver resume using PM ops should be
> good. And then it also will be better to error out if DFLL is active
> during suspend on the DFLL's driver suspend.

Doing in dfll-fcpu pm_ops is much better as it happens right after all 
clocks are restored and unlike other clock enables, dfll need dfll 
controller programming as well and is actually registered in dfll-fcpu 
driver.

With this, below is the sequence:

CPUFreq suspend switches CPU to PLLP and disables dfll

Will add dfll_suspend/resume in dfll-fcpu driver and in dfll suspend 
will check for dfll active and will error out suspend.

dfll resume does dfll reinit.

CPUFreq resume enables dfll and switches CPU to dfll.


Will go with doing in dfll-fcpu pm_ops rather than parenting dfllCPU_OUT...


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18 20:36                                                                                                   ` Sowjanya Komatineni
@ 2019-07-18 22:52                                                                                                     ` Dmitry Osipenko
  2019-07-18 23:08                                                                                                     ` Sowjanya Komatineni
  1 sibling, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-18 22:52 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: Peter De Schrijver, sboyd, Michael Turquette, Joseph Lo,
	thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree

В Thu, 18 Jul 2019 13:36:35 -0700
Sowjanya Komatineni <skomatineni@nvidia.com> пишет:

> On 7/18/19 1:26 PM, Dmitry Osipenko wrote:
> > 18.07.2019 22:42, Peter De Schrijver пишет:  
> >> On Thu, Jul 18, 2019 at 02:44:56AM +0300, Dmitry Osipenko wrote:  
> >>>> dependencies I am referring are dfll_ref, dfll_soc, and DVFS
> >>>> peripheral clocks which need to be restored prior to DFLL
> >>>> reinit.  
> >>> Okay, but that shouldn't be a problem if clock dependencies are
> >>> set up properly.
> >>>  
> >>>>>> reverse list order during restore might not work as all other
> >>>>>> clocks are in proper order no with any ref clocks for plls
> >>>>>> getting restored prior to their clients  
> >>>>> Why? The ref clocks should be registered first and be the roots
> >>>>> for PLLs and the rest. If it's not currently the case, then
> >>>>> this need to be fixed. You need to ensure that each clock is
> >>>>> modeled properly. If some child clock really depends on
> >>>>> multiple parents, then the parents need to in the correct order
> >>>>> or CCF need to be taught about such multi-dependencies.
> >>>>>
> >>>>> If some required feature is missed, then you have to implement
> >>>>> it properly and for all, that's how things are done in
> >>>>> upstream. Sometimes it's quite a lot of extra work that
> >>>>> everyone are benefiting from in the end.
> >>>>>
> >>>>> [snip]  
> >>>> Yes, we should register ref/parents before their clients.
> >>>>
> >>>> cclk_g clk is registered last after all pll and peripheral
> >>>> clocks are registers during clock init.
> >>>>
> >>>> dfllCPU_out clk is registered later during dfll-fcpu driver
> >>>> probe and gets added to the clock list.
> >>>>
> >>>> Probably the issue seems to be not linking dfll_ref and dfll_soc
> >>>> dependencies for dfllCPU_out thru clock list.
> >>>>
> >>>> clk-dfll driver during dfll_init_clks gets ref_clk and soc_clk
> >>>> reference thru DT.  
> >> The dfll does not have any parents. It has some clocks which are
> >> needed for the logic part of the dfll to function, but there's no
> >> parent clock as such unlike for peripheral clocks or PLLs where
> >> the parent is at least used as a reference. The I2C controller of
> >> the DFLL shares the lines with a normal I2C controller using some
> >> arbitration logic. That logic only works if the clock for the
> >> normal I2C controller is enabled. So you need probably 3 clocks
> >> enabled to initialize the dfll in that case. I don't think it
> >> makes sense to add complicated logic to the clock core to deal
> >> with this rather strange case. To me it makes more sense to use
> >> pmops and open code the sequence there.  
> > It looks to me that dfllCPU is a PLL and dfll_ref is its reference
> > parent, while dfll_soc clocks the logic that dynamically
> > reconfigures dfllCPU in background. I see that PLLP is defined as a
> > parent for dfll_ref and dfll_soc in the code. Hence seems dfll_ref
> > should be set as a parent for dfllCPU, no?  
> 
> dfll_soc will not be restored by the time dfllCPU resume happens
> after dfll_ref.
> 
> without dfll_soc, dfllCPU cannot be resumed either. So if we decide
> to use parent we should use dfll_soc.

Okay, my point is that the parents should be properly specified any
ways.

> > Either way is good to me, given that DFLL will be disabled during
> > suspend. Resetting DFLL on DFLL's driver resume using PM ops should
> > be good. And then it also will be better to error out if DFLL is
> > active during suspend on the DFLL's driver suspend.  
> 
> Doing in dfll-fcpu pm_ops is much better as it happens right after
> all clocks are restored and unlike other clock enables, dfll need
> dfll controller programming as well and is actually registered in
> dfll-fcpu driver.
> 
> With this, below is the sequence:
> 
> CPUFreq suspend switches CPU to PLLP and disables dfll
> 
> Will add dfll_suspend/resume in dfll-fcpu driver and in dfll suspend 
> will check for dfll active and will error out suspend.
> 
> dfll resume does dfll reinit.
> 
> CPUFreq resume enables dfll and switches CPU to dfll.
> 
> 
> Will go with doing in dfll-fcpu pm_ops rather than parenting
> dfllCPU_OUT...
> 

Sounds good.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18 20:36                                                                                                   ` Sowjanya Komatineni
  2019-07-18 22:52                                                                                                     ` Dmitry Osipenko
@ 2019-07-18 23:08                                                                                                     ` Sowjanya Komatineni
  2019-07-18 23:52                                                                                                       ` Dmitry Osipenko
  1 sibling, 1 reply; 111+ messages in thread
From: Sowjanya Komatineni @ 2019-07-18 23:08 UTC (permalink / raw)
  To: Dmitry Osipenko, Peter De Schrijver
  Cc: sboyd, Michael Turquette, Joseph Lo, thierry.reding, jonathanh,
	tglx, jason, marc.zyngier, linus.walleij, stefan, mark.rutland,
	pgaikwad, linux-clk, linux-gpio, jckuo, talho, linux-tegra,
	linux-kernel, mperttunen, spatra, robh+dt, devicetree


On 7/18/19 1:36 PM, Sowjanya Komatineni wrote:
>
> On 7/18/19 1:26 PM, Dmitry Osipenko wrote:
>> 18.07.2019 22:42, Peter De Schrijver пишет:
>>> On Thu, Jul 18, 2019 at 02:44:56AM +0300, Dmitry Osipenko wrote:
>>>>> dependencies I am referring are dfll_ref, dfll_soc, and DVFS 
>>>>> peripheral
>>>>> clocks which need to be restored prior to DFLL reinit.
>>>> Okay, but that shouldn't be a problem if clock dependencies are set up
>>>> properly.
>>>>
>>>>>>> reverse list order during restore might not work as all other 
>>>>>>> clocks are
>>>>>>> in proper order no with any ref clocks for plls getting restored 
>>>>>>> prior
>>>>>>> to their clients
>>>>>> Why? The ref clocks should be registered first and be the roots 
>>>>>> for PLLs
>>>>>> and the rest. If it's not currently the case, then this need to be
>>>>>> fixed. You need to ensure that each clock is modeled properly. If 
>>>>>> some
>>>>>> child clock really depends on multiple parents, then the parents 
>>>>>> need to
>>>>>> in the correct order or CCF need to be taught about such
>>>>>> multi-dependencies.
>>>>>>
>>>>>> If some required feature is missed, then you have to implement it
>>>>>> properly and for all, that's how things are done in upstream. 
>>>>>> Sometimes
>>>>>> it's quite a lot of extra work that everyone are benefiting from in
>>>>>> the end.
>>>>>>
>>>>>> [snip]
>>>>> Yes, we should register ref/parents before their clients.
>>>>>
>>>>> cclk_g clk is registered last after all pll and peripheral clocks are
>>>>> registers during clock init.
>>>>>
>>>>> dfllCPU_out clk is registered later during dfll-fcpu driver probe and
>>>>> gets added to the clock list.
>>>>>
>>>>> Probably the issue seems to be not linking dfll_ref and dfll_soc
>>>>> dependencies for dfllCPU_out thru clock list.
>>>>>
>>>>> clk-dfll driver during dfll_init_clks gets ref_clk and soc_clk 
>>>>> reference
>>>>> thru DT.
>>> The dfll does not have any parents. It has some clocks which are needed
>>> for the logic part of the dfll to function, but there's no parent clock
>>> as such unlike for peripheral clocks or PLLs where the parent is at
>>> least used as a reference. The I2C controller of the DFLL shares the
>>> lines with a normal I2C controller using some arbitration logic. That
>>> logic only works if the clock for the normal I2C controller is enabled.
>>> So you need probably 3 clocks enabled to initialize the dfll in that
>>> case. I don't think it makes sense to add complicated logic to the 
>>> clock
>>> core to deal with this rather strange case. To me it makes more 
>>> sense to
>>> use pmops and open code the sequence there.
>> It looks to me that dfllCPU is a PLL and dfll_ref is its reference
>> parent, while dfll_soc clocks the logic that dynamically reconfigures
>> dfllCPU in background. I see that PLLP is defined as a parent for
>> dfll_ref and dfll_soc in the code. Hence seems dfll_ref should be set as
>> a parent for dfllCPU, no?
>
> dfll_soc will not be restored by the time dfllCPU resume happens after 
> dfll_ref.
>
> without dfll_soc, dfllCPU cannot be resumed either. So if we decide to 
> use parent we should use dfll_soc.
>
>> Either way is good to me, given that DFLL will be disabled during
>> suspend. Resetting DFLL on DFLL's driver resume using PM ops should be
>> good. And then it also will be better to error out if DFLL is active
>> during suspend on the DFLL's driver suspend.
>
> Doing in dfll-fcpu pm_ops is much better as it happens right after all 
> clocks are restored and unlike other clock enables, dfll need dfll 
> controller programming as well and is actually registered in dfll-fcpu 
> driver.
>
> With this, below is the sequence:
>
> CPUFreq suspend switches CPU to PLLP and disables dfll
>
> Will add dfll_suspend/resume in dfll-fcpu driver and in dfll suspend 
> will check for dfll active and will error out suspend.
>
> dfll resume does dfll reinit.
>
> CPUFreq resume enables dfll and switches CPU to dfll.
>
>
> Will go with doing in dfll-fcpu pm_ops rather than parenting 
> dfllCPU_OUT...
>
Does is make sense to return error EBUSY if dfll is not disabled by the 
time dfll-fcpu suspend happens?

Or should I use ETIMEOUT?


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks
  2019-07-18 23:08                                                                                                     ` Sowjanya Komatineni
@ 2019-07-18 23:52                                                                                                       ` Dmitry Osipenko
  0 siblings, 0 replies; 111+ messages in thread
From: Dmitry Osipenko @ 2019-07-18 23:52 UTC (permalink / raw)
  To: Sowjanya Komatineni
  Cc: Peter De Schrijver, sboyd, Michael Turquette, Joseph Lo,
	thierry.reding, jonathanh, tglx, jason, marc.zyngier,
	linus.walleij, stefan, mark.rutland, pgaikwad, linux-clk,
	linux-gpio, jckuo, talho, linux-tegra, linux-kernel, mperttunen,
	spatra, robh+dt, devicetree

В Thu, 18 Jul 2019 16:08:48 -0700
Sowjanya Komatineni <skomatineni@nvidia.com> пишет:

> On 7/18/19 1:36 PM, Sowjanya Komatineni wrote:
> >
> > On 7/18/19 1:26 PM, Dmitry Osipenko wrote:  
> >> 18.07.2019 22:42, Peter De Schrijver пишет:  
> >>> On Thu, Jul 18, 2019 at 02:44:56AM +0300, Dmitry Osipenko wrote:  
> >>>>> dependencies I am referring are dfll_ref, dfll_soc, and DVFS 
> >>>>> peripheral
> >>>>> clocks which need to be restored prior to DFLL reinit.  
> >>>> Okay, but that shouldn't be a problem if clock dependencies are
> >>>> set up properly.
> >>>>  
> >>>>>>> reverse list order during restore might not work as all other 
> >>>>>>> clocks are
> >>>>>>> in proper order no with any ref clocks for plls getting
> >>>>>>> restored prior
> >>>>>>> to their clients  
> >>>>>> Why? The ref clocks should be registered first and be the
> >>>>>> roots for PLLs
> >>>>>> and the rest. If it's not currently the case, then this need
> >>>>>> to be fixed. You need to ensure that each clock is modeled
> >>>>>> properly. If some
> >>>>>> child clock really depends on multiple parents, then the
> >>>>>> parents need to
> >>>>>> in the correct order or CCF need to be taught about such
> >>>>>> multi-dependencies.
> >>>>>>
> >>>>>> If some required feature is missed, then you have to implement
> >>>>>> it properly and for all, that's how things are done in
> >>>>>> upstream. Sometimes
> >>>>>> it's quite a lot of extra work that everyone are benefiting
> >>>>>> from in the end.
> >>>>>>
> >>>>>> [snip]  
> >>>>> Yes, we should register ref/parents before their clients.
> >>>>>
> >>>>> cclk_g clk is registered last after all pll and peripheral
> >>>>> clocks are registers during clock init.
> >>>>>
> >>>>> dfllCPU_out clk is registered later during dfll-fcpu driver
> >>>>> probe and gets added to the clock list.
> >>>>>
> >>>>> Probably the issue seems to be not linking dfll_ref and dfll_soc
> >>>>> dependencies for dfllCPU_out thru clock list.
> >>>>>
> >>>>> clk-dfll driver during dfll_init_clks gets ref_clk and soc_clk 
> >>>>> reference
> >>>>> thru DT.  
> >>> The dfll does not have any parents. It has some clocks which are
> >>> needed for the logic part of the dfll to function, but there's no
> >>> parent clock as such unlike for peripheral clocks or PLLs where
> >>> the parent is at least used as a reference. The I2C controller of
> >>> the DFLL shares the lines with a normal I2C controller using some
> >>> arbitration logic. That logic only works if the clock for the
> >>> normal I2C controller is enabled. So you need probably 3 clocks
> >>> enabled to initialize the dfll in that case. I don't think it
> >>> makes sense to add complicated logic to the clock
> >>> core to deal with this rather strange case. To me it makes more 
> >>> sense to
> >>> use pmops and open code the sequence there.  
> >> It looks to me that dfllCPU is a PLL and dfll_ref is its reference
> >> parent, while dfll_soc clocks the logic that dynamically
> >> reconfigures dfllCPU in background. I see that PLLP is defined as
> >> a parent for dfll_ref and dfll_soc in the code. Hence seems
> >> dfll_ref should be set as a parent for dfllCPU, no?  
> >
> > dfll_soc will not be restored by the time dfllCPU resume happens
> > after dfll_ref.
> >
> > without dfll_soc, dfllCPU cannot be resumed either. So if we decide
> > to use parent we should use dfll_soc.
> >  
> >> Either way is good to me, given that DFLL will be disabled during
> >> suspend. Resetting DFLL on DFLL's driver resume using PM ops
> >> should be good. And then it also will be better to error out if
> >> DFLL is active during suspend on the DFLL's driver suspend.  
> >
> > Doing in dfll-fcpu pm_ops is much better as it happens right after
> > all clocks are restored and unlike other clock enables, dfll need
> > dfll controller programming as well and is actually registered in
> > dfll-fcpu driver.
> >
> > With this, below is the sequence:
> >
> > CPUFreq suspend switches CPU to PLLP and disables dfll
> >
> > Will add dfll_suspend/resume in dfll-fcpu driver and in dfll
> > suspend will check for dfll active and will error out suspend.
> >
> > dfll resume does dfll reinit.
> >
> > CPUFreq resume enables dfll and switches CPU to dfll.
> >
> >
> > Will go with doing in dfll-fcpu pm_ops rather than parenting 
> > dfllCPU_OUT...
> >  
> Does is make sense to return error EBUSY if dfll is not disabled by
> the time dfll-fcpu suspend happens?

Yes

> Or should I use ETIMEOUT?

No

^ permalink raw reply	[flat|nested] 111+ messages in thread

end of thread, other threads:[~2019-07-18 23:49 UTC | newest]

Thread overview: 111+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-28  2:12 [PATCH V5 00/18] SC7 entry and exit support for Tegra210 Sowjanya Komatineni
2019-06-28  2:12 ` [PATCH V5 01/18] irqchip: tegra: Do not disable COP IRQ during suspend Sowjanya Komatineni
2019-06-28  2:12 ` [PATCH V5 02/18] pinctrl: tegra: Add suspend and resume support Sowjanya Komatineni
2019-06-28 11:56   ` Dmitry Osipenko
2019-06-28 12:05     ` Dmitry Osipenko
2019-06-28 23:00       ` Sowjanya Komatineni
2019-06-29 12:38         ` Dmitry Osipenko
2019-06-29 15:40           ` Dmitry Osipenko
2019-06-29 15:46   ` Dmitry Osipenko
2019-06-29 15:58     ` Dmitry Osipenko
2019-06-29 16:28       ` Dmitry Osipenko
2019-07-04  7:31       ` Linus Walleij
2019-07-04 10:40         ` Dmitry Osipenko
2019-07-13  5:31           ` Sowjanya Komatineni
2019-07-14 21:41             ` Dmitry Osipenko
2019-07-13  5:48     ` Sowjanya Komatineni
2019-07-04  7:26   ` Linus Walleij
2019-06-28  2:12 ` [PATCH V5 03/18] clk: tegra: Save and restore divider rate Sowjanya Komatineni
2019-06-28  2:12 ` [PATCH V5 04/18] clk: tegra: pllout: Save and restore pllout context Sowjanya Komatineni
2019-06-28  2:12 ` [PATCH V5 05/18] clk: tegra: pll: Save and restore pll context Sowjanya Komatineni
2019-06-28  2:12 ` [PATCH V5 06/18] clk: tegra: Save and restore CPU and System clocks context Sowjanya Komatineni
2019-06-29 13:33   ` Dmitry Osipenko
2019-06-29 15:26   ` Dmitry Osipenko
2019-06-28  2:12 ` [PATCH V5 07/18] clk: tegra: Support for saving and restoring OSC context Sowjanya Komatineni
2019-06-28  2:12 ` [PATCH V5 08/18] clk: tegra: Add suspend resume support for DFLL Sowjanya Komatineni
2019-06-29 13:28   ` Dmitry Osipenko
2019-06-29 21:45     ` Dmitry Osipenko
2019-06-28  2:12 ` [PATCH V5 09/18] clk: tegra: Add save and restore context support for peripheral clocks Sowjanya Komatineni
2019-06-29 13:17   ` Dmitry Osipenko
2019-06-28  2:12 ` [PATCH V5 10/18] clk: tegra210: Use fence_udelay during PLLU init Sowjanya Komatineni
2019-06-28  2:12 ` [PATCH V5 11/18] clk: tegra210: Add support for Tegra210 clocks Sowjanya Komatineni
2019-06-29 13:14   ` Dmitry Osipenko
2019-06-29 15:10   ` Dmitry Osipenko
2019-07-13  5:54     ` Sowjanya Komatineni
2019-07-14 21:41       ` Dmitry Osipenko
2019-07-16  0:35         ` Sowjanya Komatineni
2019-07-16  3:00           ` Sowjanya Komatineni
2019-07-16  3:41             ` Sowjanya Komatineni
2019-07-16  3:50             ` Dmitry Osipenko
2019-07-16  4:37               ` Sowjanya Komatineni
2019-07-16  5:37                 ` Dmitry Osipenko
2019-07-16  6:20                   ` Dmitry Osipenko
2019-07-16  6:35                   ` Sowjanya Komatineni
2019-07-16  7:24                     ` Joseph Lo
2019-07-16  8:06                       ` Peter De Schrijver
2019-07-16 15:00                         ` Dmitry Osipenko
2019-07-16 16:50                           ` Sowjanya Komatineni
2019-07-16 18:19                             ` Sowjanya Komatineni
2019-07-16 18:25                               ` Dmitry Osipenko
2019-07-16 18:30                                 ` Sowjanya Komatineni
2019-07-16 18:43                                   ` Dmitry Osipenko
2019-07-16 19:26                                     ` Sowjanya Komatineni
2019-07-16 20:47                                       ` Dmitry Osipenko
2019-07-16 21:12                                         ` Sowjanya Komatineni
2019-07-16 21:21                                           ` Dmitry Osipenko
2019-07-16 21:35                                             ` Sowjanya Komatineni
2019-07-16 22:00                                               ` Dmitry Osipenko
2019-07-16 22:06                                                 ` Sowjanya Komatineni
2019-07-17  2:18                                                   ` Sowjanya Komatineni
2019-07-17  2:35                                                     ` Sowjanya Komatineni
2019-07-17  4:11                                                       ` Dmitry Osipenko
     [not found]                                                         ` <77df234f-aa40-0319-a593-f1f19f0f1c2a@nvidia.com>
2019-07-17  5:42                                                           ` Dmitry Osipenko
2019-07-17  5:55                                                             ` Sowjanya Komatineni
2019-07-17  6:33                                                               ` Dmitry Osipenko
2019-07-17  6:36                                                                 ` Sowjanya Komatineni
2019-07-17 15:17                                                                   ` Dmitry Osipenko
2019-07-17 17:29                                                                     ` Sowjanya Komatineni
2019-07-17 18:32                                                                       ` Dmitry Osipenko
2019-07-17 18:51                                                                         ` Sowjanya Komatineni
2019-07-17 18:54                                                                           ` Sowjanya Komatineni
2019-07-17 19:43                                                                             ` Dmitry Osipenko
2019-07-17 20:01                                                                               ` Sowjanya Komatineni
2019-07-17 20:11                                                                                 ` Sowjanya Komatineni
2019-07-17 21:29                                                                                   ` Sowjanya Komatineni
2019-07-17 21:30                                                                                   ` Dmitry Osipenko
2019-07-17 21:51                                                                                     ` Sowjanya Komatineni
2019-07-17 21:57                                                                                       ` Sowjanya Komatineni
2019-07-17 22:48                                                                                         ` Dmitry Osipenko
2019-07-17 23:36                                                                                           ` Sowjanya Komatineni
2019-07-17 23:44                                                                                             ` Dmitry Osipenko
2019-07-18  0:25                                                                                               ` Sowjanya Komatineni
2019-07-18  1:15                                                                                                 ` Sowjanya Komatineni
2019-07-18 16:34                                                                                                   ` Dmitry Osipenko
2019-07-18 17:22                                                                                                     ` Sowjanya Komatineni
2019-07-18 17:41                                                                                                       ` Sowjanya Komatineni
2019-07-18 18:29                                                                                                         ` Sowjanya Komatineni
2019-07-18 19:11                                                                                                           ` Sowjanya Komatineni
2019-07-18 19:42                                                                                               ` Peter De Schrijver
2019-07-18 20:26                                                                                                 ` Dmitry Osipenko
2019-07-18 20:36                                                                                                   ` Sowjanya Komatineni
2019-07-18 22:52                                                                                                     ` Dmitry Osipenko
2019-07-18 23:08                                                                                                     ` Sowjanya Komatineni
2019-07-18 23:52                                                                                                       ` Dmitry Osipenko
2019-07-17  3:54                                                     ` Dmitry Osipenko
2019-07-17  4:01                                                       ` Sowjanya Komatineni
2019-07-18 19:18                                     ` Peter De Schrijver
2019-07-18 19:24                                       ` Sowjanya Komatineni
2019-07-18 20:11                                         ` Dmitry Osipenko
2019-07-18 20:32                                           ` Dmitry Osipenko
2019-07-18 19:15                                 ` Peter De Schrijver
2019-06-29 15:13   ` Dmitry Osipenko
2019-06-28  2:12 ` [PATCH V5 12/18] soc/tegra: pmc: Allow support for more tegra wake Sowjanya Komatineni
2019-06-28  2:12 ` [PATCH V5 13/18] soc/tegra: pmc: Add pmc wake support for tegra210 Sowjanya Komatineni
2019-06-29 13:11   ` Dmitry Osipenko
2019-06-28  2:12 ` [PATCH V5 14/18] arm64: tegra: Enable wake from deep sleep on RTC alarm Sowjanya Komatineni
2019-06-28  2:12 ` [PATCH V5 15/18] soc/tegra: pmc: Configure core power request polarity Sowjanya Komatineni
2019-06-28  2:12 ` [PATCH V5 16/18] soc/tegra: pmc: Configure deep sleep control settings Sowjanya Komatineni
2019-06-29 13:00   ` Dmitry Osipenko
2019-06-29 13:02   ` Dmitry Osipenko
2019-06-28  2:12 ` [PATCH V5 17/18] arm64: dts: tegra210-p2180: Jetson TX1 SC7 timings Sowjanya Komatineni
2019-06-28  2:12 ` [PATCH V5 18/18] arm64: dts: tegra210-p3450: Jetson nano " Sowjanya Komatineni

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).