[PATCH 0/5] cpuidle: psci: Various improvements for PSCI PM domains

[PATCH 0/5] cpuidle: psci: Various improvements for PSCI PM domains

[Ulf Hansson]

The main change in this series is done in patch 5/5, which implements support
to prevent domain idlestates until all PSCI PM domain consumers are ready for
it. To reach that point the corresponding code for cpuidle-psci and
cpuidle-psci-domain, needed to be converted into platform drivers, which is
done by the earlier changes in the series.

Additionally, some improvements have been made to the error path, which becomes
easier when the code gets converted to platform drivers.

Deployment for a Qcom SoC, which actually takes full benefit of these changes
are also in the pipe, but deferring then a bit until $subject series have been
discussed.

Kind regards
Ulf Hansson

Ulf Hansson (5):
  cpuidle: psci: Fail cpuidle registration if set OSI mode failed
  cpuidle: psci: Fix error path via converting to a platform driver
  cpuidle: psci: Split into two separate build objects
  cpuidle: psci: Convert PM domain to platform driver
  cpuidle: psci: Prevent domain idlestates until consumers are ready

 drivers/cpuidle/Kconfig.arm           |  10 ++
 drivers/cpuidle/Makefile              |   5 +-
 drivers/cpuidle/cpuidle-psci-domain.c |  74 +++++++++-----
 drivers/cpuidle/cpuidle-psci.c        | 141 +++++++++++++++-----------
 drivers/cpuidle/cpuidle-psci.h        |  11 +-
 5 files changed, 150 insertions(+), 91 deletions(-)

-- 
2.20.1

[PATCH 1/5] cpuidle: psci: Fail cpuidle registration if set OSI mode failed

[Ulf Hansson]

Currently we allow the cpuidle driver registration to succeed, even if we
failed to enable the OSI mode when the hierarchical DT layout is used. This
means running in a degraded mode, by using the available idle states per
CPU, while also preventing the domain idle states.

Moving forward, this behaviour looks quite questionable to maintain, as
complexity seems to grow around it, especially when trying to add support
for deferred probe, for example.

Therefore, let's make the cpuidle driver registration to fail in this
situation, thus relying on the default architectural cpuidle backend for
WFI to be used.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/cpuidle/cpuidle-psci-domain.c | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/drivers/cpuidle/cpuidle-psci-domain.c b/drivers/cpuidle/cpuidle-psci-domain.c
index 423f03bbeb74..f07786aad673 100644
--- a/drivers/cpuidle/cpuidle-psci-domain.c
+++ b/drivers/cpuidle/cpuidle-psci-domain.c
@@ -26,7 +26,6 @@ struct psci_pd_provider {
 };
 
 static LIST_HEAD(psci_pd_providers);
-static bool osi_mode_enabled __initdata;
 
 static int psci_pd_power_off(struct generic_pm_domain *pd)
 {
@@ -272,7 +271,6 @@ static int __init psci_idle_init_domains(void)
 		goto remove_pd;
 	}
 
-	osi_mode_enabled = true;
 	of_node_put(np);
 	pr_info("Initialized CPU PM domain topology\n");
 	return pd_count;
@@ -293,9 +291,6 @@ struct device __init *psci_dt_attach_cpu(int cpu)
 {
 	struct device *dev;
 
-	if (!osi_mode_enabled)
-		return NULL;
-
 	dev = dev_pm_domain_attach_by_name(get_cpu_device(cpu), "psci");
 	if (IS_ERR_OR_NULL(dev))
 		return dev;
-- 
2.20.1

Re: [PATCH 1/5] cpuidle: psci: Fail cpuidle registration if set OSI mode failed

[Lina Iyer]

On Mon, Jun 15 2020 at 09:21 -0600, Ulf Hansson wrote:
>Currently we allow the cpuidle driver registration to succeed, even if we
>failed to enable the OSI mode when the hierarchical DT layout is used. This
>means running in a degraded mode, by using the available idle states per
>CPU, while also preventing the domain idle states.
>
>Moving forward, this behaviour looks quite questionable to maintain, as
>complexity seems to grow around it, especially when trying to add support
>for deferred probe, for example.
>
>Therefore, let's make the cpuidle driver registration to fail in this
>situation, thus relying on the default architectural cpuidle backend for
>WFI to be used.
>
>Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>

May be PATCH 3/5 should come before this change, but for this patch itself,
please consider -

Reviewed-by: Lina Iyer <ilina@codeaurora.org>

>---
> drivers/cpuidle/cpuidle-psci-domain.c | 5 -----
> 1 file changed, 5 deletions(-)
>
>diff --git a/drivers/cpuidle/cpuidle-psci-domain.c b/drivers/cpuidle/cpuidle-psci-domain.c
>index 423f03bbeb74..f07786aad673 100644
>--- a/drivers/cpuidle/cpuidle-psci-domain.c
>+++ b/drivers/cpuidle/cpuidle-psci-domain.c
>@@ -26,7 +26,6 @@ struct psci_pd_provider {
> };
>
> static LIST_HEAD(psci_pd_providers);
>-static bool osi_mode_enabled __initdata;
>
> static int psci_pd_power_off(struct generic_pm_domain *pd)
> {
>@@ -272,7 +271,6 @@ static int __init psci_idle_init_domains(void)
> 		goto remove_pd;
> 	}
>
>-	osi_mode_enabled = true;
> 	of_node_put(np);
> 	pr_info("Initialized CPU PM domain topology\n");
> 	return pd_count;
>@@ -293,9 +291,6 @@ struct device __init *psci_dt_attach_cpu(int cpu)
> {
> 	struct device *dev;
>
>-	if (!osi_mode_enabled)
>-		return NULL;
>-
> 	dev = dev_pm_domain_attach_by_name(get_cpu_device(cpu), "psci");
> 	if (IS_ERR_OR_NULL(dev))
> 		return dev;
>-- 
>2.20.1
>

Re: [PATCH 1/5] cpuidle: psci: Fail cpuidle registration if set OSI mode failed

[Sudeep Holla]

On Mon, Jun 15, 2020 at 05:20:50PM +0200, Ulf Hansson wrote:
> Currently we allow the cpuidle driver registration to succeed, even if we
> failed to enable the OSI mode when the hierarchical DT layout is used. This
> means running in a degraded mode, by using the available idle states per
> CPU, while also preventing the domain idle states.
>

Is that not better than not registering itself ? I tend to disagree here.

> Moving forward, this behaviour looks quite questionable to maintain, as
> complexity seems to grow around it, especially when trying to add support
> for deferred probe, for example.
>

I thought the sync_state in the driver must deal with that.

> Therefore, let's make the cpuidle driver registration to fail in this
> situation, thus relying on the default architectural cpuidle backend for
> WFI to be used.
>

CPU level states work w/o the need of OSI, and better than WFI. That was
the original aim. If that is not working, we must make it work instead
of falling back on WFI IMO.

--
Regards,
Sudeep

Re: [PATCH 1/5] cpuidle: psci: Fail cpuidle registration if set OSI mode failed

[Sudeep Holla]

On Fri, Jun 26, 2020 at 03:33:55PM +0100, Sudeep Holla wrote:
> On Mon, Jun 15, 2020 at 05:20:50PM +0200, Ulf Hansson wrote:
> > Currently we allow the cpuidle driver registration to succeed, even if we
> > failed to enable the OSI mode when the hierarchical DT layout is used. This
> > means running in a degraded mode, by using the available idle states per
> > CPU, while also preventing the domain idle states.
> >
> 
> Is that not better than not registering itself ? I tend to disagree here.
> 
> > Moving forward, this behaviour looks quite questionable to maintain, as
> > complexity seems to grow around it, especially when trying to add support
> > for deferred probe, for example.
> >
> 
> I thought the sync_state in the driver must deal with that.
> 

Ignore this I got confused and assumed we have this in the code while I
read 5/5 first 🙁

-- 
Regards,
Sudeep

[PATCH 2/5] cpuidle: psci: Fix error path via converting to a platform driver

[Ulf Hansson]

The current error paths for the cpuidle-psci driver, may leak memory or
possibly leave CPU devices attached to their PM domains. These are quite
harmless issues, but still deserves to be taken care of.

Although, rather than fixing them by keeping track of allocations that
needs to be freed, which tends to become a bit messy, let's convert into a
platform driver. In this way, it gets easier to fix the memory leaks as we
can rely on the devm_* functions.

Moreover, converting to a platform driver also enables support for deferred
probe, which subsequent changes takes benefit from.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/cpuidle/cpuidle-psci-domain.c |  10 +-
 drivers/cpuidle/cpuidle-psci.c        | 141 +++++++++++++++-----------
 drivers/cpuidle/cpuidle-psci.h        |   9 +-
 3 files changed, 95 insertions(+), 65 deletions(-)

diff --git a/drivers/cpuidle/cpuidle-psci-domain.c b/drivers/cpuidle/cpuidle-psci-domain.c
index f07786aad673..e48e578aaa7d 100644
--- a/drivers/cpuidle/cpuidle-psci-domain.c
+++ b/drivers/cpuidle/cpuidle-psci-domain.c
@@ -287,7 +287,7 @@ static int __init psci_idle_init_domains(void)
 }
 subsys_initcall(psci_idle_init_domains);
 
-struct device __init *psci_dt_attach_cpu(int cpu)
+struct device *psci_dt_attach_cpu(int cpu)
 {
 	struct device *dev;
 
@@ -301,3 +301,11 @@ struct device __init *psci_dt_attach_cpu(int cpu)
 
 	return dev;
 }
+
+void psci_dt_detach_cpu(struct device *dev)
+{
+	if (IS_ERR_OR_NULL(dev))
+		return;
+
+	dev_pm_domain_detach(dev, false);
+}
diff --git a/drivers/cpuidle/cpuidle-psci.c b/drivers/cpuidle/cpuidle-psci.c
index 3806f911b61c..74463841805f 100644
--- a/drivers/cpuidle/cpuidle-psci.c
+++ b/drivers/cpuidle/cpuidle-psci.c
@@ -17,9 +17,11 @@
 #include <linux/module.h>
 #include <linux/of.h>
 #include <linux/of_device.h>
+#include <linux/platform_device.h>
 #include <linux/psci.h>
 #include <linux/pm_runtime.h>
 #include <linux/slab.h>
+#include <linux/string.h>
 
 #include <asm/cpuidle.h>
 
@@ -33,7 +35,7 @@ struct psci_cpuidle_data {
 
 static DEFINE_PER_CPU_READ_MOSTLY(struct psci_cpuidle_data, psci_cpuidle_data);
 static DEFINE_PER_CPU(u32, domain_state);
-static bool psci_cpuidle_use_cpuhp __initdata;
+static bool psci_cpuidle_use_cpuhp;
 
 void psci_set_domain_state(u32 state)
 {
@@ -104,7 +106,7 @@ static int psci_idle_cpuhp_down(unsigned int cpu)
 	return 0;
 }
 
-static void __init psci_idle_init_cpuhp(void)
+static void psci_idle_init_cpuhp(void)
 {
 	int err;
 
@@ -127,30 +129,13 @@ static int psci_enter_idle_state(struct cpuidle_device *dev,
 	return psci_enter_state(idx, state[idx]);
 }
 
-static struct cpuidle_driver psci_idle_driver __initdata = {
-	.name = "psci_idle",
-	.owner = THIS_MODULE,
-	/*
-	 * PSCI idle states relies on architectural WFI to
-	 * be represented as state index 0.
-	 */
-	.states[0] = {
-		.enter                  = psci_enter_idle_state,
-		.exit_latency           = 1,
-		.target_residency       = 1,
-		.power_usage		= UINT_MAX,
-		.name                   = "WFI",
-		.desc                   = "ARM WFI",
-	}
-};
-
-static const struct of_device_id psci_idle_state_match[] __initconst = {
+static const struct of_device_id psci_idle_state_match[] = {
 	{ .compatible = "arm,idle-state",
 	  .data = psci_enter_idle_state },
 	{ },
 };
 
-int __init psci_dt_parse_state_node(struct device_node *np, u32 *state)
+int psci_dt_parse_state_node(struct device_node *np, u32 *state)
 {
 	int err = of_property_read_u32(np, "arm,psci-suspend-param", state);
 
@@ -167,9 +152,9 @@ int __init psci_dt_parse_state_node(struct device_node *np, u32 *state)
 	return 0;
 }
 
-static int __init psci_dt_cpu_init_topology(struct cpuidle_driver *drv,
-					    struct psci_cpuidle_data *data,
-					    unsigned int state_count, int cpu)
+static int psci_dt_cpu_init_topology(struct cpuidle_driver *drv,
+				     struct psci_cpuidle_data *data,
+				     unsigned int state_count, int cpu)
 {
 	/* Currently limit the hierarchical topology to be used in OSI mode. */
 	if (!psci_has_osi_support())
@@ -190,9 +175,9 @@ static int __init psci_dt_cpu_init_topology(struct cpuidle_driver *drv,
 	return 0;
 }
 
-static int __init psci_dt_cpu_init_idle(struct cpuidle_driver *drv,
-					struct device_node *cpu_node,
-					unsigned int state_count, int cpu)
+static int psci_dt_cpu_init_idle(struct device *dev, struct cpuidle_driver *drv,
+				 struct device_node *cpu_node,
+				 unsigned int state_count, int cpu)
 {
 	int i, ret = 0;
 	u32 *psci_states;
@@ -200,7 +185,8 @@ static int __init psci_dt_cpu_init_idle(struct cpuidle_driver *drv,
 	struct psci_cpuidle_data *data = per_cpu_ptr(&psci_cpuidle_data, cpu);
 
 	state_count++; /* Add WFI state too */
-	psci_states = kcalloc(state_count, sizeof(*psci_states), GFP_KERNEL);
+	psci_states = devm_kcalloc(dev, state_count, sizeof(*psci_states),
+				   GFP_KERNEL);
 	if (!psci_states)
 		return -ENOMEM;
 
@@ -213,32 +199,26 @@ static int __init psci_dt_cpu_init_idle(struct cpuidle_driver *drv,
 		of_node_put(state_node);
 
 		if (ret)
-			goto free_mem;
+			return ret;
 
 		pr_debug("psci-power-state %#x index %d\n", psci_states[i], i);
 	}
 
-	if (i != state_count) {
-		ret = -ENODEV;
-		goto free_mem;
-	}
+	if (i != state_count)
+		return -ENODEV;
 
 	/* Initialize optional data, used for the hierarchical topology. */
 	ret = psci_dt_cpu_init_topology(drv, data, state_count, cpu);
 	if (ret < 0)
-		goto free_mem;
+		return ret;
 
 	/* Idle states parsed correctly, store them in the per-cpu struct. */
 	data->psci_states = psci_states;
 	return 0;
-
-free_mem:
-	kfree(psci_states);
-	return ret;
 }
 
-static __init int psci_cpu_init_idle(struct cpuidle_driver *drv,
-				     unsigned int cpu, unsigned int state_count)
+static int psci_cpu_init_idle(struct device *dev, struct cpuidle_driver *drv,
+			      unsigned int cpu, unsigned int state_count)
 {
 	struct device_node *cpu_node;
 	int ret;
@@ -254,14 +234,22 @@ static __init int psci_cpu_init_idle(struct cpuidle_driver *drv,
 	if (!cpu_node)
 		return -ENODEV;
 
-	ret = psci_dt_cpu_init_idle(drv, cpu_node, state_count, cpu);
+	ret = psci_dt_cpu_init_idle(dev, drv, cpu_node, state_count, cpu);
 
 	of_node_put(cpu_node);
 
 	return ret;
 }
 
-static int __init psci_idle_init_cpu(int cpu)
+static void psci_cpu_deinit_idle(int cpu)
+{
+	struct psci_cpuidle_data *data = per_cpu_ptr(&psci_cpuidle_data, cpu);
+
+	psci_dt_detach_cpu(data->dev);
+	psci_cpuidle_use_cpuhp = false;
+}
+
+static int psci_idle_init_cpu(struct device *dev, int cpu)
 {
 	struct cpuidle_driver *drv;
 	struct device_node *cpu_node;
@@ -284,17 +272,26 @@ static int __init psci_idle_init_cpu(int cpu)
 	if (ret)
 		return ret;
 
-	drv = kmemdup(&psci_idle_driver, sizeof(*drv), GFP_KERNEL);
+	drv = devm_kzalloc(dev, sizeof(*drv), GFP_KERNEL);
 	if (!drv)
 		return -ENOMEM;
 
+	drv->name = "psci_idle";
+	drv->owner = THIS_MODULE;
 	drv->cpumask = (struct cpumask *)cpumask_of(cpu);
 
 	/*
-	 * Initialize idle states data, starting at index 1, since
-	 * by default idle state 0 is the quiescent state reached
-	 * by the cpu by executing the wfi instruction.
-	 *
+	 * PSCI idle states relies on architectural WFI to be represented as
+	 * state index 0.
+	 */
+	drv->states[0].enter = psci_enter_idle_state;
+	drv->states[0].exit_latency = 1;
+	drv->states[0].target_residency = 1;
+	drv->states[0].power_usage = UINT_MAX;
+	strcpy(drv->states[0].name, "WFI");
+	strcpy(drv->states[0].desc, "ARM WFI");
+
+	/*
 	 * If no DT idle states are detected (ret == 0) let the driver
 	 * initialization fail accordingly since there is no reason to
 	 * initialize the idle driver if only wfi is supported, the
@@ -302,48 +299,45 @@ static int __init psci_idle_init_cpu(int cpu)
 	 * on idle entry.
 	 */
 	ret = dt_init_idle_driver(drv, psci_idle_state_match, 1);
-	if (ret <= 0) {
-		ret = ret ? : -ENODEV;
-		goto out_kfree_drv;
-	}
+	if (ret <= 0)
+		return ret ? : -ENODEV;
 
 	/*
 	 * Initialize PSCI idle states.
 	 */
-	ret = psci_cpu_init_idle(drv, cpu, ret);
+	ret = psci_cpu_init_idle(dev, drv, cpu, ret);
 	if (ret) {
 		pr_err("CPU %d failed to PSCI idle\n", cpu);
-		goto out_kfree_drv;
+		return ret;
 	}
 
 	ret = cpuidle_register(drv, NULL);
 	if (ret)
-		goto out_kfree_drv;
+		goto deinit;
 
 	cpuidle_cooling_register(drv);
 
 	return 0;
-
-out_kfree_drv:
-	kfree(drv);
+deinit:
+	psci_cpu_deinit_idle(cpu);
 	return ret;
 }
 
 /*
- * psci_idle_init - Initializes PSCI cpuidle driver
+ * psci_idle_probe - Initializes PSCI cpuidle driver
  *
  * Initializes PSCI cpuidle driver for all CPUs, if any CPU fails
  * to register cpuidle driver then rollback to cancel all CPUs
  * registration.
  */
-static int __init psci_idle_init(void)
+static int psci_cpuidle_probe(struct platform_device *pdev)
 {
 	int cpu, ret;
 	struct cpuidle_driver *drv;
 	struct cpuidle_device *dev;
 
 	for_each_possible_cpu(cpu) {
-		ret = psci_idle_init_cpu(cpu);
+		ret = psci_idle_init_cpu(&pdev->dev, cpu);
 		if (ret)
 			goto out_fail;
 	}
@@ -356,9 +350,34 @@ static int __init psci_idle_init(void)
 		dev = per_cpu(cpuidle_devices, cpu);
 		drv = cpuidle_get_cpu_driver(dev);
 		cpuidle_unregister(drv);
-		kfree(drv);
+		psci_cpu_deinit_idle(cpu);
 	}
 
 	return ret;
 }
+
+static struct platform_driver psci_cpuidle_driver = {
+	.probe = psci_cpuidle_probe,
+	.driver = {
+		.name = "psci-cpuidle",
+	},
+};
+
+static int __init psci_idle_init(void)
+{
+	struct platform_device *pdev;
+	int ret;
+
+	ret = platform_driver_register(&psci_cpuidle_driver);
+	if (ret)
+		return ret;
+
+	pdev = platform_device_register_simple("psci-cpuidle", -1, NULL, 0);
+	if (IS_ERR(pdev)) {
+		platform_driver_unregister(&psci_cpuidle_driver);
+		return PTR_ERR(pdev);
+	}
+
+	return 0;
+}
 device_initcall(psci_idle_init);
diff --git a/drivers/cpuidle/cpuidle-psci.h b/drivers/cpuidle/cpuidle-psci.h
index 7299a04dd467..0690d66df829 100644
--- a/drivers/cpuidle/cpuidle-psci.h
+++ b/drivers/cpuidle/cpuidle-psci.h
@@ -3,15 +3,18 @@
 #ifndef __CPUIDLE_PSCI_H
 #define __CPUIDLE_PSCI_H
 
+struct device;
 struct device_node;
 
 void psci_set_domain_state(u32 state);
-int __init psci_dt_parse_state_node(struct device_node *np, u32 *state);
+int psci_dt_parse_state_node(struct device_node *np, u32 *state);
 
 #ifdef CONFIG_PM_GENERIC_DOMAINS_OF
-struct device __init *psci_dt_attach_cpu(int cpu);
+struct device *psci_dt_attach_cpu(int cpu);
+void psci_dt_detach_cpu(struct device *dev);
 #else
-static inline struct device __init *psci_dt_attach_cpu(int cpu) { return NULL; }
+static inline struct device *psci_dt_attach_cpu(int cpu) { return NULL; }
+static inline void psci_dt_detach_cpu(struct device *dev) { }
 #endif
 
 #endif /* __CPUIDLE_PSCI_H */
-- 
2.20.1

Re: [PATCH 2/5] cpuidle: psci: Fix error path via converting to a platform driver

[Sudeep Holla]

On Mon, Jun 15, 2020 at 05:20:51PM +0200, Ulf Hansson wrote:
> The current error paths for the cpuidle-psci driver, may leak memory or
> possibly leave CPU devices attached to their PM domains. These are quite
> harmless issues, but still deserves to be taken care of.
>
> Although, rather than fixing them by keeping track of allocations that
> needs to be freed, which tends to become a bit messy, let's convert into a
> platform driver. In this way, it gets easier to fix the memory leaks as we
> can rely on the devm_* functions.
>
> Moreover, converting to a platform driver also enables support for deferred
> probe, which subsequent changes takes benefit from.
>

Though I don't have strong opinion, I don't like platform device for cpuidle.
But that's not main issue. I am more worried about the need for whole
deferred probe for cpuidle. Is this due to OSI and Qcom dependencies ?
Ideally, the firmware is already to accept suspend calls soon after they
are powered on in psci f/w.

--
Regards,
Sudeep

Re: [PATCH 2/5] cpuidle: psci: Fix error path via converting to a platform driver

[Ulf Hansson]

On Fri, 26 Jun 2020 at 16:42, Sudeep Holla <sudeep.holla@arm.com> wrote:
>
> On Mon, Jun 15, 2020 at 05:20:51PM +0200, Ulf Hansson wrote:
> > The current error paths for the cpuidle-psci driver, may leak memory or
> > possibly leave CPU devices attached to their PM domains. These are quite
> > harmless issues, but still deserves to be taken care of.
> >
> > Although, rather than fixing them by keeping track of allocations that
> > needs to be freed, which tends to become a bit messy, let's convert into a
> > platform driver. In this way, it gets easier to fix the memory leaks as we
> > can rely on the devm_* functions.
> >
> > Moreover, converting to a platform driver also enables support for deferred
> > probe, which subsequent changes takes benefit from.
> >
>
> Though I don't have strong opinion, I don't like platform device for cpuidle.
> But that's not main issue. I am more worried about the need for whole
> deferred probe for cpuidle. Is this due to OSI and Qcom dependencies ?
> Ideally, the firmware is already to accept suspend calls soon after they
> are powered on in psci f/w.

I agree that a platform device for cpuidle isn't very nice, but that's
the best I could figure out to use. If you have a better idea, please
tell me.

From the PSCI FW point of view you are right. Everything is ready.

So, yes, as you stated, this boils down to the initialization of the
PSCI PM domains, registered in cpuidle-psci-domain.c, which I am
converting to platform driver in patch4. That change means that
cpuidle-psci-domain driver are about to support deferred probe (and
more importantly the ->sync_state() callback), which means that when
psci_dt_attach_cpu() is called, it may return -EPROBE_DEFER - hence we
should cope with it.

I hope this further clarifies the reason behind $subject patch!?

Thanks for reviewing!

Kind regards
Uffe

[PATCH 3/5] cpuidle: psci: Split into two separate build objects

[Ulf Hansson]

The combined build object for the PSCI cpuidle driver and the PSCI PM
domain, is a bit messy. Therefore let's split it up by adding a new Kconfig
ARM_PSCI_CPUIDLE_DOMAIN and convert into two separate objects.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/cpuidle/Kconfig.arm    | 10 ++++++++++
 drivers/cpuidle/Makefile       |  5 ++---
 drivers/cpuidle/cpuidle-psci.h |  2 +-
 3 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/cpuidle/Kconfig.arm b/drivers/cpuidle/Kconfig.arm
index 51a7e89085c0..0844fadc4be8 100644
--- a/drivers/cpuidle/Kconfig.arm
+++ b/drivers/cpuidle/Kconfig.arm
@@ -23,6 +23,16 @@ config ARM_PSCI_CPUIDLE
 	  It provides an idle driver that is capable of detecting and
 	  managing idle states through the PSCI firmware interface.
 
+config ARM_PSCI_CPUIDLE_DOMAIN
+	bool "PSCI CPU idle Domain"
+	depends on ARM_PSCI_CPUIDLE
+	depends on PM_GENERIC_DOMAINS_OF
+	default y
+	help
+	  Select this to enable the PSCI based CPUidle driver to use PM domains,
+	  which is needed to support the hierarchical DT based layout of the
+	  idle states.
+
 config ARM_BIG_LITTLE_CPUIDLE
 	bool "Support for ARM big.LITTLE processors"
 	depends on ARCH_VEXPRESS_TC2_PM || ARCH_EXYNOS || COMPILE_TEST
diff --git a/drivers/cpuidle/Makefile b/drivers/cpuidle/Makefile
index f07800cbb43f..26bbc5e74123 100644
--- a/drivers/cpuidle/Makefile
+++ b/drivers/cpuidle/Makefile
@@ -21,9 +21,8 @@ obj-$(CONFIG_ARM_U8500_CPUIDLE)         += cpuidle-ux500.o
 obj-$(CONFIG_ARM_AT91_CPUIDLE)          += cpuidle-at91.o
 obj-$(CONFIG_ARM_EXYNOS_CPUIDLE)        += cpuidle-exynos.o
 obj-$(CONFIG_ARM_CPUIDLE)		+= cpuidle-arm.o
-obj-$(CONFIG_ARM_PSCI_CPUIDLE)		+= cpuidle_psci.o
-cpuidle_psci-y				:= cpuidle-psci.o
-cpuidle_psci-$(CONFIG_PM_GENERIC_DOMAINS_OF) += cpuidle-psci-domain.o
+obj-$(CONFIG_ARM_PSCI_CPUIDLE)		+= cpuidle-psci.o
+obj-$(CONFIG_ARM_PSCI_CPUIDLE_DOMAIN)	+= cpuidle-psci-domain.o
 obj-$(CONFIG_ARM_TEGRA_CPUIDLE)		+= cpuidle-tegra.o
 obj-$(CONFIG_ARM_QCOM_SPM_CPUIDLE)	+= cpuidle-qcom-spm.o
 
diff --git a/drivers/cpuidle/cpuidle-psci.h b/drivers/cpuidle/cpuidle-psci.h
index 0690d66df829..d8e925e84c27 100644
--- a/drivers/cpuidle/cpuidle-psci.h
+++ b/drivers/cpuidle/cpuidle-psci.h
@@ -9,7 +9,7 @@ struct device_node;
 void psci_set_domain_state(u32 state);
 int psci_dt_parse_state_node(struct device_node *np, u32 *state);
 
-#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
+#ifdef CONFIG_ARM_PSCI_CPUIDLE_DOMAIN
 struct device *psci_dt_attach_cpu(int cpu);
 void psci_dt_detach_cpu(struct device *dev);
 #else
-- 
2.20.1

Re: [PATCH 3/5] cpuidle: psci: Split into two separate build objects

[Lina Iyer]

On Mon, Jun 15 2020 at 09:21 -0600, Ulf Hansson wrote:
>The combined build object for the PSCI cpuidle driver and the PSCI PM
>domain, is a bit messy. Therefore let's split it up by adding a new Kconfig
>ARM_PSCI_CPUIDLE_DOMAIN and convert into two separate objects.
>
>Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
Reviewed-by: Lina Iyer <ilina@codeaurora.org>

>---
> drivers/cpuidle/Kconfig.arm    | 10 ++++++++++
> drivers/cpuidle/Makefile       |  5 ++---
> drivers/cpuidle/cpuidle-psci.h |  2 +-
> 3 files changed, 13 insertions(+), 4 deletions(-)
>
>diff --git a/drivers/cpuidle/Kconfig.arm b/drivers/cpuidle/Kconfig.arm
>index 51a7e89085c0..0844fadc4be8 100644
>--- a/drivers/cpuidle/Kconfig.arm
>+++ b/drivers/cpuidle/Kconfig.arm
>@@ -23,6 +23,16 @@ config ARM_PSCI_CPUIDLE
> 	  It provides an idle driver that is capable of detecting and
> 	  managing idle states through the PSCI firmware interface.
>
>+config ARM_PSCI_CPUIDLE_DOMAIN
>+	bool "PSCI CPU idle Domain"
>+	depends on ARM_PSCI_CPUIDLE
>+	depends on PM_GENERIC_DOMAINS_OF
>+	default y
>+	help
>+	  Select this to enable the PSCI based CPUidle driver to use PM domains,
>+	  which is needed to support the hierarchical DT based layout of the
>+	  idle states.
>+
> config ARM_BIG_LITTLE_CPUIDLE
> 	bool "Support for ARM big.LITTLE processors"
> 	depends on ARCH_VEXPRESS_TC2_PM || ARCH_EXYNOS || COMPILE_TEST
>diff --git a/drivers/cpuidle/Makefile b/drivers/cpuidle/Makefile
>index f07800cbb43f..26bbc5e74123 100644
>--- a/drivers/cpuidle/Makefile
>+++ b/drivers/cpuidle/Makefile
>@@ -21,9 +21,8 @@ obj-$(CONFIG_ARM_U8500_CPUIDLE)         += cpuidle-ux500.o
> obj-$(CONFIG_ARM_AT91_CPUIDLE)          += cpuidle-at91.o
> obj-$(CONFIG_ARM_EXYNOS_CPUIDLE)        += cpuidle-exynos.o
> obj-$(CONFIG_ARM_CPUIDLE)		+= cpuidle-arm.o
>-obj-$(CONFIG_ARM_PSCI_CPUIDLE)		+= cpuidle_psci.o
>-cpuidle_psci-y				:= cpuidle-psci.o
>-cpuidle_psci-$(CONFIG_PM_GENERIC_DOMAINS_OF) += cpuidle-psci-domain.o
>+obj-$(CONFIG_ARM_PSCI_CPUIDLE)		+= cpuidle-psci.o
>+obj-$(CONFIG_ARM_PSCI_CPUIDLE_DOMAIN)	+= cpuidle-psci-domain.o
> obj-$(CONFIG_ARM_TEGRA_CPUIDLE)		+= cpuidle-tegra.o
> obj-$(CONFIG_ARM_QCOM_SPM_CPUIDLE)	+= cpuidle-qcom-spm.o
>
>diff --git a/drivers/cpuidle/cpuidle-psci.h b/drivers/cpuidle/cpuidle-psci.h
>index 0690d66df829..d8e925e84c27 100644
>--- a/drivers/cpuidle/cpuidle-psci.h
>+++ b/drivers/cpuidle/cpuidle-psci.h
>@@ -9,7 +9,7 @@ struct device_node;
> void psci_set_domain_state(u32 state);
> int psci_dt_parse_state_node(struct device_node *np, u32 *state);
>
>-#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
>+#ifdef CONFIG_ARM_PSCI_CPUIDLE_DOMAIN
> struct device *psci_dt_attach_cpu(int cpu);
> void psci_dt_detach_cpu(struct device *dev);
> #else
>-- 
>2.20.1
>

Re: [PATCH 3/5] cpuidle: psci: Split into two separate build objects

[Sudeep Holla]

On Mon, Jun 15, 2020 at 05:20:52PM +0200, Ulf Hansson wrote:
> The combined build object for the PSCI cpuidle driver and the PSCI PM
> domain, is a bit messy. Therefore let's split it up by adding a new Kconfig
> ARM_PSCI_CPUIDLE_DOMAIN and convert into two separate objects.
>

Reviewed-by: Sudeep Holla <sudeep.holla@arm.com>

-- 
Regards,
Sudeep

[PATCH 4/5] cpuidle: psci: Convert PM domain to platform driver

[Ulf Hansson]

To enable support for deferred probing and to allow implementation of the
->sync_state() callback from subsequent changes, let's convert into a
platform driver.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/cpuidle/cpuidle-psci-domain.c | 45 +++++++++++++++++----------
 1 file changed, 28 insertions(+), 17 deletions(-)

diff --git a/drivers/cpuidle/cpuidle-psci-domain.c b/drivers/cpuidle/cpuidle-psci-domain.c
index e48e578aaa7d..bf527d2bb4b6 100644
--- a/drivers/cpuidle/cpuidle-psci-domain.c
+++ b/drivers/cpuidle/cpuidle-psci-domain.c
@@ -12,6 +12,7 @@
 #include <linux/cpu.h>
 #include <linux/device.h>
 #include <linux/kernel.h>
+#include <linux/platform_device.h>
 #include <linux/pm_domain.h>
 #include <linux/pm_runtime.h>
 #include <linux/psci.h>
@@ -42,8 +43,8 @@ static int psci_pd_power_off(struct generic_pm_domain *pd)
 	return 0;
 }
 
-static int __init psci_pd_parse_state_nodes(struct genpd_power_state *states,
-					int state_count)
+static int psci_pd_parse_state_nodes(struct genpd_power_state *states,
+				     int state_count)
 {
 	int i, ret;
 	u32 psci_state, *psci_state_buf;
@@ -72,7 +73,7 @@ static int __init psci_pd_parse_state_nodes(struct genpd_power_state *states,
 	return ret;
 }
 
-static int __init psci_pd_parse_states(struct device_node *np,
+static int psci_pd_parse_states(struct device_node *np,
 			struct genpd_power_state **states, int *state_count)
 {
 	int ret;
@@ -100,7 +101,7 @@ static void psci_pd_free_states(struct genpd_power_state *states,
 	kfree(states);
 }
 
-static int __init psci_pd_init(struct device_node *np)
+static int psci_pd_init(struct device_node *np)
 {
 	struct generic_pm_domain *pd;
 	struct psci_pd_provider *pd_provider;
@@ -167,7 +168,7 @@ static int __init psci_pd_init(struct device_node *np)
 	return ret;
 }
 
-static void __init psci_pd_remove(void)
+static void psci_pd_remove(void)
 {
 	struct psci_pd_provider *pd_provider, *it;
 	struct generic_pm_domain *genpd;
@@ -185,7 +186,7 @@ static void __init psci_pd_remove(void)
 	}
 }
 
-static int __init psci_pd_init_topology(struct device_node *np, bool add)
+static int psci_pd_init_topology(struct device_node *np, bool add)
 {
 	struct device_node *node;
 	struct of_phandle_args child, parent;
@@ -211,24 +212,24 @@ static int __init psci_pd_init_topology(struct device_node *np, bool add)
 	return 0;
 }
 
-static int __init psci_pd_add_topology(struct device_node *np)
+static int psci_pd_add_topology(struct device_node *np)
 {
 	return psci_pd_init_topology(np, true);
 }
 
-static void __init psci_pd_remove_topology(struct device_node *np)
+static void psci_pd_remove_topology(struct device_node *np)
 {
 	psci_pd_init_topology(np, false);
 }
 
-static const struct of_device_id psci_of_match[] __initconst = {
+static const struct of_device_id psci_of_match[] = {
 	{ .compatible = "arm,psci-1.0" },
 	{}
 };
 
-static int __init psci_idle_init_domains(void)
+static int psci_cpuidle_domain_probe(struct platform_device *pdev)
 {
-	struct device_node *np = of_find_matching_node(NULL, psci_of_match);
+	struct device_node *np = pdev->dev.of_node;
 	struct device_node *node;
 	int ret = 0, pd_count = 0;
 
@@ -237,7 +238,7 @@ static int __init psci_idle_init_domains(void)
 
 	/* Currently limit the hierarchical topology to be used in OSI mode. */
 	if (!psci_has_osi_support())
-		goto out;
+		return 0;
 
 	/*
 	 * Parse child nodes for the "#power-domain-cells" property and
@@ -256,7 +257,7 @@ static int __init psci_idle_init_domains(void)
 
 	/* Bail out if not using the hierarchical CPU topology. */
 	if (!pd_count)
-		goto out;
+		return 0;
 
 	/* Link genpd masters/subdomains to model the CPU topology. */
 	ret = psci_pd_add_topology(np);
@@ -271,9 +272,8 @@ static int __init psci_idle_init_domains(void)
 		goto remove_pd;
 	}
 
-	of_node_put(np);
 	pr_info("Initialized CPU PM domain topology\n");
-	return pd_count;
+	return 0;
 
 put_node:
 	of_node_put(node);
@@ -281,10 +281,21 @@ static int __init psci_idle_init_domains(void)
 	if (pd_count)
 		psci_pd_remove();
 	pr_err("failed to create CPU PM domains ret=%d\n", ret);
-out:
-	of_node_put(np);
 	return ret;
 }
+
+static struct platform_driver psci_cpuidle_domain_driver = {
+	.probe  = psci_cpuidle_domain_probe,
+	.driver = {
+		.name = "psci-cpuidle-domain",
+		.of_match_table = psci_of_match,
+	},
+};
+
+static int __init psci_idle_init_domains(void)
+{
+	return platform_driver_register(&psci_cpuidle_domain_driver);
+}
 subsys_initcall(psci_idle_init_domains);
 
 struct device *psci_dt_attach_cpu(int cpu)
-- 
2.20.1

Re: [PATCH 4/5] cpuidle: psci: Convert PM domain to platform driver

[Lina Iyer]

On Mon, Jun 15 2020 at 09:21 -0600, Ulf Hansson wrote:
>To enable support for deferred probing and to allow implementation of the
>->sync_state() callback from subsequent changes, let's convert into a
>platform driver.
>
>Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
Reviewed-by: Lina Iyer <ilina@codeaurora.org>

>---
> drivers/cpuidle/cpuidle-psci-domain.c | 45 +++++++++++++++++----------
> 1 file changed, 28 insertions(+), 17 deletions(-)
>
>diff --git a/drivers/cpuidle/cpuidle-psci-domain.c b/drivers/cpuidle/cpuidle-psci-domain.c
>index e48e578aaa7d..bf527d2bb4b6 100644
>--- a/drivers/cpuidle/cpuidle-psci-domain.c
>+++ b/drivers/cpuidle/cpuidle-psci-domain.c
>@@ -12,6 +12,7 @@
> #include <linux/cpu.h>
> #include <linux/device.h>
> #include <linux/kernel.h>
>+#include <linux/platform_device.h>
> #include <linux/pm_domain.h>
> #include <linux/pm_runtime.h>
> #include <linux/psci.h>
>@@ -42,8 +43,8 @@ static int psci_pd_power_off(struct generic_pm_domain *pd)
> 	return 0;
> }
>
>-static int __init psci_pd_parse_state_nodes(struct genpd_power_state *states,
>-					int state_count)
>+static int psci_pd_parse_state_nodes(struct genpd_power_state *states,
>+				     int state_count)
> {
> 	int i, ret;
> 	u32 psci_state, *psci_state_buf;
>@@ -72,7 +73,7 @@ static int __init psci_pd_parse_state_nodes(struct genpd_power_state *states,
> 	return ret;
> }
>
>-static int __init psci_pd_parse_states(struct device_node *np,
>+static int psci_pd_parse_states(struct device_node *np,
> 			struct genpd_power_state **states, int *state_count)
> {
> 	int ret;
>@@ -100,7 +101,7 @@ static void psci_pd_free_states(struct genpd_power_state *states,
> 	kfree(states);
> }
>
>-static int __init psci_pd_init(struct device_node *np)
>+static int psci_pd_init(struct device_node *np)
> {
> 	struct generic_pm_domain *pd;
> 	struct psci_pd_provider *pd_provider;
>@@ -167,7 +168,7 @@ static int __init psci_pd_init(struct device_node *np)
> 	return ret;
> }
>
>-static void __init psci_pd_remove(void)
>+static void psci_pd_remove(void)
> {
> 	struct psci_pd_provider *pd_provider, *it;
> 	struct generic_pm_domain *genpd;
>@@ -185,7 +186,7 @@ static void __init psci_pd_remove(void)
> 	}
> }
>
>-static int __init psci_pd_init_topology(struct device_node *np, bool add)
>+static int psci_pd_init_topology(struct device_node *np, bool add)
> {
> 	struct device_node *node;
> 	struct of_phandle_args child, parent;
>@@ -211,24 +212,24 @@ static int __init psci_pd_init_topology(struct device_node *np, bool add)
> 	return 0;
> }
>
>-static int __init psci_pd_add_topology(struct device_node *np)
>+static int psci_pd_add_topology(struct device_node *np)
> {
> 	return psci_pd_init_topology(np, true);
> }
>
>-static void __init psci_pd_remove_topology(struct device_node *np)
>+static void psci_pd_remove_topology(struct device_node *np)
> {
> 	psci_pd_init_topology(np, false);
> }
>
>-static const struct of_device_id psci_of_match[] __initconst = {
>+static const struct of_device_id psci_of_match[] = {
> 	{ .compatible = "arm,psci-1.0" },
> 	{}
> };
>
>-static int __init psci_idle_init_domains(void)
>+static int psci_cpuidle_domain_probe(struct platform_device *pdev)
> {
>-	struct device_node *np = of_find_matching_node(NULL, psci_of_match);
>+	struct device_node *np = pdev->dev.of_node;
> 	struct device_node *node;
> 	int ret = 0, pd_count = 0;
>
>@@ -237,7 +238,7 @@ static int __init psci_idle_init_domains(void)
>
> 	/* Currently limit the hierarchical topology to be used in OSI mode. */
> 	if (!psci_has_osi_support())
>-		goto out;
>+		return 0;
>
> 	/*
> 	 * Parse child nodes for the "#power-domain-cells" property and
>@@ -256,7 +257,7 @@ static int __init psci_idle_init_domains(void)
>
> 	/* Bail out if not using the hierarchical CPU topology. */
> 	if (!pd_count)
>-		goto out;
>+		return 0;
>
> 	/* Link genpd masters/subdomains to model the CPU topology. */
> 	ret = psci_pd_add_topology(np);
>@@ -271,9 +272,8 @@ static int __init psci_idle_init_domains(void)
> 		goto remove_pd;
> 	}
>
>-	of_node_put(np);
> 	pr_info("Initialized CPU PM domain topology\n");
>-	return pd_count;
>+	return 0;
>
> put_node:
> 	of_node_put(node);
>@@ -281,10 +281,21 @@ static int __init psci_idle_init_domains(void)
> 	if (pd_count)
> 		psci_pd_remove();
> 	pr_err("failed to create CPU PM domains ret=%d\n", ret);
>-out:
>-	of_node_put(np);
> 	return ret;
> }
>+
>+static struct platform_driver psci_cpuidle_domain_driver = {
>+	.probe  = psci_cpuidle_domain_probe,
>+	.driver = {
>+		.name = "psci-cpuidle-domain",
>+		.of_match_table = psci_of_match,
>+	},
>+};
>+
>+static int __init psci_idle_init_domains(void)
>+{
>+	return platform_driver_register(&psci_cpuidle_domain_driver);
>+}
> subsys_initcall(psci_idle_init_domains);
>
> struct device *psci_dt_attach_cpu(int cpu)
>-- 
>2.20.1
>

[PATCH 5/5] cpuidle: psci: Prevent domain idlestates until consumers are ready

[Ulf Hansson]

Depending on the SoC/platform, additional devices may be part of the PSCI
PM domain topology. This is the case with 'qcom,rpmh-rsc' device, for
example, even if this is not yet visible in the corresponding DTS-files.

Without going into too much details, a device like the 'qcom,rpmh-rsc' may
have HW constraints that needs to be obeyed to, before a domain idlestate
can be picked.

Therefore, let's implement the ->sync_state() callback to receive a
notification when all consumers of the PSCI PM domain providers have been
attached/probed to it. In this way, we can make sure all constraints from
all relevant devices, are taken into account before allowing a domain
idlestate to be picked.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
---
 drivers/cpuidle/cpuidle-psci-domain.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/cpuidle/cpuidle-psci-domain.c b/drivers/cpuidle/cpuidle-psci-domain.c
index bf527d2bb4b6..b6e9649ab0da 100644
--- a/drivers/cpuidle/cpuidle-psci-domain.c
+++ b/drivers/cpuidle/cpuidle-psci-domain.c
@@ -27,6 +27,7 @@ struct psci_pd_provider {
 };
 
 static LIST_HEAD(psci_pd_providers);
+static bool psci_pd_allow_domain_state;
 
 static int psci_pd_power_off(struct generic_pm_domain *pd)
 {
@@ -36,6 +37,9 @@ static int psci_pd_power_off(struct generic_pm_domain *pd)
 	if (!state->data)
 		return 0;
 
+	if (!psci_pd_allow_domain_state)
+		return -EBUSY;
+
 	/* OSI mode is enabled, set the corresponding domain state. */
 	pd_state = state->data;
 	psci_set_domain_state(*pd_state);
@@ -222,6 +226,15 @@ static void psci_pd_remove_topology(struct device_node *np)
 	psci_pd_init_topology(np, false);
 }
 
+static void psci_cpuidle_domain_sync_state(struct device *dev)
+{
+	/*
+	 * All devices have now been attached/probed to the PM domain topology,
+	 * hence it's fine to allow domain states to be picked.
+	 */
+	psci_pd_allow_domain_state = true;
+}
+
 static const struct of_device_id psci_of_match[] = {
 	{ .compatible = "arm,psci-1.0" },
 	{}
@@ -289,6 +302,7 @@ static struct platform_driver psci_cpuidle_domain_driver = {
 	.driver = {
 		.name = "psci-cpuidle-domain",
 		.of_match_table = psci_of_match,
+		.sync_state = psci_cpuidle_domain_sync_state,
 	},
 };
 
-- 
2.20.1

Re: [PATCH 5/5] cpuidle: psci: Prevent domain idlestates until consumers are ready

[Saravana Kannan]

On Mon, Jun 15, 2020 at 8:21 AM Ulf Hansson <ulf.hansson@linaro.org> wrote:
>
> Depending on the SoC/platform, additional devices may be part of the PSCI
> PM domain topology. This is the case with 'qcom,rpmh-rsc' device, for
> example, even if this is not yet visible in the corresponding DTS-files.
>
> Without going into too much details, a device like the 'qcom,rpmh-rsc' may
> have HW constraints that needs to be obeyed to, before a domain idlestate
> can be picked.
>
> Therefore, let's implement the ->sync_state() callback to receive a
> notification when all consumers of the PSCI PM domain providers have been
> attached/probed to it. In this way, we can make sure all constraints from
> all relevant devices, are taken into account before allowing a domain
> idlestate to be picked.
>
> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> ---
>  drivers/cpuidle/cpuidle-psci-domain.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/drivers/cpuidle/cpuidle-psci-domain.c b/drivers/cpuidle/cpuidle-psci-domain.c
> index bf527d2bb4b6..b6e9649ab0da 100644
> --- a/drivers/cpuidle/cpuidle-psci-domain.c
> +++ b/drivers/cpuidle/cpuidle-psci-domain.c
> @@ -27,6 +27,7 @@ struct psci_pd_provider {
>  };
>
>  static LIST_HEAD(psci_pd_providers);
> +static bool psci_pd_allow_domain_state;

Is there ever only 1 device that's probed by this driver? If yes, this
is okay. Otherwise, you'll need to handle this on a per device basis.

-Saravana

Re: [PATCH 5/5] cpuidle: psci: Prevent domain idlestates until consumers are ready

[Ulf Hansson]

On Mon, 15 Jun 2020 at 20:06, Saravana Kannan <saravanak@google.com> wrote:
>
> On Mon, Jun 15, 2020 at 8:21 AM Ulf Hansson <ulf.hansson@linaro.org> wrote:
> >
> > Depending on the SoC/platform, additional devices may be part of the PSCI
> > PM domain topology. This is the case with 'qcom,rpmh-rsc' device, for
> > example, even if this is not yet visible in the corresponding DTS-files.
> >
> > Without going into too much details, a device like the 'qcom,rpmh-rsc' may
> > have HW constraints that needs to be obeyed to, before a domain idlestate
> > can be picked.
> >
> > Therefore, let's implement the ->sync_state() callback to receive a
> > notification when all consumers of the PSCI PM domain providers have been
> > attached/probed to it. In this way, we can make sure all constraints from
> > all relevant devices, are taken into account before allowing a domain
> > idlestate to be picked.
> >
> > Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> > ---
> >  drivers/cpuidle/cpuidle-psci-domain.c | 14 ++++++++++++++
> >  1 file changed, 14 insertions(+)
> >
> > diff --git a/drivers/cpuidle/cpuidle-psci-domain.c b/drivers/cpuidle/cpuidle-psci-domain.c
> > index bf527d2bb4b6..b6e9649ab0da 100644
> > --- a/drivers/cpuidle/cpuidle-psci-domain.c
> > +++ b/drivers/cpuidle/cpuidle-psci-domain.c
> > @@ -27,6 +27,7 @@ struct psci_pd_provider {
> >  };
> >
> >  static LIST_HEAD(psci_pd_providers);
> > +static bool psci_pd_allow_domain_state;
>
> Is there ever only 1 device that's probed by this driver? If yes, this
> is okay. Otherwise, you'll need to handle this on a per device basis.

There is only one device. Subnodes, may exist to describe a
hierarchical description of the topology of the power-domains [1].

Kind regards
Uffe

[1] Documentation/devicetree/bindings/arm/psci.yaml

Re: [PATCH 5/5] cpuidle: psci: Prevent domain idlestates until consumers are ready

[Saravana Kannan]

On Mon, Jun 15, 2020 at 11:50 PM Ulf Hansson <ulf.hansson@linaro.org> wrote:
>
> On Mon, 15 Jun 2020 at 20:06, Saravana Kannan <saravanak@google.com> wrote:
> >
> > On Mon, Jun 15, 2020 at 8:21 AM Ulf Hansson <ulf.hansson@linaro.org> wrote:
> > >
> > > Depending on the SoC/platform, additional devices may be part of the PSCI
> > > PM domain topology. This is the case with 'qcom,rpmh-rsc' device, for
> > > example, even if this is not yet visible in the corresponding DTS-files.
> > >
> > > Without going into too much details, a device like the 'qcom,rpmh-rsc' may
> > > have HW constraints that needs to be obeyed to, before a domain idlestate
> > > can be picked.
> > >
> > > Therefore, let's implement the ->sync_state() callback to receive a
> > > notification when all consumers of the PSCI PM domain providers have been
> > > attached/probed to it. In this way, we can make sure all constraints from
> > > all relevant devices, are taken into account before allowing a domain
> > > idlestate to be picked.
> > >
> > > Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
> > > ---
> > >  drivers/cpuidle/cpuidle-psci-domain.c | 14 ++++++++++++++
> > >  1 file changed, 14 insertions(+)
> > >
> > > diff --git a/drivers/cpuidle/cpuidle-psci-domain.c b/drivers/cpuidle/cpuidle-psci-domain.c
> > > index bf527d2bb4b6..b6e9649ab0da 100644
> > > --- a/drivers/cpuidle/cpuidle-psci-domain.c
> > > +++ b/drivers/cpuidle/cpuidle-psci-domain.c
> > > @@ -27,6 +27,7 @@ struct psci_pd_provider {
> > >  };
> > >
> > >  static LIST_HEAD(psci_pd_providers);
> > > +static bool psci_pd_allow_domain_state;
> >
> > Is there ever only 1 device that's probed by this driver? If yes, this
> > is okay. Otherwise, you'll need to handle this on a per device basis.
>
> There is only one device. Subnodes, may exist to describe a
> hierarchical description of the topology of the power-domains [1].

Thanks. In that case:

Acked-by: Saravana Kannan <saravanak@google.com>

-Saravana

Re: [PATCH 0/5] cpuidle: psci: Various improvements for PSCI PM domains

[Ulf Hansson]

On Mon, 15 Jun 2020 at 17:20, Ulf Hansson <ulf.hansson@linaro.org> wrote:
>
> The main change in this series is done in patch 5/5, which implements support
> to prevent domain idlestates until all PSCI PM domain consumers are ready for
> it. To reach that point the corresponding code for cpuidle-psci and
> cpuidle-psci-domain, needed to be converted into platform drivers, which is
> done by the earlier changes in the series.
>
> Additionally, some improvements have been made to the error path, which becomes
> easier when the code gets converted to platform drivers.
>
> Deployment for a Qcom SoC, which actually takes full benefit of these changes
> are also in the pipe, but deferring then a bit until $subject series have been
> discussed.

Sudeep, Lorenzo,

Would you mind sharing your opinions on this series please?

Kind regards
Uffe

>
> Kind regards
> Ulf Hansson
>
> Ulf Hansson (5):
>   cpuidle: psci: Fail cpuidle registration if set OSI mode failed
>   cpuidle: psci: Fix error path via converting to a platform driver
>   cpuidle: psci: Split into two separate build objects
>   cpuidle: psci: Convert PM domain to platform driver
>   cpuidle: psci: Prevent domain idlestates until consumers are ready
>
>  drivers/cpuidle/Kconfig.arm           |  10 ++
>  drivers/cpuidle/Makefile              |   5 +-
>  drivers/cpuidle/cpuidle-psci-domain.c |  74 +++++++++-----
>  drivers/cpuidle/cpuidle-psci.c        | 141 +++++++++++++++-----------
>  drivers/cpuidle/cpuidle-psci.h        |  11 +-
>  5 files changed, 150 insertions(+), 91 deletions(-)
>
> --
> 2.20.1
>

Re: [PATCH 0/5] cpuidle: psci: Various improvements for PSCI PM domains

[Lukasz Luba]

Hi Ulf,

On 6/15/20 4:20 PM, Ulf Hansson wrote:
> The main change in this series is done in patch 5/5, which implements support
> to prevent domain idlestates until all PSCI PM domain consumers are ready for
> it. To reach that point the corresponding code for cpuidle-psci and
> cpuidle-psci-domain, needed to be converted into platform drivers, which is
> done by the earlier changes in the series.
> 
> Additionally, some improvements have been made to the error path, which becomes
> easier when the code gets converted to platform drivers.
> 
> Deployment for a Qcom SoC, which actually takes full benefit of these changes
> are also in the pipe, but deferring then a bit until $subject series have been
> discussed.
> 
> Kind regards
> Ulf Hansson
> 
> Ulf Hansson (5):
>    cpuidle: psci: Fail cpuidle registration if set OSI mode failed
>    cpuidle: psci: Fix error path via converting to a platform driver
>    cpuidle: psci: Split into two separate build objects
>    cpuidle: psci: Convert PM domain to platform driver
>    cpuidle: psci: Prevent domain idlestates until consumers are ready
> 
>   drivers/cpuidle/Kconfig.arm           |  10 ++
>   drivers/cpuidle/Makefile              |   5 +-
>   drivers/cpuidle/cpuidle-psci-domain.c |  74 +++++++++-----
>   drivers/cpuidle/cpuidle-psci.c        | 141 +++++++++++++++-----------
>   drivers/cpuidle/cpuidle-psci.h        |  11 +-
>   5 files changed, 150 insertions(+), 91 deletions(-)
> 

Since I am interested in some CPU idle statistics (residency, etc),
I would like to help you and Sudeep in reviewing the patch set.

Could you clarify some bit below?
1. There is Qcom SoC which has dependencies between PSCI PM domain
consumers and this CPU idle - thus we cannot register and use CPU
idle driver till related drivers fully setup.
2. The proposed solution is to use platform driver and plat. device
for this CPU idle driver, to have access to deferred probe mechanism and
wait for the consumer drivers fully probed state.
3. Do you have maybe some estimations how long it takes for these
consumers to be fully probed?
4. Changing just this CPU idle driver registration point (to
late_initcall()) phase in time is not a solution.

Regards,
Lukasz

Re: [PATCH 0/5] cpuidle: psci: Various improvements for PSCI PM domains

[Ulf Hansson]

Hi Lukaz,

On Tue, 30 Jun 2020 at 12:23, Lukasz Luba <lukasz.luba@arm.com> wrote:
>
> Hi Ulf,
>
> On 6/15/20 4:20 PM, Ulf Hansson wrote:
> > The main change in this series is done in patch 5/5, which implements support
> > to prevent domain idlestates until all PSCI PM domain consumers are ready for
> > it. To reach that point the corresponding code for cpuidle-psci and
> > cpuidle-psci-domain, needed to be converted into platform drivers, which is
> > done by the earlier changes in the series.
> >
> > Additionally, some improvements have been made to the error path, which becomes
> > easier when the code gets converted to platform drivers.
> >
> > Deployment for a Qcom SoC, which actually takes full benefit of these changes
> > are also in the pipe, but deferring then a bit until $subject series have been
> > discussed.
> >
> > Kind regards
> > Ulf Hansson
> >
> > Ulf Hansson (5):
> >    cpuidle: psci: Fail cpuidle registration if set OSI mode failed
> >    cpuidle: psci: Fix error path via converting to a platform driver
> >    cpuidle: psci: Split into two separate build objects
> >    cpuidle: psci: Convert PM domain to platform driver
> >    cpuidle: psci: Prevent domain idlestates until consumers are ready
> >
> >   drivers/cpuidle/Kconfig.arm           |  10 ++
> >   drivers/cpuidle/Makefile              |   5 +-
> >   drivers/cpuidle/cpuidle-psci-domain.c |  74 +++++++++-----
> >   drivers/cpuidle/cpuidle-psci.c        | 141 +++++++++++++++-----------
> >   drivers/cpuidle/cpuidle-psci.h        |  11 +-
> >   5 files changed, 150 insertions(+), 91 deletions(-)
> >
>
> Since I am interested in some CPU idle statistics (residency, etc),
> I would like to help you and Sudeep in reviewing the patch set.

Thanks, much appreciated!

>
> Could you clarify some bit below?
> 1. There is Qcom SoC which has dependencies between PSCI PM domain
> consumers and this CPU idle - thus we cannot register and use CPU
> idle driver till related drivers fully setup.

I think you got it right, but let me clarify.

On Qcom SoC, PSCI PM domain consumers aren't solely CPU devices, but
also the 'qcom,rpmh-rsc' device is a consumer, for example.

That doesn't mean the CPU idle driver can't be probed/initialized, but
rather that the PSCI PM domain must not be powered off. The power off
needs to wait until all the consumers of the PSCI PM domain have been
registered/probed.

See more details in the commit message of patch5.

> 2. The proposed solution is to use platform driver and plat. device
> for this CPU idle driver, to have access to deferred probe mechanism and
> wait for the consumer drivers fully probed state.

Correct, but let me fill in some more.

I would like to use the ->sync_state() callback of the PSCI PM domain
driver, as a way to understand that all consumers have been probed.

> 3. Do you have maybe some estimations how long it takes for these
> consumers to be fully probed?

I am not sure I understand the reason for the question.

Anyway, at this point, I am looking at the qcom,rpmh-rsc device, which
is being probed by the drivers/soc/qcom/rpmh-rsc.c driver. Moving
forward, in principle it can be any device/driver that is a consumer
of the PSCI PM domain. I am not even excluding that drivers can be
modules. It should work for all cases.

> 4. Changing just this CPU idle driver registration point (to
> late_initcall()) phase in time is not a solution.

Correct, it doesn't work.

Playing with initcalls doesn't guarantee anything when it comes to
making sure all consumers are ready.

Hope this clarifies things a bit for you, but just tell me if there is
anything more I can do to further explain things.

Kind regards
Uffe

Re: [PATCH 0/5] cpuidle: psci: Various improvements for PSCI PM domains

[Lukasz Luba]

On 7/7/20 12:53 PM, Ulf Hansson wrote:
> Hi Lukaz,
> 
> On Tue, 30 Jun 2020 at 12:23, Lukasz Luba <lukasz.luba@arm.com> wrote:
>>
>> Hi Ulf,
>>
>> On 6/15/20 4:20 PM, Ulf Hansson wrote:
>>> The main change in this series is done in patch 5/5, which implements support
>>> to prevent domain idlestates until all PSCI PM domain consumers are ready for
>>> it. To reach that point the corresponding code for cpuidle-psci and
>>> cpuidle-psci-domain, needed to be converted into platform drivers, which is
>>> done by the earlier changes in the series.
>>>
>>> Additionally, some improvements have been made to the error path, which becomes
>>> easier when the code gets converted to platform drivers.
>>>
>>> Deployment for a Qcom SoC, which actually takes full benefit of these changes
>>> are also in the pipe, but deferring then a bit until $subject series have been
>>> discussed.
>>>
>>> Kind regards
>>> Ulf Hansson
>>>
>>> Ulf Hansson (5):
>>>     cpuidle: psci: Fail cpuidle registration if set OSI mode failed
>>>     cpuidle: psci: Fix error path via converting to a platform driver
>>>     cpuidle: psci: Split into two separate build objects
>>>     cpuidle: psci: Convert PM domain to platform driver
>>>     cpuidle: psci: Prevent domain idlestates until consumers are ready
>>>
>>>    drivers/cpuidle/Kconfig.arm           |  10 ++
>>>    drivers/cpuidle/Makefile              |   5 +-
>>>    drivers/cpuidle/cpuidle-psci-domain.c |  74 +++++++++-----
>>>    drivers/cpuidle/cpuidle-psci.c        | 141 +++++++++++++++-----------
>>>    drivers/cpuidle/cpuidle-psci.h        |  11 +-
>>>    5 files changed, 150 insertions(+), 91 deletions(-)
>>>
>>
>> Since I am interested in some CPU idle statistics (residency, etc),
>> I would like to help you and Sudeep in reviewing the patch set.
> 
> Thanks, much appreciated!
> 
>>
>> Could you clarify some bit below?
>> 1. There is Qcom SoC which has dependencies between PSCI PM domain
>> consumers and this CPU idle - thus we cannot register and use CPU
>> idle driver till related drivers fully setup.
> 
> I think you got it right, but let me clarify.
> 
> On Qcom SoC, PSCI PM domain consumers aren't solely CPU devices, but
> also the 'qcom,rpmh-rsc' device is a consumer, for example.
> 
> That doesn't mean the CPU idle driver can't be probed/initialized, but
> rather that the PSCI PM domain must not be powered off. The power off
> needs to wait until all the consumers of the PSCI PM domain have been
> registered/probed.
> 
> See more details in the commit message of patch5.
> 
>> 2. The proposed solution is to use platform driver and plat. device
>> for this CPU idle driver, to have access to deferred probe mechanism and
>> wait for the consumer drivers fully probed state.
> 
> Correct, but let me fill in some more.
> 
> I would like to use the ->sync_state() callback of the PSCI PM domain
> driver, as a way to understand that all consumers have been probed.
> 
>> 3. Do you have maybe some estimations how long it takes for these
>> consumers to be fully probed?
> 
> I am not sure I understand the reason for the question.

I was wondering if this is because of HW issue of long setup, thus
we need to wait a bit longer with drivers deferred probing.
But this is not the case as I can see now.


> 
> Anyway, at this point, I am looking at the qcom,rpmh-rsc device, which
> is being probed by the drivers/soc/qcom/rpmh-rsc.c driver. Moving
> forward, in principle it can be any device/driver that is a consumer
> of the PSCI PM domain. I am not even excluding that drivers can be
> modules. It should work for all cases.

The late_initcall won't help, this is a really tough requirement:
being a module...


> 
>> 4. Changing just this CPU idle driver registration point (to
>> late_initcall()) phase in time is not a solution.
> 
> Correct, it doesn't work.
> 
> Playing with initcalls doesn't guarantee anything when it comes to
> making sure all consumers are ready.

I agree, especially when modules are involved.

> 
> Hope this clarifies things a bit for you, but just tell me if there is
> anything more I can do to further explain things.

Yes, thank you. I can see now why you need this explicit ->sync_state()
callback.
I don't see better solution to what you have proposed here.
I will go through the patches once again to check and add some
reviewed-by.

Regards,
Lukasz

> 
> Kind regards
> Uffe
> 

Re: [PATCH 0/5] cpuidle: psci: Various improvements for PSCI PM domains

[Ulf Hansson]

On Tue, 7 Jul 2020 at 14:37, Lukasz Luba <lukasz.luba@arm.com> wrote:
>
>
>
> On 7/7/20 12:53 PM, Ulf Hansson wrote:
> > Hi Lukaz,
> >
> > On Tue, 30 Jun 2020 at 12:23, Lukasz Luba <lukasz.luba@arm.com> wrote:
> >>
> >> Hi Ulf,
> >>
> >> On 6/15/20 4:20 PM, Ulf Hansson wrote:
> >>> The main change in this series is done in patch 5/5, which implements support
> >>> to prevent domain idlestates until all PSCI PM domain consumers are ready for
> >>> it. To reach that point the corresponding code for cpuidle-psci and
> >>> cpuidle-psci-domain, needed to be converted into platform drivers, which is
> >>> done by the earlier changes in the series.
> >>>
> >>> Additionally, some improvements have been made to the error path, which becomes
> >>> easier when the code gets converted to platform drivers.
> >>>
> >>> Deployment for a Qcom SoC, which actually takes full benefit of these changes
> >>> are also in the pipe, but deferring then a bit until $subject series have been
> >>> discussed.
> >>>
> >>> Kind regards
> >>> Ulf Hansson
> >>>
> >>> Ulf Hansson (5):
> >>>     cpuidle: psci: Fail cpuidle registration if set OSI mode failed
> >>>     cpuidle: psci: Fix error path via converting to a platform driver
> >>>     cpuidle: psci: Split into two separate build objects
> >>>     cpuidle: psci: Convert PM domain to platform driver
> >>>     cpuidle: psci: Prevent domain idlestates until consumers are ready
> >>>
> >>>    drivers/cpuidle/Kconfig.arm           |  10 ++
> >>>    drivers/cpuidle/Makefile              |   5 +-
> >>>    drivers/cpuidle/cpuidle-psci-domain.c |  74 +++++++++-----
> >>>    drivers/cpuidle/cpuidle-psci.c        | 141 +++++++++++++++-----------
> >>>    drivers/cpuidle/cpuidle-psci.h        |  11 +-
> >>>    5 files changed, 150 insertions(+), 91 deletions(-)
> >>>
> >>
> >> Since I am interested in some CPU idle statistics (residency, etc),
> >> I would like to help you and Sudeep in reviewing the patch set.
> >
> > Thanks, much appreciated!
> >
> >>
> >> Could you clarify some bit below?
> >> 1. There is Qcom SoC which has dependencies between PSCI PM domain
> >> consumers and this CPU idle - thus we cannot register and use CPU
> >> idle driver till related drivers fully setup.
> >
> > I think you got it right, but let me clarify.
> >
> > On Qcom SoC, PSCI PM domain consumers aren't solely CPU devices, but
> > also the 'qcom,rpmh-rsc' device is a consumer, for example.
> >
> > That doesn't mean the CPU idle driver can't be probed/initialized, but
> > rather that the PSCI PM domain must not be powered off. The power off
> > needs to wait until all the consumers of the PSCI PM domain have been
> > registered/probed.
> >
> > See more details in the commit message of patch5.
> >
> >> 2. The proposed solution is to use platform driver and plat. device
> >> for this CPU idle driver, to have access to deferred probe mechanism and
> >> wait for the consumer drivers fully probed state.
> >
> > Correct, but let me fill in some more.
> >
> > I would like to use the ->sync_state() callback of the PSCI PM domain
> > driver, as a way to understand that all consumers have been probed.
> >
> >> 3. Do you have maybe some estimations how long it takes for these
> >> consumers to be fully probed?
> >
> > I am not sure I understand the reason for the question.
>
> I was wondering if this is because of HW issue of long setup, thus
> we need to wait a bit longer with drivers deferred probing.
> But this is not the case as I can see now.
>
>
> >
> > Anyway, at this point, I am looking at the qcom,rpmh-rsc device, which
> > is being probed by the drivers/soc/qcom/rpmh-rsc.c driver. Moving
> > forward, in principle it can be any device/driver that is a consumer
> > of the PSCI PM domain. I am not even excluding that drivers can be
> > modules. It should work for all cases.
>
> The late_initcall won't help, this is a really tough requirement:
> being a module...
>
>
> >
> >> 4. Changing just this CPU idle driver registration point (to
> >> late_initcall()) phase in time is not a solution.
> >
> > Correct, it doesn't work.
> >
> > Playing with initcalls doesn't guarantee anything when it comes to
> > making sure all consumers are ready.
>
> I agree, especially when modules are involved.
>
> >
> > Hope this clarifies things a bit for you, but just tell me if there is
> > anything more I can do to further explain things.
>
> Yes, thank you. I can see now why you need this explicit ->sync_state()
> callback.
> I don't see better solution to what you have proposed here.
> I will go through the patches once again to check and add some
> reviewed-by.
>

Thanks a lot for your time! I am just about to post a v2, to re-order
the series so patch 3 comes first, according to suggestions from Lina.

Please move over to review that version instead, in a few minutes.

Kind regards
Uffe

Re: [PATCH 0/5] cpuidle: psci: Various improvements for PSCI PM domains

[Lukasz Luba]

On 7/7/20 1:51 PM, Ulf Hansson wrote:
> On Tue, 7 Jul 2020 at 14:37, Lukasz Luba <lukasz.luba@arm.com> wrote:
>>
>>
>>
>> On 7/7/20 12:53 PM, Ulf Hansson wrote:
>>> Hi Lukaz,
>>>
>>> On Tue, 30 Jun 2020 at 12:23, Lukasz Luba <lukasz.luba@arm.com> wrote:
>>>>
>>>> Hi Ulf,
>>>>
>>>> On 6/15/20 4:20 PM, Ulf Hansson wrote:
>>>>> The main change in this series is done in patch 5/5, which implements support
>>>>> to prevent domain idlestates until all PSCI PM domain consumers are ready for
>>>>> it. To reach that point the corresponding code for cpuidle-psci and
>>>>> cpuidle-psci-domain, needed to be converted into platform drivers, which is
>>>>> done by the earlier changes in the series.
>>>>>
>>>>> Additionally, some improvements have been made to the error path, which becomes
>>>>> easier when the code gets converted to platform drivers.
>>>>>
>>>>> Deployment for a Qcom SoC, which actually takes full benefit of these changes
>>>>> are also in the pipe, but deferring then a bit until $subject series have been
>>>>> discussed.
>>>>>
>>>>> Kind regards
>>>>> Ulf Hansson
>>>>>
>>>>> Ulf Hansson (5):
>>>>>      cpuidle: psci: Fail cpuidle registration if set OSI mode failed
>>>>>      cpuidle: psci: Fix error path via converting to a platform driver
>>>>>      cpuidle: psci: Split into two separate build objects
>>>>>      cpuidle: psci: Convert PM domain to platform driver
>>>>>      cpuidle: psci: Prevent domain idlestates until consumers are ready
>>>>>
>>>>>     drivers/cpuidle/Kconfig.arm           |  10 ++
>>>>>     drivers/cpuidle/Makefile              |   5 +-
>>>>>     drivers/cpuidle/cpuidle-psci-domain.c |  74 +++++++++-----
>>>>>     drivers/cpuidle/cpuidle-psci.c        | 141 +++++++++++++++-----------
>>>>>     drivers/cpuidle/cpuidle-psci.h        |  11 +-
>>>>>     5 files changed, 150 insertions(+), 91 deletions(-)
>>>>>
>>>>
>>>> Since I am interested in some CPU idle statistics (residency, etc),
>>>> I would like to help you and Sudeep in reviewing the patch set.
>>>
>>> Thanks, much appreciated!
>>>
>>>>
>>>> Could you clarify some bit below?
>>>> 1. There is Qcom SoC which has dependencies between PSCI PM domain
>>>> consumers and this CPU idle - thus we cannot register and use CPU
>>>> idle driver till related drivers fully setup.
>>>
>>> I think you got it right, but let me clarify.
>>>
>>> On Qcom SoC, PSCI PM domain consumers aren't solely CPU devices, but
>>> also the 'qcom,rpmh-rsc' device is a consumer, for example.
>>>
>>> That doesn't mean the CPU idle driver can't be probed/initialized, but
>>> rather that the PSCI PM domain must not be powered off. The power off
>>> needs to wait until all the consumers of the PSCI PM domain have been
>>> registered/probed.
>>>
>>> See more details in the commit message of patch5.
>>>
>>>> 2. The proposed solution is to use platform driver and plat. device
>>>> for this CPU idle driver, to have access to deferred probe mechanism and
>>>> wait for the consumer drivers fully probed state.
>>>
>>> Correct, but let me fill in some more.
>>>
>>> I would like to use the ->sync_state() callback of the PSCI PM domain
>>> driver, as a way to understand that all consumers have been probed.
>>>
>>>> 3. Do you have maybe some estimations how long it takes for these
>>>> consumers to be fully probed?
>>>
>>> I am not sure I understand the reason for the question.
>>
>> I was wondering if this is because of HW issue of long setup, thus
>> we need to wait a bit longer with drivers deferred probing.
>> But this is not the case as I can see now.
>>
>>
>>>
>>> Anyway, at this point, I am looking at the qcom,rpmh-rsc device, which
>>> is being probed by the drivers/soc/qcom/rpmh-rsc.c driver. Moving
>>> forward, in principle it can be any device/driver that is a consumer
>>> of the PSCI PM domain. I am not even excluding that drivers can be
>>> modules. It should work for all cases.
>>
>> The late_initcall won't help, this is a really tough requirement:
>> being a module...
>>
>>
>>>
>>>> 4. Changing just this CPU idle driver registration point (to
>>>> late_initcall()) phase in time is not a solution.
>>>
>>> Correct, it doesn't work.
>>>
>>> Playing with initcalls doesn't guarantee anything when it comes to
>>> making sure all consumers are ready.
>>
>> I agree, especially when modules are involved.
>>
>>>
>>> Hope this clarifies things a bit for you, but just tell me if there is
>>> anything more I can do to further explain things.
>>
>> Yes, thank you. I can see now why you need this explicit ->sync_state()
>> callback.
>> I don't see better solution to what you have proposed here.
>> I will go through the patches once again to check and add some
>> reviewed-by.
>>
> 
> Thanks a lot for your time! I am just about to post a v2, to re-order
> the series so patch 3 comes first, according to suggestions from Lina.
> 
> Please move over to review that version instead, in a few minutes.

OK, got it in my mailbox, thanks!

Regards,
Lukasz