[PATCH 0/2] Link consumer with clock driver

[PATCH 0/2] Link consumer with clock driver

[Miquel Raynal]

Hello,

While working on suspend to RAM feature, I ran into troubles multiple
times when clocks where not suspending/resuming at the desired time. I
had a look at the core and I think the same logic as in the
regulator's core may be applied here to (very easily) fix this issue:
using device links.

The only additional change I had to do was to always (when available)
populate the device entry of the core clock structure so that it could
be used later. This is the purpose of patch 1. Patch 2 actually adds
support for device links.

As I am not used to hack into the clock subsystem I might have missed
something big preventing such change but so far I could not see
anything wrong with it. As this touches core code, I am of course
entirely open to suggestions.

Thanks,
Miquèl


Miquel Raynal (2):
  clk: core: clarify the check for runtime PM
  clk: core: link consumer with clock driver

 drivers/clk/clk.c            | 31 +++++++++++++++++++++++++------
 drivers/clk/clkdev.c         | 13 ++++++++++---
 include/linux/clk-provider.h |  2 ++
 3 files changed, 37 insertions(+), 9 deletions(-)

-- 
2.19.1

[PATCH 1/2] clk: core: clarify the check for runtime PM

[Miquel Raynal]

Currently, the core->dev entry is populated only if runtime PM is
enabled. Doing so prevents accessing the device structure in any
case.

Keep the same logic but instead of using the presence of core->dev as
the only condition, also check the status of
pm_runtime_enabled(). Then, we can set the core->dev pointer at any
time as long as a device structure is available.

This change will help supporting device links in the clock subsystem.

Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com>
---
 drivers/clk/clk.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c
index af011974d4ec..b799347c5fd6 100644
--- a/drivers/clk/clk.c
+++ b/drivers/clk/clk.c
@@ -97,7 +97,7 @@ static int clk_pm_runtime_get(struct clk_core *core)
 {
 	int ret = 0;
 
-	if (!core->dev)
+	if (!core->dev || !pm_runtime_enabled(core->dev))
 		return 0;
 
 	ret = pm_runtime_get_sync(core->dev);
@@ -106,7 +106,7 @@ static int clk_pm_runtime_get(struct clk_core *core)
 
 static void clk_pm_runtime_put(struct clk_core *core)
 {
-	if (!core->dev)
+	if (!core->dev || !pm_runtime_enabled(core->dev))
 		return;
 
 	pm_runtime_put_sync(core->dev);
@@ -226,7 +226,7 @@ static bool clk_core_is_enabled(struct clk_core *core)
 	 * taking enable spinlock, but the below check is needed if one tries
 	 * to call it from other places.
 	 */
-	if (core->dev) {
+	if (core->dev && pm_runtime_enabled(core->dev)) {
 		pm_runtime_get_noresume(core->dev);
 		if (!pm_runtime_active(core->dev)) {
 			ret = false;
@@ -236,7 +236,7 @@ static bool clk_core_is_enabled(struct clk_core *core)
 
 	ret = core->ops->is_enabled(core->hw);
 done:
-	if (core->dev)
+	if (core->dev && pm_runtime_enabled(core->dev))
 		pm_runtime_put(core->dev);
 
 	return ret;
@@ -3272,8 +3272,7 @@ struct clk *clk_register(struct device *dev, struct clk_hw *hw)
 	}
 	core->ops = hw->init->ops;
 
-	if (dev && pm_runtime_enabled(dev))
-		core->dev = dev;
+	core->dev = dev;
 	if (dev && dev->driver)
 		core->owner = dev->driver->owner;
 	core->hw = hw;
-- 
2.19.1

[PATCH 2/2] clk: core: link consumer with clock driver

[Miquel Raynal]

One major concern when, for instance, suspending/resuming a platform
is to never access registers before the underlying clock has been
resumed, otherwise most of the time the kernel will just crash. One
solution is to use syscore operations when registering clock drivers
suspend/resume callbacks. One problem of using syscore_ops is that the
suspend/resume scheduling will depend on the order of the
registrations, which brings (unacceptable) randomness in the process.

A feature called device links has been introduced to handle such
situation. It creates dependencies between consumers and providers,
enforcing e.g. the suspend/resume order when needed. Such feature is
already in use for regulators.

Add device links support in the clock subsystem by creating/deleting
the links at get/put time.

Example of a boot (ESPRESSObin, A3700 SoC) with devices linked to clocks:
ahci-mvebu d00e0000.sata: Linked as a consumer to d0013000.nb-periph-clk
mvneta d0030000.ethernet: Linked as a consumer to d0018000.sb-periph-clk
xhci-hcd d0058000.usb: Linked as a consumer to d0018000.sb-periph-clk
xenon-sdhci d00d0000.sdhci: Linked as a consumer to d0013000.nb-periph-clk
xenon-sdhci d00d0000.sdhci: Dropping the link to d0013000.nb-periph-clk
advk-pcie d0070000.pcie: Linked as a consumer to d0018000.sb-periph-clk
xenon-sdhci d00d0000.sdhci: Linked as a consumer to d0013000.nb-periph-clk
xenon-sdhci d00d0000.sdhci: Linked as a consumer to regulator.1
cpu cpu0: Linked as a consumer to d0013000.nb-periph-clk
cpu cpu0: Dropping the link to d0013000.nb-periph-clk
cpu cpu0: Linked as a consumer to d0013000.nb-periph-clk

Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com>
---
 drivers/clk/clk.c            | 20 ++++++++++++++++++++
 drivers/clk/clkdev.c         | 13 ++++++++++---
 include/linux/clk-provider.h |  2 ++
 3 files changed, 32 insertions(+), 3 deletions(-)

diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c
index b799347c5fd6..33a0f2b0533a 100644
--- a/drivers/clk/clk.c
+++ b/drivers/clk/clk.c
@@ -90,6 +90,7 @@ struct clk {
 	unsigned long max_rate;
 	unsigned int exclusive_count;
 	struct hlist_node clks_node;
+	struct device_link *link;
 };
 
 /***           runtime pm          ***/
@@ -262,6 +263,25 @@ struct clk_hw *__clk_get_hw(struct clk *clk)
 }
 EXPORT_SYMBOL_GPL(__clk_get_hw);
 
+void __clk_device_link(struct device *consumer, struct clk *clk)
+{
+	if (!consumer || !clk || !clk->core)
+		return;
+
+	clk->link = device_link_add(consumer, clk->core->dev,
+				    DL_FLAG_STATELESS);
+}
+EXPORT_SYMBOL_GPL(__clk_device_link);
+
+void __clk_device_unlink(struct clk *clk)
+{
+	if (!clk || !clk->link)
+		return;
+
+	device_link_del(clk->link);
+}
+EXPORT_SYMBOL_GPL(__clk_device_unlink);
+
 unsigned int clk_hw_get_num_parents(const struct clk_hw *hw)
 {
 	return hw->core->num_parents;
diff --git a/drivers/clk/clkdev.c b/drivers/clk/clkdev.c
index 9ab3db8b3988..fccfd4c01457 100644
--- a/drivers/clk/clkdev.c
+++ b/drivers/clk/clkdev.c
@@ -194,20 +194,27 @@ EXPORT_SYMBOL(clk_get_sys);
 struct clk *clk_get(struct device *dev, const char *con_id)
 {
 	const char *dev_id = dev ? dev_name(dev) : NULL;
-	struct clk *clk;
+	struct clk *clk = NULL;
 
 	if (dev && dev->of_node) {
 		clk = __of_clk_get_by_name(dev->of_node, dev_id, con_id);
-		if (!IS_ERR(clk) || PTR_ERR(clk) == -EPROBE_DEFER)
+		if (PTR_ERR(clk) == -EPROBE_DEFER)
 			return clk;
 	}
 
-	return clk_get_sys(dev_id, con_id);
+	if (IS_ERR_OR_NULL(clk))
+		clk = clk_get_sys(dev_id, con_id);
+
+	if (!IS_ERR(clk))
+		__clk_device_link(dev, clk);
+
+	return clk;
 }
 EXPORT_SYMBOL(clk_get);
 
 void clk_put(struct clk *clk)
 {
+	__clk_device_unlink(clk);
 	__clk_put(clk);
 }
 EXPORT_SYMBOL(clk_put);
diff --git a/include/linux/clk-provider.h b/include/linux/clk-provider.h
index 60c51871b04b..c7ba8098f854 100644
--- a/include/linux/clk-provider.h
+++ b/include/linux/clk-provider.h
@@ -781,6 +781,8 @@ void devm_clk_hw_unregister(struct device *dev, struct clk_hw *hw);
 const char *__clk_get_name(const struct clk *clk);
 const char *clk_hw_get_name(const struct clk_hw *hw);
 struct clk_hw *__clk_get_hw(struct clk *clk);
+void __clk_device_link(struct device *consumer, struct clk *clk);
+void __clk_device_unlink(struct clk *clk);
 unsigned int clk_hw_get_num_parents(const struct clk_hw *hw);
 struct clk_hw *clk_hw_get_parent(const struct clk_hw *hw);
 struct clk_hw *clk_hw_get_parent_by_index(const struct clk_hw *hw,
-- 
2.19.1

Re: [PATCH 2/2] clk: core: link consumer with clock driver

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 2399 bytes --]

Hi Miquel,

I love your patch! Yet something to improve:

[auto build test ERROR on clk/clk-next]
[also build test ERROR on v4.20-rc3 next-20181122]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Miquel-Raynal/Link-consumer-with-clock-driver/20181123-113833
base:   https://git.kernel.org/pub/scm/linux/kernel/git/clk/linux.git clk-next
config: sh-titan_defconfig (attached as .config)
compiler: sh4-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        GCC_VERSION=7.2.0 make.cross ARCH=sh 

All errors (new ones prefixed by >>):

   drivers//clk/clkdev.c: In function 'clk_get':
>> drivers//clk/clkdev.c:209:3: error: implicit declaration of function '__clk_device_link'; did you mean '__clk_free_clk'? [-Werror=implicit-function-declaration]
      __clk_device_link(dev, clk);
      ^~~~~~~~~~~~~~~~~
      __clk_free_clk
   drivers//clk/clkdev.c: In function 'clk_put':
>> drivers//clk/clkdev.c:217:2: error: implicit declaration of function '__clk_device_unlink'; did you mean 'device_online'? [-Werror=implicit-function-declaration]
     __clk_device_unlink(clk);
     ^~~~~~~~~~~~~~~~~~~
     device_online
   cc1: some warnings being treated as errors

vim +209 drivers//clk/clkdev.c

   193	
   194	struct clk *clk_get(struct device *dev, const char *con_id)
   195	{
   196		const char *dev_id = dev ? dev_name(dev) : NULL;
   197		struct clk *clk = NULL;
   198	
   199		if (dev && dev->of_node) {
   200			clk = __of_clk_get_by_name(dev->of_node, dev_id, con_id);
   201			if (PTR_ERR(clk) == -EPROBE_DEFER)
   202				return clk;
   203		}
   204	
   205		if (IS_ERR_OR_NULL(clk))
   206			clk = clk_get_sys(dev_id, con_id);
   207	
   208		if (!IS_ERR(clk))
 > 209			__clk_device_link(dev, clk);
   210	
   211		return clk;
   212	}
   213	EXPORT_SYMBOL(clk_get);
   214	
   215	void clk_put(struct clk *clk)
   216	{
 > 217		__clk_device_unlink(clk);
   218		__clk_put(clk);
   219	}
   220	EXPORT_SYMBOL(clk_put);
   221	

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 16439 bytes --]

Re: [PATCH 2/2] clk: core: link consumer with clock driver

[Miquel Raynal]

Hello,

kbuild test robot <lkp@intel.com> wrote on Fri, 23 Nov 2018 16:30:00
+0800:

> Hi Miquel,
> 
> I love your patch! Yet something to improve:
> 
> [auto build test ERROR on clk/clk-next]
> [also build test ERROR on v4.20-rc3 next-20181122]
> [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
> 
> url:    https://github.com/0day-ci/linux/commits/Miquel-Raynal/Link-consumer-with-clock-driver/20181123-113833
> base:   https://git.kernel.org/pub/scm/linux/kernel/git/clk/linux.git clk-next
> config: sh-titan_defconfig (attached as .config)
> compiler: sh4-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
> reproduce:
>         wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
>         chmod +x ~/bin/make.cross
>         # save the attached .config to linux build tree
>         GCC_VERSION=7.2.0 make.cross ARCH=sh 
> 
> All errors (new ones prefixed by >>):
> 
>    drivers//clk/clkdev.c: In function 'clk_get':
> >> drivers//clk/clkdev.c:209:3: error: implicit declaration of function '__clk_device_link'; did you mean '__clk_free_clk'? [-Werror=implicit-function-declaration]  
>       __clk_device_link(dev, clk);
>       ^~~~~~~~~~~~~~~~~
>       __clk_free_clk
>    drivers//clk/clkdev.c: In function 'clk_put':
> >> drivers//clk/clkdev.c:217:2: error: implicit declaration of function '__clk_device_unlink'; did you mean 'device_online'? [-Werror=implicit-function-declaration]  
>      __clk_device_unlink(clk);
>      ^~~~~~~~~~~~~~~~~~~
>      device_online
>    cc1: some warnings being treated as errors

I figured thanks to this report that this code won't compile with
architectures not compliant to the common clock framework. I see there
is the following block in clkdev.c.

#if defined(CONFIG_OF) && defined(CONFIG_COMMON_CLK)
#else
#endif

Would you agree with me adding dummy functions in the #else section
like:

static inline void __clk_device_link(struct device *consumer, struct clk *clk)
{
       return;
}

static inline void __clk_device_unlink(struct clk *clk)
{
       return;
}

Do you want me to also declare these functions in the #if section
(with the external keyword) to balance the above declarations?

Thanks for your input.
Miquèl

> 
> vim +209 drivers//clk/clkdev.c
> 
>    193	
>    194	struct clk *clk_get(struct device *dev, const char *con_id)
>    195	{
>    196		const char *dev_id = dev ? dev_name(dev) : NULL;
>    197		struct clk *clk = NULL;
>    198	
>    199		if (dev && dev->of_node) {
>    200			clk = __of_clk_get_by_name(dev->of_node, dev_id, con_id);
>    201			if (PTR_ERR(clk) == -EPROBE_DEFER)
>    202				return clk;
>    203		}
>    204	
>    205		if (IS_ERR_OR_NULL(clk))
>    206			clk = clk_get_sys(dev_id, con_id);
>    207	
>    208		if (!IS_ERR(clk))
>  > 209			__clk_device_link(dev, clk);  
>    210	
>    211		return clk;
>    212	}
>    213	EXPORT_SYMBOL(clk_get);
>    214	
>    215	void clk_put(struct clk *clk)
>    216	{
>  > 217		__clk_device_unlink(clk);  
>    218		__clk_put(clk);
>    219	}
>    220	EXPORT_SYMBOL(clk_put);
>    221	
> 
> ---
> 0-DAY kernel test infrastructure                Open Source Technology Center
> https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

Re: [PATCH 2/2] clk: core: link consumer with clock driver

[Stephen Boyd]

Quoting Miquel Raynal (2018-11-23 01:11:32)
> Would you agree with me adding dummy functions in the #else section
> like:
> 
> static inline void __clk_device_link(struct device *consumer, struct clk *clk)
> {
>        return;
> }
> 
> static inline void __clk_device_unlink(struct clk *clk)
> {
>        return;
> }
> 
> Do you want me to also declare these functions in the #if section
> (with the external keyword) to balance the above declarations?

Why can't we do the linking in __clk_get() and __clk_put()?

Re: [PATCH 2/2] clk: core: link consumer with clock driver

[Miquel Raynal]

Hi Stephen,

Stephen Boyd <sboyd@kernel.org> wrote on Fri, 30 Nov 2018 01:26:20
-0800:

> Quoting Miquel Raynal (2018-11-23 01:11:32)
> > Would you agree with me adding dummy functions in the #else section
> > like:
> > 
> > static inline void __clk_device_link(struct device *consumer, struct clk *clk)
> > {
> >        return;
> > }
> > 
> > static inline void __clk_device_unlink(struct clk *clk)
> > {
> >        return;
> > }
> > 
> > Do you want me to also declare these functions in the #if section
> > (with the external keyword) to balance the above declarations?  
> 
> Why can't we do the linking in __clk_get() and __clk_put()?
> 

Because we need the caller's 'struct device' to make the link and
this is not available in __clk_get(). I tried to ad it as parameter but
I don't think it is possible to retrieve a 'struct device' from the
device name. The functions where this is problematic are:
* clk.c:__of_clk_get_from_provider()
* clkdev.c:clk_get_sys()

By the way in my new version I called the helpers:
* clk_{link,unlink}_hierarchy()
* clk_{link,unlink}_consumer()

I will send a new version with these helpers, but if you have anything
in mind to help me achieve the above request, I will welcome the idea.


Thanks,
Miquèl

Re: [PATCH 2/2] clk: core: link consumer with clock driver

[Stephen Boyd]

Quoting Miquel Raynal (2018-11-30 02:20:52)
> Hi Stephen,
> 
> Stephen Boyd <sboyd@kernel.org> wrote on Fri, 30 Nov 2018 01:26:20
> -0800:
> 
> > Quoting Miquel Raynal (2018-11-23 01:11:32)
> > > Would you agree with me adding dummy functions in the #else section
> > > like:
> > > 
> > > static inline void __clk_device_link(struct device *consumer, struct clk *clk)
> > > {
> > >        return;
> > > }
> > > 
> > > static inline void __clk_device_unlink(struct clk *clk)
> > > {
> > >        return;
> > > }
> > > 
> > > Do you want me to also declare these functions in the #if section
> > > (with the external keyword) to balance the above declarations?  
> > 
> > Why can't we do the linking in __clk_get() and __clk_put()?
> > 
> 
> Because we need the caller's 'struct device' to make the link and
> this is not available in __clk_get(). I tried to ad it as parameter but
> I don't think it is possible to retrieve a 'struct device' from the
> device name. The functions where this is problematic are:
> * clk.c:__of_clk_get_from_provider()
> * clkdev.c:clk_get_sys()
> 
> By the way in my new version I called the helpers:
> * clk_{link,unlink}_hierarchy()
> * clk_{link,unlink}_consumer()
> 
> I will send a new version with these helpers, but if you have anything
> in mind to help me achieve the above request, I will welcome the idea.
> 

We can do the linking in __clk_get() and __clk_put() if we poke into the
struct clk -> struct clk_core and bury the struct device into each
clk_core structure.

Re: [PATCH 2/2] clk: core: link consumer with clock driver

[Miquel Raynal]

Hi Stephen,

Stephen Boyd <sboyd@kernel.org> wrote on Mon, 03 Dec 2018 11:20:31
-0800:

> Quoting Miquel Raynal (2018-11-30 02:20:52)
> > Hi Stephen,
> > 
> > Stephen Boyd <sboyd@kernel.org> wrote on Fri, 30 Nov 2018 01:26:20
> > -0800:
> >   
> > > Quoting Miquel Raynal (2018-11-23 01:11:32)  
> > > > Would you agree with me adding dummy functions in the #else section
> > > > like:
> > > > 
> > > > static inline void __clk_device_link(struct device *consumer, struct clk *clk)
> > > > {
> > > >        return;
> > > > }
> > > > 
> > > > static inline void __clk_device_unlink(struct clk *clk)
> > > > {
> > > >        return;
> > > > }
> > > > 
> > > > Do you want me to also declare these functions in the #if section
> > > > (with the external keyword) to balance the above declarations?    
> > > 
> > > Why can't we do the linking in __clk_get() and __clk_put()?
> > >   
> > 
> > Because we need the caller's 'struct device' to make the link and
> > this is not available in __clk_get(). I tried to ad it as parameter but
> > I don't think it is possible to retrieve a 'struct device' from the
> > device name. The functions where this is problematic are:
> > * clk.c:__of_clk_get_from_provider()
> > * clkdev.c:clk_get_sys()
> > 
> > By the way in my new version I called the helpers:
> > * clk_{link,unlink}_hierarchy()
> > * clk_{link,unlink}_consumer()
> > 
> > I will send a new version with these helpers, but if you have anything
> > in mind to help me achieve the above request, I will welcome the idea.
> >   
> 
> We can do the linking in __clk_get() and __clk_put() if we poke into the
> struct clk -> struct clk_core and bury the struct device into each
> clk_core structure.
> 

I meant the consumer device's structure. Yes, from a struct clk, the
first change in patch 1/2 let's us do clk->core->dev to get the clock
device. But for linking I need both the clock device and the consumer
device; and the latter will be missing in __clk_get().


Thanks,
Miquèl

Re: [PATCH 2/2] clk: core: link consumer with clock driver

[Stephen Boyd]

Quoting Miquel Raynal (2018-12-03 14:16:55)
> Stephen Boyd <sboyd@kernel.org> wrote on Mon, 03 Dec 2018 11:20:31
> -0800:
> > Quoting Miquel Raynal (2018-11-30 02:20:52)
> > > Stephen Boyd <sboyd@kernel.org> wrote on Fri, 30 Nov 2018 01:26:20
> > > -0800:
> > > > Quoting Miquel Raynal (2018-11-23 01:11:32)  
> > > 
> > > Because we need the caller's 'struct device' to make the link and
> > > this is not available in __clk_get(). I tried to ad it as parameter but
> > > I don't think it is possible to retrieve a 'struct device' from the
> > > device name. The functions where this is problematic are:
> > > * clk.c:__of_clk_get_from_provider()
> > > * clkdev.c:clk_get_sys()
> > > 
> > > By the way in my new version I called the helpers:
> > > * clk_{link,unlink}_hierarchy()
> > > * clk_{link,unlink}_consumer()
> > > 
> > > I will send a new version with these helpers, but if you have anything
> > > in mind to help me achieve the above request, I will welcome the idea.
> > >   
> > 
> > We can do the linking in __clk_get() and __clk_put() if we poke into the
> > struct clk -> struct clk_core and bury the struct device into each
> > clk_core structure.
> > 
> 
> I meant the consumer device's structure. Yes, from a struct clk, the
> first change in patch 1/2 let's us do clk->core->dev to get the clock
> device. But for linking I need both the clock device and the consumer
> device; and the latter will be missing in __clk_get().
> 

Ah ok, sounds like this is how it has to be then.

Re: [PATCH 2/2] clk: core: link consumer with clock driver

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 4539 bytes --]

Hi Miquel,

On Thu, Nov 22, 2018 at 10:22:12PM +0100, Miquel Raynal wrote:
> One major concern when, for instance, suspending/resuming a platform
> is to never access registers before the underlying clock has been
> resumed, otherwise most of the time the kernel will just crash. One
> solution is to use syscore operations when registering clock drivers
> suspend/resume callbacks. One problem of using syscore_ops is that the
> suspend/resume scheduling will depend on the order of the
> registrations, which brings (unacceptable) randomness in the process.
> 
> A feature called device links has been introduced to handle such
> situation. It creates dependencies between consumers and providers,
> enforcing e.g. the suspend/resume order when needed. Such feature is
> already in use for regulators.
> 
> Add device links support in the clock subsystem by creating/deleting
> the links at get/put time.
> 
> Example of a boot (ESPRESSObin, A3700 SoC) with devices linked to clocks:
> ahci-mvebu d00e0000.sata: Linked as a consumer to d0013000.nb-periph-clk
> mvneta d0030000.ethernet: Linked as a consumer to d0018000.sb-periph-clk
> xhci-hcd d0058000.usb: Linked as a consumer to d0018000.sb-periph-clk
> xenon-sdhci d00d0000.sdhci: Linked as a consumer to d0013000.nb-periph-clk
> xenon-sdhci d00d0000.sdhci: Dropping the link to d0013000.nb-periph-clk
> advk-pcie d0070000.pcie: Linked as a consumer to d0018000.sb-periph-clk
> xenon-sdhci d00d0000.sdhci: Linked as a consumer to d0013000.nb-periph-clk
> xenon-sdhci d00d0000.sdhci: Linked as a consumer to regulator.1
> cpu cpu0: Linked as a consumer to d0013000.nb-periph-clk
> cpu cpu0: Dropping the link to d0013000.nb-periph-clk
> cpu cpu0: Linked as a consumer to d0013000.nb-periph-clk
> 
> Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com>
> ---
>  drivers/clk/clk.c            | 20 ++++++++++++++++++++
>  drivers/clk/clkdev.c         | 13 ++++++++++---
>  include/linux/clk-provider.h |  2 ++
>  3 files changed, 32 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c
> index b799347c5fd6..33a0f2b0533a 100644
> --- a/drivers/clk/clk.c
> +++ b/drivers/clk/clk.c
> @@ -90,6 +90,7 @@ struct clk {
>  	unsigned long max_rate;
>  	unsigned int exclusive_count;
>  	struct hlist_node clks_node;
> +	struct device_link *link;
>  };
>  
>  /***           runtime pm          ***/
> @@ -262,6 +263,25 @@ struct clk_hw *__clk_get_hw(struct clk *clk)
>  }
>  EXPORT_SYMBOL_GPL(__clk_get_hw);
>  
> +void __clk_device_link(struct device *consumer, struct clk *clk)
> +{
> +	if (!consumer || !clk || !clk->core)
> +		return;
> +
> +	clk->link = device_link_add(consumer, clk->core->dev,
> +				    DL_FLAG_STATELESS);
> +}
> +EXPORT_SYMBOL_GPL(__clk_device_link);
> +
> +void __clk_device_unlink(struct clk *clk)
> +{
> +	if (!clk || !clk->link)
> +		return;
> +
> +	device_link_del(clk->link);
> +}
> +EXPORT_SYMBOL_GPL(__clk_device_unlink);
> +
>  unsigned int clk_hw_get_num_parents(const struct clk_hw *hw)
>  {
>  	return hw->core->num_parents;
> diff --git a/drivers/clk/clkdev.c b/drivers/clk/clkdev.c
> index 9ab3db8b3988..fccfd4c01457 100644
> --- a/drivers/clk/clkdev.c
> +++ b/drivers/clk/clkdev.c
> @@ -194,20 +194,27 @@ EXPORT_SYMBOL(clk_get_sys);
>  struct clk *clk_get(struct device *dev, const char *con_id)
>  {
>  	const char *dev_id = dev ? dev_name(dev) : NULL;
> -	struct clk *clk;
> +	struct clk *clk = NULL;
>  
>  	if (dev && dev->of_node) {
>  		clk = __of_clk_get_by_name(dev->of_node, dev_id, con_id);
> -		if (!IS_ERR(clk) || PTR_ERR(clk) == -EPROBE_DEFER)
> +		if (PTR_ERR(clk) == -EPROBE_DEFER)
>  			return clk;
>  	}
>  
> -	return clk_get_sys(dev_id, con_id);
> +	if (IS_ERR_OR_NULL(clk))
> +		clk = clk_get_sys(dev_id, con_id);
> +
> +	if (!IS_ERR(clk))
> +		__clk_device_link(dev, clk);
> +
> +	return clk;

I think this doesn't address all the cases. In your case, where you
have one consumer that is not a clock, and one provider that is a
clock, it works just fine.

However, if you have clocks providers chained, for example with one
oscillator, a clock controller, and a device, the link will be created
between the device and the controller, but there will be no link
between the controller and the oscillator.

Adding a link in __clk_init_parent looks like it would address that
case.

Maxime

-- 
Maxime Ripard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH 2/2] clk: core: link consumer with clock driver

[Miquel Raynal]

Hi Maxime,

Maxime Ripard <maxime.ripard@bootlin.com> wrote on Tue, 27 Nov 2018
13:38:58 +0100:

> Hi Miquel,
> 
> On Thu, Nov 22, 2018 at 10:22:12PM +0100, Miquel Raynal wrote:
> > One major concern when, for instance, suspending/resuming a platform
> > is to never access registers before the underlying clock has been
> > resumed, otherwise most of the time the kernel will just crash. One
> > solution is to use syscore operations when registering clock drivers
> > suspend/resume callbacks. One problem of using syscore_ops is that the
> > suspend/resume scheduling will depend on the order of the
> > registrations, which brings (unacceptable) randomness in the process.
> > 
> > A feature called device links has been introduced to handle such
> > situation. It creates dependencies between consumers and providers,
> > enforcing e.g. the suspend/resume order when needed. Such feature is
> > already in use for regulators.
> > 
> > Add device links support in the clock subsystem by creating/deleting
> > the links at get/put time.
> > 
> > Example of a boot (ESPRESSObin, A3700 SoC) with devices linked to clocks:
> > ahci-mvebu d00e0000.sata: Linked as a consumer to d0013000.nb-periph-clk
> > mvneta d0030000.ethernet: Linked as a consumer to d0018000.sb-periph-clk
> > xhci-hcd d0058000.usb: Linked as a consumer to d0018000.sb-periph-clk
> > xenon-sdhci d00d0000.sdhci: Linked as a consumer to d0013000.nb-periph-clk
> > xenon-sdhci d00d0000.sdhci: Dropping the link to d0013000.nb-periph-clk
> > advk-pcie d0070000.pcie: Linked as a consumer to d0018000.sb-periph-clk
> > xenon-sdhci d00d0000.sdhci: Linked as a consumer to d0013000.nb-periph-clk
> > xenon-sdhci d00d0000.sdhci: Linked as a consumer to regulator.1
> > cpu cpu0: Linked as a consumer to d0013000.nb-periph-clk
> > cpu cpu0: Dropping the link to d0013000.nb-periph-clk
> > cpu cpu0: Linked as a consumer to d0013000.nb-periph-clk
> > 
> > Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com>
> > ---
> >  drivers/clk/clk.c            | 20 ++++++++++++++++++++
> >  drivers/clk/clkdev.c         | 13 ++++++++++---
> >  include/linux/clk-provider.h |  2 ++
> >  3 files changed, 32 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c
> > index b799347c5fd6..33a0f2b0533a 100644
> > --- a/drivers/clk/clk.c
> > +++ b/drivers/clk/clk.c
> > @@ -90,6 +90,7 @@ struct clk {
> >  	unsigned long max_rate;
> >  	unsigned int exclusive_count;
> >  	struct hlist_node clks_node;
> > +	struct device_link *link;
> >  };
> >  
> >  /***           runtime pm          ***/
> > @@ -262,6 +263,25 @@ struct clk_hw *__clk_get_hw(struct clk *clk)
> >  }
> >  EXPORT_SYMBOL_GPL(__clk_get_hw);
> >  
> > +void __clk_device_link(struct device *consumer, struct clk *clk)
> > +{
> > +	if (!consumer || !clk || !clk->core)
> > +		return;
> > +
> > +	clk->link = device_link_add(consumer, clk->core->dev,
> > +				    DL_FLAG_STATELESS);
> > +}
> > +EXPORT_SYMBOL_GPL(__clk_device_link);
> > +
> > +void __clk_device_unlink(struct clk *clk)
> > +{
> > +	if (!clk || !clk->link)
> > +		return;
> > +
> > +	device_link_del(clk->link);
> > +}
> > +EXPORT_SYMBOL_GPL(__clk_device_unlink);
> > +
> >  unsigned int clk_hw_get_num_parents(const struct clk_hw *hw)
> >  {
> >  	return hw->core->num_parents;
> > diff --git a/drivers/clk/clkdev.c b/drivers/clk/clkdev.c
> > index 9ab3db8b3988..fccfd4c01457 100644
> > --- a/drivers/clk/clkdev.c
> > +++ b/drivers/clk/clkdev.c
> > @@ -194,20 +194,27 @@ EXPORT_SYMBOL(clk_get_sys);
> >  struct clk *clk_get(struct device *dev, const char *con_id)
> >  {
> >  	const char *dev_id = dev ? dev_name(dev) : NULL;
> > -	struct clk *clk;
> > +	struct clk *clk = NULL;
> >  
> >  	if (dev && dev->of_node) {
> >  		clk = __of_clk_get_by_name(dev->of_node, dev_id, con_id);
> > -		if (!IS_ERR(clk) || PTR_ERR(clk) == -EPROBE_DEFER)
> > +		if (PTR_ERR(clk) == -EPROBE_DEFER)
> >  			return clk;
> >  	}
> >  
> > -	return clk_get_sys(dev_id, con_id);
> > +	if (IS_ERR_OR_NULL(clk))
> > +		clk = clk_get_sys(dev_id, con_id);
> > +
> > +	if (!IS_ERR(clk))
> > +		__clk_device_link(dev, clk);
> > +
> > +	return clk;  
> 
> I think this doesn't address all the cases. In your case, where you
> have one consumer that is not a clock, and one provider that is a
> clock, it works just fine.
> 
> However, if you have clocks providers chained, for example with one
> oscillator, a clock controller, and a device, the link will be created
> between the device and the controller, but there will be no link
> between the controller and the oscillator.

Indeed, there is not clock_get() in this case so you are right, a link
is missing if we want to track dependencies between clocks.

> 
> Adding a link in __clk_init_parent looks like it would address that
> case.

I will add one, thanks for the pointer.

> 
> Maxime
> 

I think device links must be managed in the following situations:
* A device gets a clock -> add a link between the device and the clock,
  save it in 'struct clk'.
* A device puts a clock -> remove the above link.
* A clock gets registered -> link the core clock with the parent core
  clock.
* A clock gets reparentized -> remove the above link and if there is a
  new parent, link the core clock to the parent core clock.
* A clock gets deregistered -> remove the link with the parent core
  clock if any and remove the link with each child clock if any (this
  will be done automatically because each child will get reparentized
  first).

I assume that a clock getting deregistered is not an input source for
any device anymore and thus there is already no more link to destroy on
this regard.


Thanks,
Miquèl

Re: [PATCH 0/2] Link consumer with clock driver

[Stephen Boyd]

Quoting Miquel Raynal (2018-11-22 13:22:10)
> Hello,
> 
> While working on suspend to RAM feature, I ran into troubles multiple
> times when clocks where not suspending/resuming at the desired time. I
> had a look at the core and I think the same logic as in the
> regulator's core may be applied here to (very easily) fix this issue:
> using device links.

Thanks! I've wanted to add device links to the clk framework for a while
now but haven't gotten around to it. Can you expand on the scenario that
you've run into where you need this?

> 
> The only additional change I had to do was to always (when available)
> populate the device entry of the core clock structure so that it could
> be used later. This is the purpose of patch 1. Patch 2 actually adds
> support for device links.

That's fine. We should do it into clk_core all the time for other
reasons too.

> 
> As I am not used to hack into the clock subsystem I might have missed
> something big preventing such change but so far I could not see
> anything wrong with it. As this touches core code, I am of course
> entirely open to suggestions.
> 

Two questions:

 1) Do device links work if we make a loop between devices? I'm thinking
 of the case where we have two clock controllers that provide clks to
 each other and could potentially register some clks and then clk_get()
 the ones they depend on. I suspect loops don't work and so we need to
 be aware of this and somehow prevent it.

 2) Do we need to link clk consumers to anything besides the device
 providing the clk they get from clk_get()? Put another way, do we need
 to make links through the clk tree and any potential parent clks of
 whatever the device is using at the time. Would clk tree changing
 topology because some new parent is chosen need to affect
 suspend/resume ordering? Or would that all need to be handled by clk
 framework figuring out the ordering of the tree at suspend/resume time
 and suspending clock controller drivers in the right order?

Re: [PATCH 0/2] Link consumer with clock driver

[Miquel Raynal]

Hi Stephen,

+Maxime

Stephen Boyd <sboyd@kernel.org> wrote on Fri, 30 Nov 2018 01:24:57
-0800:

> Quoting Miquel Raynal (2018-11-22 13:22:10)
> > Hello,
> > 
> > While working on suspend to RAM feature, I ran into troubles multiple
> > times when clocks where not suspending/resuming at the desired time. I
> > had a look at the core and I think the same logic as in the
> > regulator's core may be applied here to (very easily) fix this issue:
> > using device links.  
> 
> Thanks! I've wanted to add device links to the clk framework for a while
> now but haven't gotten around to it. Can you expand on the scenario that
> you've run into where you need this?

Well, it happened that when working on suspend support on an
ESPRESSObin, the peripheral clock driver (armada_37xx_periph.c) was not
suspended after/resumed before all the drivers depending on it.

> 
> > 
> > The only additional change I had to do was to always (when available)
> > populate the device entry of the core clock structure so that it could
> > be used later. This is the purpose of patch 1. Patch 2 actually adds
> > support for device links.  
> 
> That's fine. We should do it into clk_core all the time for other
> reasons too.
> 
> > 
> > As I am not used to hack into the clock subsystem I might have missed
> > something big preventing such change but so far I could not see
> > anything wrong with it. As this touches core code, I am of course
> > entirely open to suggestions.
> >   
> 
> Two questions:
> 
>  1) Do device links work if we make a loop between devices? I'm thinking
>  of the case where we have two clock controllers that provide clks to
>  each other and could potentially register some clks and then clk_get()
>  the ones they depend on. I suspect loops don't work and so we need to
>  be aware of this and somehow prevent it.

For what I understand, when requesting the creation of a link there is
a check with "device_is_dependent()" that prevents reverse
dependencies, see [1].

Could you please share a situation where there is a loop between clocks
or between a device and its clock source? I don't see any.

[1] https://elixir.bootlin.com/linux/latest/source/drivers/base/core.c#L214

> 
>  2) Do we need to link clk consumers to anything besides the device
>  providing the clk they get from clk_get()? Put another way, do we need
>  to make links through the clk tree and any potential parent clks of
>  whatever the device is using at the time. Would clk tree changing
>  topology because some new parent is chosen need to affect
>  suspend/resume ordering? Or would that all need to be handled by clk
>  framework figuring out the ordering of the tree at suspend/resume time
>  and suspending clock controller drivers in the right order?
> 

In the v2 (coming) I followed Maxime Ripard suggestion and what you
describe above is handled automatically, let me explain the scenario.


The order of probe has no importance because the framework already
handles orphaned clocks so let's be simple and say there are two root
clocks, not depending on anything, that are probed first: xtal0 and
xtal1. None of these clocks have a parent, there is no device link in
the game, yet.

   +----------------+            +----------------+
   |                |            |                |
   |                |            |                |
   |   xtal0 core   |            |   xtal1 core   |
   |                |            |                |
   |                |            |                |
   +-------^^-------+            +-------^^-------+
           ||                            ||
           ||                            ||
   +----------------+            +----------------+
   |                |            |                |
   |   xtal0 clk    |            |   xtal1 clk    |
   |                |            |                |
   +----------------+            +----------------+
 
Then, a peripheral clock periph0 is probed. His parent is xtal1. The
clock_register_*() call will run __clk_init_parent() and a link between
periph0's core and xtal1's core will be created and stored in
periph0's core->parent_clk_link entry.

   +----------------+            +----------------+
   |                |            |                |
   |                |            |                |
   |   xtal0 core   |            |   xtal1 core   |
   |                |            |                |
   |                |            |                |
   +-------^^-------+            +-------^^-------+
           ||                            ||
           ||                            ||
   +----------------+            +----------------+
   |                |            |                |
   |   xtal0 clk    |            |   xtal1 clk    |
   |                |            |                |
   +----------------+            +-------^--------+
                                         |
                                         |
                          +--------------+
                          |   ->parent_clk_link
                          |
                  +----------------+
                  |                |
                  |                |
                  |  periph0 core  |
                  |                |
                  |                |
                  +-------^^-------+
                          ||
                          ||
                  +----------------+
                  |                |
                  |  periph0 clk 0 |
                  |                |
                  +----------------+

Then, device0 is probed and "get" the periph0 clock. clk_get() will be
called and a struct clk will be instantiated for device0 (called in
the figure clk 1). A link between device0 and the new clk 1 instance of
periph0 will be created and stored in the clk->consumer_link entry.

   +----------------+            +----------------+
   |                |            |                |
   |                |            |                |
   |   xtal0 core   |            |   xtal1 core   |
   |                |            |                |
   |                |            |                |
   +-------^^-------+            +-------^^-------+
           ||                            ||
           ||                            ||
   +----------------+            +----------------+
   |                |            |                |
   |   xtal0 clk    |            |   xtal1 clk    |
   |                |            |                |
   +----------------+            +-------^--------+
                                         |
                                         |
                          +--------------+
                          |   ->parent_clk_link
                          |
                  +----------------+
                  |                |
                  |                |
                  |  periph0 core  |
                  |                <-------------+
                  |                <-------------|
                  +-------^^-------+            ||
                          ||                    ||
                          ||                    ||
                  +----------------+    +----------------+
                  |                |    |                |
                  |  periph0 clk 0 |    |  periph0 clk 1 |
                  |                |    |                |
                  +----------------+    +----------------+
                                                |
                                                | ->consumer_link
                                                |
                                                |
                                                |
                                        +-------v--------+
                                        |    device0     |
                                        +----------------+

Right now, device0 is linked to periph0, itself linked to xtal1 so
everything is fine.

Now let's get some fun: the new parent of periph0 is xtal1. The process
will call clk_reparent(), periph0's core->parent_clk_link will be
destroyed and a new link to xtal1 will be setup and stored. The
situation is now that device0 is linked to periph0 and periph0 is
linked to xtal1, so the dependency between device0 and xtal1 is still
clear.

   +----------------+            +----------------+
   |                |            |                |
   |                |            |                |
   |   xtal0 core   |            |   xtal1 core   |
   |                |            |                |
   |                |            |                |
   +-------^^-------+            +-------^^-------+
           ||                            ||
           ||                            ||
   +----------------+            +----------------+
   |                |            |                |
   |   xtal0 clk    |            |   xtal1 clk    |
   |                |            |                |
   +-------^--------+            +----------------+
           |
           |                           \ /
           +----------------------------x
      ->parent_clk_link   |            / \
                          |
                  +----------------+
                  |                |
                  |                |
                  |  periph0 core  |
                  |                <-------------+
                  |                <-------------|
                  +-------^^-------+            ||
                          ||                    ||
                          ||                    ||
                  +----------------+    +----------------+
                  |                |    |                |
                  |  periph0 clk 0 |    |  periph0 clk 1 |
                  |                |    |                |
                  +----------------+    +----------------+
                                                |
                                                | ->consumer_link
                                                |
                                                |
                                                |
                                        +-------v--------+
                                        |    device0     |
                                        +----------------+

I assume periph0 cannot be removed while there are devices using it,
same for xtal0.

What can happen is that device0 'put' the clock periph0. The relevant
link is deleted and the clk instance dropped.

   +----------------+            +----------------+
   |                |            |                |
   |                |            |                |
   |   xtal0 core   |            |   xtal1 core   |
   |                |            |                |
   |                |            |                |
   +-------^^-------+            +-------^^-------+
           ||                            ||
           ||                            ||
   +----------------+            +----------------+
   |                |            |                |
   |   xtal0 clk    |            |   xtal1 clk    |
   |                |            |                |
   +-------^--------+            +----------------+
           |
           |                           \ /
           +----------------------------x
      ->parent_clk_link   |            / \
                          |
                  +----------------+
                  |                |
                  |                |
                  |  periph0 core  |
                  |                |
                  |                |
                  +-------^^-------+
                          ||
                          ||
                  +----------------+
                  |                |
                  |  periph0 clk 0 |
                  |                |
                  +----------------+

Now we can unregister periph0: link with the parent will be destroyed
and the clock may be safely removed. 

   +----------------+            +----------------+
   |                |            |                |
   |                |            |                |
   |   xtal0 core   |            |   xtal1 core   |
   |                |            |                |
   |                |            |                |
   +-------^^-------+            +-------^^-------+
           ||                            ||
           ||                            ||
   +----------------+            +----------------+
   |                |            |                |
   |   xtal0 clk    |            |   xtal1 clk    |
   |                |            |                |
   +----------------+            +----------------+


This is my understanding of the common clock framework and how links
can be added to it. I hope this was clear enough :)


Thanks,
Miquèl