[RFC] clk: Remove cached cores in parent map during unregister

[RFC] clk: Remove cached cores in parent map during unregister

[Bjorn Andersson]

As clocks are registered their parents are resolved and the parent_map
is updated to cache the clk_core objects of each existing parent.
But in the event of a clock being unregistered this cache will carry
dangling pointers if not invalidated, so do this for all children of the
clock being unregistered.

Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
---

This resolves the issue seen where the DSI PLL (and it's provided clocks) is
being registered and unregistered multiple times due to probe deferral.

Marking it RFC because I don't fully understand the life of the clock yet.

 drivers/clk/clk.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c
index c0990703ce54..8cd1ad977c50 100644
--- a/drivers/clk/clk.c
+++ b/drivers/clk/clk.c
@@ -2423,11 +2423,14 @@ bool clk_has_parent(struct clk *clk, struct clk *parent)
 EXPORT_SYMBOL_GPL(clk_has_parent);
 
 static int clk_core_set_parent_nolock(struct clk_core *core,
-				      struct clk_core *parent)
+				      struct clk_core *parent,
+				      bool invalidate_parent)
 {
+	struct clk_core *old_parent = core->parent;
 	int ret = 0;
 	int p_index = 0;
 	unsigned long p_rate = 0;
+	int i;
 
 	lockdep_assert_held(&prepare_lock);
 
@@ -2481,6 +2484,14 @@ static int clk_core_set_parent_nolock(struct clk_core *core,
 		__clk_recalc_accuracies(core);
 	}
 
+	/* invalidate the parent cache */
+	if (!parent && invalidate_parent) {
+		for (i = 0; i < core->num_parents; i++) {
+			if (core->parents[i].core == old_parent)
+				core->parents[i].core = NULL;
+		}
+	}
+
 runtime_put:
 	clk_pm_runtime_put(core);
 
@@ -2517,7 +2528,8 @@ int clk_set_parent(struct clk *clk, struct clk *parent)
 		clk_core_rate_unprotect(clk->core);
 
 	ret = clk_core_set_parent_nolock(clk->core,
-					 parent ? parent->core : NULL);
+					 parent ? parent->core : NULL,
+					 false);
 
 	if (clk->exclusive_count)
 		clk_core_rate_protect(clk->core);
@@ -3772,7 +3784,7 @@ void clk_unregister(struct clk *clk)
 		/* Reparent all children to the orphan list. */
 		hlist_for_each_entry_safe(child, t, &clk->core->children,
 					  child_node)
-			clk_core_set_parent_nolock(child, NULL);
+			clk_core_set_parent_nolock(child, NULL, true);
 	}
 
 	hlist_del_init(&clk->core->child_node);
-- 
2.18.0

Re: [RFC] clk: Remove cached cores in parent map during unregister

[Stephen Boyd]

Quoting Bjorn Andersson (2019-07-22 22:14:46)
> As clocks are registered their parents are resolved and the parent_map
> is updated to cache the clk_core objects of each existing parent.
> But in the event of a clock being unregistered this cache will carry
> dangling pointers if not invalidated, so do this for all children of the
> clock being unregistered.
> 
> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
> ---
> 
> This resolves the issue seen where the DSI PLL (and it's provided clocks) is
> being registered and unregistered multiple times due to probe deferral.
> 
> Marking it RFC because I don't fully understand the life of the clock yet.

The concept sounds sane but the implementation is going to be not much
fun. The problem is that a clk can be in many different parent caches,
even ones for clks that aren't currently parented to it. We would need
to walk the entire tree(s) and find anywhere that we've cached the
clk_core pointer and invalidate it. Maybe we can speed that up a little
bit by keeping a reference to the entry of each parent cache that is for
the parent we're removing, essentially holding an inverse cache, but I'm
not sure it will provide any benefit besides wasting space for this one
operation that we shouldn't be doing very often if at all.

It certainly sounds easier to iterate through the whole tree and just
invalidate entries in all the caches under the prepare lock. We can
optimize it later.

> 
>  drivers/clk/clk.c | 18 +++++++++++++++---
>  1 file changed, 15 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c
> index c0990703ce54..8cd1ad977c50 100644
> --- a/drivers/clk/clk.c
> +++ b/drivers/clk/clk.c
> @@ -2517,7 +2528,8 @@ int clk_set_parent(struct clk *clk, struct clk *parent)
>                 clk_core_rate_unprotect(clk->core);
>  
>         ret = clk_core_set_parent_nolock(clk->core,
> -                                        parent ? parent->core : NULL);
> +                                        parent ? parent->core : NULL,
> +                                        false);

Nitpick. I'd prefer another function to invalidate the cache of a clk
given another clk_core pointer.

>  
>         if (clk->exclusive_count)
>                 clk_core_rate_protect(clk->core);

Re: [RFC] clk: Remove cached cores in parent map during unregister

[Stephen Boyd]

Quoting Stephen Boyd (2019-07-29 15:46:51)
> Quoting Bjorn Andersson (2019-07-22 22:14:46)
> > As clocks are registered their parents are resolved and the parent_map
> > is updated to cache the clk_core objects of each existing parent.
> > But in the event of a clock being unregistered this cache will carry
> > dangling pointers if not invalidated, so do this for all children of the
> > clock being unregistered.
> > 
> > Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
> > ---
> > 
> > This resolves the issue seen where the DSI PLL (and it's provided clocks) is
> > being registered and unregistered multiple times due to probe deferral.
> > 
> > Marking it RFC because I don't fully understand the life of the clock yet.
> 
> The concept sounds sane but the implementation is going to be not much
> fun. The problem is that a clk can be in many different parent caches,
> even ones for clks that aren't currently parented to it. We would need
> to walk the entire tree(s) and find anywhere that we've cached the
> clk_core pointer and invalidate it. Maybe we can speed that up a little
> bit by keeping a reference to the entry of each parent cache that is for
> the parent we're removing, essentially holding an inverse cache, but I'm
> not sure it will provide any benefit besides wasting space for this one
> operation that we shouldn't be doing very often if at all.
> 
> It certainly sounds easier to iterate through the whole tree and just
> invalidate entries in all the caches under the prepare lock. We can
> optimize it later.

Here's an attempt at the simple approach. There's another problem where
the cached 'hw' member of the parent data is held around when we don't
know when the caller has destroyed it. Not much else we can do for that
though.

---8<---
diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c
index c0990703ce54..f42a803fb11a 100644
--- a/drivers/clk/clk.c
+++ b/drivers/clk/clk.c
@@ -3737,6 +3737,37 @@ static const struct clk_ops clk_nodrv_ops = {
 	.set_parent	= clk_nodrv_set_parent,
 };
 
+static void clk_core_evict_parent_cache_subtree(struct clk_core *root,
+						struct clk_core *target)
+{
+	int i;
+	struct clk_core *child;
+
+	if (!root)
+		return;
+
+	for (i = 0; i < root->num_parents; i++)
+		if (root->parents[i].core == target)
+			root->parents[i].core = NULL;
+
+	hlist_for_each_entry(child, &root->children, child_node)
+		clk_core_evict_parent_cache_subtree(child, target);
+}
+
+/* Remove this clk from all parent caches */
+static void clk_core_evict_parent_cache(struct clk_core *core)
+{
+	struct hlist_head **lists;
+	struct clk_core *root;
+
+	lockdep_assert_held(&prepare_lock);
+
+	for (lists = all_lists; *lists; lists++)
+		hlist_for_each_entry(root, *lists, child_node)
+			clk_core_evict_parent_cache_subtree(root, core);
+
+}
+
 /**
  * clk_unregister - unregister a currently registered clock
  * @clk: clock to unregister
@@ -3775,6 +3806,8 @@ void clk_unregister(struct clk *clk)
 			clk_core_set_parent_nolock(child, NULL);
 	}
 
+	clk_core_evict_parent_cache(clk->core);
+
 	hlist_del_init(&clk->core->child_node);
 
 	if (clk->core->prepare_count)

Re: [RFC] clk: Remove cached cores in parent map during unregister

[saiprakash.ranjan]

Hi Stephen,

On 2019-08-21 18:10, Stephen Boyd wrote:
> 
> Here's an attempt at the simple approach. There's another problem where
> the cached 'hw' member of the parent data is held around when we don't
> know when the caller has destroyed it. Not much else we can do for that
> though.
> 
> ---8<---
> diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c
> index c0990703ce54..f42a803fb11a 100644
> --- a/drivers/clk/clk.c
> +++ b/drivers/clk/clk.c
> @@ -3737,6 +3737,37 @@ static const struct clk_ops clk_nodrv_ops = {
>  	.set_parent	= clk_nodrv_set_parent,
>  };
> 
> +static void clk_core_evict_parent_cache_subtree(struct clk_core *root,
> +						struct clk_core *target)
> +{
> +	int i;
> +	struct clk_core *child;
> +
> +	if (!root)
> +		return;
> +
> +	for (i = 0; i < root->num_parents; i++)
> +		if (root->parents[i].core == target)
> +			root->parents[i].core = NULL;
> +
> +	hlist_for_each_entry(child, &root->children, child_node)
> +		clk_core_evict_parent_cache_subtree(child, target);
> +}
> +
> +/* Remove this clk from all parent caches */
> +static void clk_core_evict_parent_cache(struct clk_core *core)
> +{
> +	struct hlist_head **lists;
> +	struct clk_core *root;
> +
> +	lockdep_assert_held(&prepare_lock);
> +
> +	for (lists = all_lists; *lists; lists++)
> +		hlist_for_each_entry(root, *lists, child_node)
> +			clk_core_evict_parent_cache_subtree(root, core);
> +
> +}
> +
>  /**
>   * clk_unregister - unregister a currently registered clock
>   * @clk: clock to unregister
> @@ -3775,6 +3806,8 @@ void clk_unregister(struct clk *clk)
>  			clk_core_set_parent_nolock(child, NULL);
>  	}
> 
> +	clk_core_evict_parent_cache(clk->core);
> +
>  	hlist_del_init(&clk->core->child_node);
> 
>  	if (clk->core->prepare_count)


Thanks for the patch. This fixes the dispcc issue which I was observing 
with latest kernel on cheza.
You can have the tested by when you post the actual patch.

Tested-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org>

For reference, below is the error log:

[    3.992243] msm ae00000.mdss: Adding to iommu group 1
[    3.999877] msm ae00000.mdss: bound ae01000.mdp (ops 
0xffffff8010a87b78)
[    4.023597] msm ae00000.mdss: failed to bind ae94000.dsi (ops 
0xffffff8010a8b360): -517
[    4.031898] msm ae00000.mdss: master bind failed: -517
[    5.297509] msm ae00000.mdss: bound ae01000.mdp (ops 
0xffffff8010a87b78)
[    5.320906] msm ae00000.mdss: failed to bind ae94000.dsi (ops 
0xffffff8010a8b360): -517
[    5.329261] msm ae00000.mdss: master bind failed: -517
[    5.648667] msm ae00000.mdss: bound ae01000.mdp (ops 
0xffffff8010a87b78)
[    5.672005] clk_core_set_parent_nolock: clk dsi0_phy_pll_out_byteclk 
can not be parent of clk disp_cc_mdss_byte0_clk_src p_index=-22 <--- 
error
[    5.699799] msm ae00000.mdss: failed to bind ae94000.dsi (ops 
0xffffff8010a8b360): -22
[    5.708086] msm ae00000.mdss: master bind failed: -22
[    5.714052] msm: probe of ae00000.mdss failed with error -22

[    6.033541] clk_core_set_parent_nolock: clk dsi0_phy_pll_out_dsiclk 
can not be parent of clk disp_cc_mdss_pclk0_clk_src p_index=-22 <--- 
error

Thanks,
Sai

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [RFC] clk: Remove cached cores in parent map during unregister

[Stephen Boyd]

Quoting Stephen Boyd (2019-08-21 11:10:08)
> Quoting Stephen Boyd (2019-07-29 15:46:51)
> > Quoting Bjorn Andersson (2019-07-22 22:14:46)
> > > As clocks are registered their parents are resolved and the parent_map
> > > is updated to cache the clk_core objects of each existing parent.
> > > But in the event of a clock being unregistered this cache will carry
> > > dangling pointers if not invalidated, so do this for all children of the
> > > clock being unregistered.
> > > 
> > > Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
> > > ---
> > > 
> > > This resolves the issue seen where the DSI PLL (and it's provided clocks) is
> > > being registered and unregistered multiple times due to probe deferral.
> > > 
> > > Marking it RFC because I don't fully understand the life of the clock yet.
> > 
> > The concept sounds sane but the implementation is going to be not much
> > fun. The problem is that a clk can be in many different parent caches,
> > even ones for clks that aren't currently parented to it. We would need
> > to walk the entire tree(s) and find anywhere that we've cached the
> > clk_core pointer and invalidate it. Maybe we can speed that up a little
> > bit by keeping a reference to the entry of each parent cache that is for
> > the parent we're removing, essentially holding an inverse cache, but I'm
> > not sure it will provide any benefit besides wasting space for this one
> > operation that we shouldn't be doing very often if at all.
> > 
> > It certainly sounds easier to iterate through the whole tree and just
> > invalidate entries in all the caches under the prepare lock. We can
> > optimize it later.
> 
> Here's an attempt at the simple approach. There's another problem where
> the cached 'hw' member of the parent data is held around when we don't
> know when the caller has destroyed it. Not much else we can do for that
> though.
> 
> ---8<---
> diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c
> index c0990703ce54..f42a803fb11a 100644
> --- a/drivers/clk/clk.c
> +++ b/drivers/clk/clk.c
> @@ -3737,6 +3737,37 @@ static const struct clk_ops clk_nodrv_ops = {
>         .set_parent     = clk_nodrv_set_parent,
>  };
>  
> +static void clk_core_evict_parent_cache_subtree(struct clk_core *root,
> +                                               struct clk_core *target)
> +{
> +       int i;
> +       struct clk_core *child;
> +
> +       if (!root)
> +               return;

I don't think we need this part. Child is always a valid pointer.

Re: [RFC] clk: Remove cached cores in parent map during unregister

[Raul Rangel]

On Mon, Aug 26, 2019 at 02:24:14PM -0700, Stephen Boyd wrote:
> > 
> > ---8<---
> > diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c
> > index c0990703ce54..f42a803fb11a 100644
> > --- a/drivers/clk/clk.c
> > +++ b/drivers/clk/clk.c
> > @@ -3737,6 +3737,37 @@ static const struct clk_ops clk_nodrv_ops = {
> >         .set_parent     = clk_nodrv_set_parent,
> >  };
> >  
> > +static void clk_core_evict_parent_cache_subtree(struct clk_core *root,
> > +                                               struct clk_core *target)
> > +{
> > +       int i;
> > +       struct clk_core *child;
> > +
> > +       if (!root)
> > +               return;
> 
> I don't think we need this part. Child is always a valid pointer.
> 

Bjorn or Saiprakash
Are there any plans to send out Stephen's proposed patch?

Thanks

Re: [RFC] clk: Remove cached cores in parent map during unregister

[Sai Prakash Ranjan]

On 2019-09-17 21:04, Raul Rangel wrote:
> On Mon, Aug 26, 2019 at 02:24:14PM -0700, Stephen Boyd wrote:
>> >
>> > ---8<---
>> > diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c
>> > index c0990703ce54..f42a803fb11a 100644
>> > --- a/drivers/clk/clk.c
>> > +++ b/drivers/clk/clk.c
>> > @@ -3737,6 +3737,37 @@ static const struct clk_ops clk_nodrv_ops = {
>> >         .set_parent     = clk_nodrv_set_parent,
>> >  };
>> >
>> > +static void clk_core_evict_parent_cache_subtree(struct clk_core *root,
>> > +                                               struct clk_core *target)
>> > +{
>> > +       int i;
>> > +       struct clk_core *child;
>> > +
>> > +       if (!root)
>> > +               return;
>> 
>> I don't think we need this part. Child is always a valid pointer.
>> 
> 
> Bjorn or Saiprakash
> Are there any plans to send out Stephen's proposed patch?
> 

Stephen has already sent out an updated patch for this here:

https://lore.kernel.org/lkml/20190828181959.204401-1-sboyd@kernel.org/

Thanks,
Sai

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member
of Code Aurora Forum, hosted by The Linux Foundation