[PATCH] PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

[PATCH] PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

[Stephen Boyd]

pm_qos_update_request_timeout() updates a qos and then schedules
a delayed work item to bring the qos back down to the default
after the timeout. When the work item runs, pm_qos_work_fn() will
call pm_qos_update_request() and deadlock because it tries to
cancel itself via cancel_delayed_work_sync(). Future callers of
that qos will also hang waiting to cancel the work that is
canceling itself. Before ed1ac6e (PM: don't use
[delayed_]work_pending(), 2013-01-11) this didn't happen because
the work function wouldn't try to cancel itself.

Let's just do the little bit of pm_qos_update_request() here so
that we don't deadlock.

Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
---
 kernel/power/qos.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/kernel/power/qos.c b/kernel/power/qos.c
index 06fe285..d52d314 100644
--- a/kernel/power/qos.c
+++ b/kernel/power/qos.c
@@ -308,7 +308,11 @@ static void pm_qos_work_fn(struct work_struct *work)
 						  struct pm_qos_request,
 						  work);
 
-	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
+	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
+		pm_qos_update_target(
+				pm_qos_array[req->pm_qos_class]->constraints,
+				&req->node, PM_QOS_UPDATE_REQ,
+				PM_QOS_DEFAULT_VALUE);
 }
 
 /**
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCH] PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

[Stephen Boyd]

On 08/08/13 13:13, Stephen Boyd wrote:
> pm_qos_update_request_timeout() updates a qos and then schedules
> a delayed work item to bring the qos back down to the default
> after the timeout. When the work item runs, pm_qos_work_fn() will
> call pm_qos_update_request() and deadlock because it tries to
> cancel itself via cancel_delayed_work_sync(). Future callers of
> that qos will also hang waiting to cancel the work that is
> canceling itself. Before ed1ac6e (PM: don't use
> [delayed_]work_pending(), 2013-01-11) this didn't happen because
> the work function wouldn't try to cancel itself.
>
> Let's just do the little bit of pm_qos_update_request() here so
> that we don't deadlock.
>
> Cc: Tejun Heo <tj@kernel.org>
> Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
> ---

Ping?

>  kernel/power/qos.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/power/qos.c b/kernel/power/qos.c
> index 06fe285..d52d314 100644
> --- a/kernel/power/qos.c
> +++ b/kernel/power/qos.c
> @@ -308,7 +308,11 @@ static void pm_qos_work_fn(struct work_struct *work)
>  						  struct pm_qos_request,
>  						  work);
>  
> -	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
> +	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
> +		pm_qos_update_target(
> +				pm_qos_array[req->pm_qos_class]->constraints,
> +				&req->node, PM_QOS_UPDATE_REQ,
> +				PM_QOS_DEFAULT_VALUE);
>  }
>  
>  /**


-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCH] PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

[Tejun Heo]

Hello, Stephen.

On Thu, Aug 08, 2013 at 01:13:57PM -0700, Stephen Boyd wrote:
> pm_qos_update_request_timeout() updates a qos and then schedules
> a delayed work item to bring the qos back down to the default
> after the timeout. When the work item runs, pm_qos_work_fn() will
> call pm_qos_update_request() and deadlock because it tries to
> cancel itself via cancel_delayed_work_sync(). Future callers of
> that qos will also hang waiting to cancel the work that is
> canceling itself. Before ed1ac6e (PM: don't use
> [delayed_]work_pending(), 2013-01-11) this didn't happen because
> the work function wouldn't try to cancel itself.

I see.  That must have been racy tho.  If the work item execution
races someone else queuing the work item, the same deadlock could
happen, right?

> Let's just do the little bit of pm_qos_update_request() here so
> that we don't deadlock.
>
> Cc: Tejun Heo <tj@kernel.org>
> Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
> ---
>  kernel/power/qos.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/power/qos.c b/kernel/power/qos.c
> index 06fe285..d52d314 100644
> --- a/kernel/power/qos.c
> +++ b/kernel/power/qos.c
> @@ -308,7 +308,11 @@ static void pm_qos_work_fn(struct work_struct *work)
>  						  struct pm_qos_request,
>  						  work);
>  
> -	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
> +	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
> +		pm_qos_update_target(
> +				pm_qos_array[req->pm_qos_class]->constraints,
> +				&req->node, PM_QOS_UPDATE_REQ,
> +				PM_QOS_DEFAULT_VALUE);

Maybe it'd be cleaner to add a param or internal variant of
pm_qos_update_request()?

Thanks.

-- 
tejun

Re: [PATCH] PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

[Stephen Boyd]

On 08/13/13 09:43, Tejun Heo wrote:
> Hello, Stephen.
>
> On Thu, Aug 08, 2013 at 01:13:57PM -0700, Stephen Boyd wrote:
>> pm_qos_update_request_timeout() updates a qos and then schedules
>> a delayed work item to bring the qos back down to the default
>> after the timeout. When the work item runs, pm_qos_work_fn() will
>> call pm_qos_update_request() and deadlock because it tries to
>> cancel itself via cancel_delayed_work_sync(). Future callers of
>> that qos will also hang waiting to cancel the work that is
>> canceling itself. Before ed1ac6e (PM: don't use
>> [delayed_]work_pending(), 2013-01-11) this didn't happen because
>> the work function wouldn't try to cancel itself.
> I see.  That must have been racy tho.  If the work item execution
> races someone else queuing the work item, the same deadlock could
> happen, right?

Yes you're right. It was always racy.

>
>> Let's just do the little bit of pm_qos_update_request() here so
>> that we don't deadlock.
>>
>> Cc: Tejun Heo <tj@kernel.org>
>> Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
>> ---
>>  kernel/power/qos.c | 6 +++++-
>>  1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/kernel/power/qos.c b/kernel/power/qos.c
>> index 06fe285..d52d314 100644
>> --- a/kernel/power/qos.c
>> +++ b/kernel/power/qos.c
>> @@ -308,7 +308,11 @@ static void pm_qos_work_fn(struct work_struct *work)
>>  						  struct pm_qos_request,
>>  						  work);
>>  
>> -	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
>> +	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
>> +		pm_qos_update_target(
>> +				pm_qos_array[req->pm_qos_class]->constraints,
>> +				&req->node, PM_QOS_UPDATE_REQ,
>> +				PM_QOS_DEFAULT_VALUE);
> Maybe it'd be cleaner to add a param or internal variant of
> pm_qos_update_request()?

Maybe, but I was trying to make a minimal fix here.

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCH] PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

[Tejun Heo]

Hello,

On Tue, Aug 13, 2013 at 09:46:26AM -0700, Stephen Boyd wrote:
> >> +	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
> >> +		pm_qos_update_target(
> >> +				pm_qos_array[req->pm_qos_class]->constraints,
> >> +				&req->node, PM_QOS_UPDATE_REQ,
> >> +				PM_QOS_DEFAULT_VALUE);
> > Maybe it'd be cleaner to add a param or internal variant of
> > pm_qos_update_request()?
> 
> Maybe, but I was trying to make a minimal fix here.

Hmmm.... it just looks like things can easily get out of sync with the
complex function call.  I don't think it'll be too invasive if you
introduce an internal variant which doesn't do the canceling.  Rafael,
what do you think?

Thanks.

-- 
tejun

Re: [PATCH] PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

[Rafael J. Wysocki]

On Tuesday, August 13, 2013 01:01:46 PM Tejun Heo wrote:
> Hello,
> 
> On Tue, Aug 13, 2013 at 09:46:26AM -0700, Stephen Boyd wrote:
> > >> +	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
> > >> +		pm_qos_update_target(
> > >> +				pm_qos_array[req->pm_qos_class]->constraints,
> > >> +				&req->node, PM_QOS_UPDATE_REQ,
> > >> +				PM_QOS_DEFAULT_VALUE);
> > > Maybe it'd be cleaner to add a param or internal variant of
> > > pm_qos_update_request()?
> > 
> > Maybe, but I was trying to make a minimal fix here.
> 
> Hmmm.... it just looks like things can easily get out of sync with the
> complex function call.

Yes, that's just duplicated code.

> I don't think it'll be too invasive if you introduce an internal variant
> which doesn't do the canceling.  Rafael, what do you think?

I'd move the part of pm_qos_update_request() below the
cancel_delayed_work_sync() to a separate static function that'd be
called from two places.

Thanks,
Rafael


-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

Re: [PATCH] PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

[Stephen Boyd]

On 08/13, Rafael J. Wysocki wrote:
> On Tuesday, August 13, 2013 01:01:46 PM Tejun Heo wrote:
> > Hello,
> > 
> > On Tue, Aug 13, 2013 at 09:46:26AM -0700, Stephen Boyd wrote:
> > > >> +	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
> > > >> +		pm_qos_update_target(
> > > >> +				pm_qos_array[req->pm_qos_class]->constraints,
> > > >> +				&req->node, PM_QOS_UPDATE_REQ,
> > > >> +				PM_QOS_DEFAULT_VALUE);
> > > > Maybe it'd be cleaner to add a param or internal variant of
> > > > pm_qos_update_request()?
> > > 
> > > Maybe, but I was trying to make a minimal fix here.
> > 
> > Hmmm.... it just looks like things can easily get out of sync with the
> > complex function call.
> 
> Yes, that's just duplicated code.
> 
> > I don't think it'll be too invasive if you introduce an internal variant
> > which doesn't do the canceling.  Rafael, what do you think?
> 
> I'd move the part of pm_qos_update_request() below the
> cancel_delayed_work_sync() to a separate static function that'd be
> called from two places.
> 

Ok I will throw this all into one patch and resend.

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

[PATCH] PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

[Stephen Boyd]

pm_qos_update_request_timeout() updates a qos and then schedules
a delayed work item to bring the qos back down to the default
after the timeout. When the work item runs, pm_qos_work_fn() will
call pm_qos_update_request() and deadlock because it tries to
cancel itself via cancel_delayed_work_sync(). Future callers of
that qos will also hang waiting to cancel the work that is
canceling itself. Let's extract the little bit of code that does
the real work of pm_qos_update_request() and call it from the
work function so that we don't deadlock.

Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
---
 kernel/power/qos.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/kernel/power/qos.c b/kernel/power/qos.c
index 06fe285..a394297 100644
--- a/kernel/power/qos.c
+++ b/kernel/power/qos.c
@@ -296,6 +296,17 @@ int pm_qos_request_active(struct pm_qos_request *req)
 }
 EXPORT_SYMBOL_GPL(pm_qos_request_active);
 
+static void __pm_qos_update_request(struct pm_qos_request *req,
+			   s32 new_value)
+{
+	trace_pm_qos_update_request(req->pm_qos_class, new_value);
+
+	if (new_value != req->node.prio)
+		pm_qos_update_target(
+			pm_qos_array[req->pm_qos_class]->constraints,
+			&req->node, PM_QOS_UPDATE_REQ, new_value);
+}
+
 /**
  * pm_qos_work_fn - the timeout handler of pm_qos_update_request_timeout
  * @work: work struct for the delayed work (timeout)
@@ -308,7 +319,7 @@ static void pm_qos_work_fn(struct work_struct *work)
 						  struct pm_qos_request,
 						  work);
 
-	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
+	__pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
 }
 
 /**
@@ -364,12 +375,7 @@ void pm_qos_update_request(struct pm_qos_request *req,
 	}
 
 	cancel_delayed_work_sync(&req->work);
-
-	trace_pm_qos_update_request(req->pm_qos_class, new_value);
-	if (new_value != req->node.prio)
-		pm_qos_update_target(
-			pm_qos_array[req->pm_qos_class]->constraints,
-			&req->node, PM_QOS_UPDATE_REQ, new_value);
+	__pm_qos_update_request(req, new_value);
 }
 EXPORT_SYMBOL_GPL(pm_qos_update_request);
 
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCH] PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

[Tejun Heo]

Hello,

On Tue, Aug 13, 2013 at 02:12:40PM -0700, Stephen Boyd wrote:
> @@ -308,7 +319,7 @@ static void pm_qos_work_fn(struct work_struct *work)
>  						  struct pm_qos_request,
>  						  work);
>  
> -	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
> +	__pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);

Maybe a short comment explaining why this is different would be nice?
Other than that,

 Reviewed-by: Tejun Heo <tj@kernel.org>

Thanks.

-- 
tejun

Re: [PATCH] PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

[Rafael J. Wysocki]

On Tuesday, August 13, 2013 06:13:25 PM Tejun Heo wrote:
> Hello,
> 
> On Tue, Aug 13, 2013 at 02:12:40PM -0700, Stephen Boyd wrote:
> > @@ -308,7 +319,7 @@ static void pm_qos_work_fn(struct work_struct *work)
> >  						  struct pm_qos_request,
> >  						  work);
> >  
> > -	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
> > +	__pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
> 
> Maybe a short comment explaining why this is different would be nice?
> Other than that,
> 
>  Reviewed-by: Tejun Heo <tj@kernel.org>

Thanks guys, I'm going to push that as a fix for 3.11-rc6 and stable.

Rafael