[PATCH 00/25] Fix some PM runtime issues at the media subsystem

[PATCH 00/25] Fix some PM runtime issues at the media subsystem

[Mauro Carvalho Chehab]

As part of an effort to cleanup pm_runtime*get* calls, I detected a number
of issues at the media subsystem.

Most of the patches here were submitted previously at:

	https://lore.kernel.org/linux-media/cover.1619621413.git.mchehab+huawei@kernel.org/

This series contain just the bug fixes and other related issues that are
present with the current code on media.

It address the points from the existing reviews. I also did my own
set of reviews, in order to avoid regressions.

Changes from v4 of the previous changeset:

- reworked i2c/css RPM get logic;
- dropped two patches that could cause regressions;
- am437x: keep using pm_runtime_get_sync on suspend/resume;
- atmel: fix the returned code and add a print on failures at start streaming;
- simplify some checks for return code > 0;
- mdk-vcodec: properly handle RPM errors at device on logic;
- venus: rework venus_sys_error_handler() logic;
- sti/delta: fix an issue at the error checking logic.

Mauro Carvalho Chehab (25):
  staging: media: rkvdec: fix pm_runtime_get_sync() usage count
  staging: media: imx7-mipi-csis: fix pm_runtime_get_sync() usage count
  media: venus: Rework error fail recover logic
  media: s5p_cec: decrement usage count if disabled
  media: i2c: ccs-core: return the right error code at suspend
  media: i2c: imx334: fix the pm runtime get logic
  media: exynos-gsc: don't resume at remove time
  media: atmel: properly get pm_runtime
  media: hantro: do a PM resume earlier
  media: marvel-ccic: fix some issues when getting pm_runtime
  media: mdk-mdp: fix pm_runtime_get_sync() usage count
  media: rcar_fdp1: simplify error check logic at fdp_open()
  media: rcar_fdp1: fix pm_runtime_get_sync() usage count
  media: renesas-ceu: Properly check for PM errors
  media: s5p: fix pm_runtime_get_sync() usage count
  media: am437x: fix pm_runtime_get_sync() usage count
  media: sh_vou: fix pm_runtime_get_sync() usage count
  media: mtk-vcodec: fix PM runtime get logic
  media: s5p-jpeg: fix pm_runtime_get_sync() usage count
  media: sti/delta: use pm_runtime_resume_and_get()
  media: sunxi: fix pm_runtime_get_sync() usage count
  media: sti/bdisp: fix pm_runtime_get_sync() usage count
  media: exynos4-is: fix pm_runtime_get_sync() usage count
  media: exynos-gsc: fix pm_runtime_get_sync() usage count
  media: i2c: ccs-core: fix pm_runtime_get_sync() usage count

 drivers/media/cec/platform/s5p/s5p_cec.c      |  7 ++-
 drivers/media/i2c/ccs/ccs-core.c              | 41 ++++++++-----
 drivers/media/i2c/imx334.c                    |  7 ++-
 drivers/media/platform/am437x/am437x-vpfe.c   | 15 ++++-
 drivers/media/platform/atmel/atmel-isc-base.c | 30 +++++++---
 drivers/media/platform/atmel/atmel-isi.c      | 19 ++++--
 drivers/media/platform/exynos-gsc/gsc-core.c  | 11 ++--
 drivers/media/platform/exynos-gsc/gsc-m2m.c   |  4 +-
 .../media/platform/exynos4-is/fimc-capture.c  |  6 +-
 drivers/media/platform/exynos4-is/fimc-is.c   |  4 +-
 .../platform/exynos4-is/fimc-isp-video.c      |  3 +-
 drivers/media/platform/exynos4-is/fimc-isp.c  |  7 +--
 drivers/media/platform/exynos4-is/fimc-lite.c |  5 +-
 drivers/media/platform/exynos4-is/fimc-m2m.c  |  5 +-
 drivers/media/platform/exynos4-is/media-dev.c |  9 +--
 drivers/media/platform/exynos4-is/mipi-csis.c | 10 ++--
 .../media/platform/marvell-ccic/mcam-core.c   |  9 ++-
 drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c  |  6 +-
 .../platform/mtk-vcodec/mtk_vcodec_dec_drv.c  |  4 +-
 .../platform/mtk-vcodec/mtk_vcodec_dec_pm.c   |  8 ++-
 .../platform/mtk-vcodec/mtk_vcodec_dec_pm.h   |  2 +-
 drivers/media/platform/qcom/venus/core.c      | 59 +++++++++++++++----
 drivers/media/platform/rcar_fdp1.c            | 28 ++++++---
 drivers/media/platform/renesas-ceu.c          |  4 +-
 drivers/media/platform/s5p-jpeg/jpeg-core.c   |  5 +-
 drivers/media/platform/sh_vou.c               |  6 +-
 drivers/media/platform/sti/bdisp/bdisp-v4l2.c |  7 ++-
 drivers/media/platform/sti/delta/delta-v4l2.c |  8 +--
 .../sunxi/sun8i-rotate/sun8i_rotate.c         |  2 +-
 drivers/staging/media/hantro/hantro_drv.c     |  7 ++-
 drivers/staging/media/imx/imx7-mipi-csis.c    |  7 +--
 drivers/staging/media/rkvdec/rkvdec.c         |  2 +-
 32 files changed, 220 insertions(+), 127 deletions(-)

-- 
2.30.2

[PATCH 03/25] media: venus: Rework error fail recover logic

[Mauro Carvalho Chehab]

The Venus code has a sort of watchdog that attempts to recover
from IP errors, implemented as a delayed work job, which
calls venus_sys_error_handler().

Right now, it has several issues:

1. It assumes that PM runtime resume never fails

2. It internally runs two while() loops that also assume that
   PM runtime will never fail to go idle:

	while (pm_runtime_active(core->dev_dec) || pm_runtime_active(core->dev_enc))
		msleep(10);

...

	while (core->pmdomains[0] && pm_runtime_active(core->pmdomains[0]))
		usleep_range(1000, 1500);

3. It uses an OR to merge all return codes and then report to the user

4. If the hardware never recovers, it keeps running on every 10ms,
   flooding the syslog with 2 messages (so, up to 200 messages
   per second).

Rework the code, in order to prevent that, by:

1. check the return code from PM runtime resume;
2. don't let the while() loops run forever;
3. store the failed event;
4. use warn ratelimited when it fails to recover.

Fixes: af2c3834c8ca ("[media] media: venus: adding core part and helper functions")
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 drivers/media/platform/qcom/venus/core.c | 59 +++++++++++++++++++-----
 1 file changed, 47 insertions(+), 12 deletions(-)

diff --git a/drivers/media/platform/qcom/venus/core.c b/drivers/media/platform/qcom/venus/core.c
index 54bac7ec14c5..4d0482743c0a 100644
--- a/drivers/media/platform/qcom/venus/core.c
+++ b/drivers/media/platform/qcom/venus/core.c
@@ -78,22 +78,32 @@ static const struct hfi_core_ops venus_core_ops = {
 	.event_notify = venus_event_notify,
 };
 
+#define RPM_WAIT_FOR_IDLE_MAX_ATTEMPTS 10
+
 static void venus_sys_error_handler(struct work_struct *work)
 {
 	struct venus_core *core =
 			container_of(work, struct venus_core, work.work);
-	int ret = 0;
+	int ret, i, max_attempts = RPM_WAIT_FOR_IDLE_MAX_ATTEMPTS;
+	bool failed = false;
+	const char *err_msg = "";
 
-	pm_runtime_get_sync(core->dev);
+	ret = pm_runtime_get_sync(core->dev);
+	if (ret < 0) {
+		err_msg = "resume runtime PM\n";
+		max_attempts = 0;
+		failed = true;
+	}
 
 	hfi_core_deinit(core, true);
 
-	dev_warn(core->dev, "system error has occurred, starting recovery!\n");
-
 	mutex_lock(&core->lock);
 
-	while (pm_runtime_active(core->dev_dec) || pm_runtime_active(core->dev_enc))
+	for (i = 0; i < max_attempts; i++) {
+		if (!pm_runtime_active(core->dev_dec) && !pm_runtime_active(core->dev_enc))
+			break;
 		msleep(10);
+	}
 
 	venus_shutdown(core);
 
@@ -101,31 +111,56 @@ static void venus_sys_error_handler(struct work_struct *work)
 
 	pm_runtime_put_sync(core->dev);
 
-	while (core->pmdomains[0] && pm_runtime_active(core->pmdomains[0]))
+	for (i = 0; i < max_attempts; i++) {
+		if (!core->pmdomains[0] || !pm_runtime_active(core->pmdomains[0]))
+			break;
 		usleep_range(1000, 1500);
+	}
 
 	hfi_reinit(core);
 
-	pm_runtime_get_sync(core->dev);
+	ret = pm_runtime_get_sync(core->dev);
+	if (ret < 0) {
+		err_msg = "resume runtime PM\n";
+		max_attempts = 0;
+		failed = true;
+	}
 
-	ret |= venus_boot(core);
-	ret |= hfi_core_resume(core, true);
+	ret = venus_boot(core);
+	if (ret && !failed) {
+		err_msg = "boot Venus\n";
+		failed = true;
+	}
+
+	ret = hfi_core_resume(core, true);
+	if (ret && !failed) {
+		err_msg = "resume HFI\n";
+		failed = true;
+	}
 
 	enable_irq(core->irq);
 
 	mutex_unlock(&core->lock);
 
-	ret |= hfi_core_init(core);
+	ret = hfi_core_init(core);
+	if (ret && !failed) {
+		err_msg = "init HFI\n";
+		failed = true;
+	}
 
 	pm_runtime_put_sync(core->dev);
 
-	if (ret) {
+	if (failed) {
 		disable_irq_nosync(core->irq);
-		dev_warn(core->dev, "recovery failed (%d)\n", ret);
+		dev_warn_ratelimited(core->dev,
+				     "System error has occurred, recovery failed to %s\n",
+				     err_msg);
 		schedule_delayed_work(&core->work, msecs_to_jiffies(10));
 		return;
 	}
 
+	dev_warn(core->dev, "system error has occurred (recovered)\n");
+
 	mutex_lock(&core->lock);
 	core->sys_error = false;
 	mutex_unlock(&core->lock);
-- 
2.30.2

Re: [PATCH 03/25] media: venus: Rework error fail recover logic

[Jonathan Cameron]

On Wed, 5 May 2021 11:41:53 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> The Venus code has a sort of watchdog that attempts to recover
> from IP errors, implemented as a delayed work job, which
> calls venus_sys_error_handler().
> 
> Right now, it has several issues:
> 
> 1. It assumes that PM runtime resume never fails
> 
> 2. It internally runs two while() loops that also assume that
>    PM runtime will never fail to go idle:
> 
> 	while (pm_runtime_active(core->dev_dec) || pm_runtime_active(core->dev_enc))
> 		msleep(10);
> 
> ...
> 
> 	while (core->pmdomains[0] && pm_runtime_active(core->pmdomains[0]))
> 		usleep_range(1000, 1500);
> 
> 3. It uses an OR to merge all return codes and then report to the user
> 
> 4. If the hardware never recovers, it keeps running on every 10ms,
>    flooding the syslog with 2 messages (so, up to 200 messages
>    per second).
> 
> Rework the code, in order to prevent that, by:
> 
> 1. check the return code from PM runtime resume;
> 2. don't let the while() loops run forever;
> 3. store the failed event;
> 4. use warn ratelimited when it fails to recover.
> 
> Fixes: af2c3834c8ca ("[media] media: venus: adding core part and helper functions")
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

Trivial comments inline, otherwise based on no knowledge at all of the
actual hardware, the fix looks sane.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

> ---
>  drivers/media/platform/qcom/venus/core.c | 59 +++++++++++++++++++-----
>  1 file changed, 47 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/media/platform/qcom/venus/core.c b/drivers/media/platform/qcom/venus/core.c
> index 54bac7ec14c5..4d0482743c0a 100644
> --- a/drivers/media/platform/qcom/venus/core.c
> +++ b/drivers/media/platform/qcom/venus/core.c
> @@ -78,22 +78,32 @@ static const struct hfi_core_ops venus_core_ops = {
>  	.event_notify = venus_event_notify,
>  };
>  
> +#define RPM_WAIT_FOR_IDLE_MAX_ATTEMPTS 10
> +
>  static void venus_sys_error_handler(struct work_struct *work)
>  {
>  	struct venus_core *core =
>  			container_of(work, struct venus_core, work.work);
> -	int ret = 0;
> +	int ret, i, max_attempts = RPM_WAIT_FOR_IDLE_MAX_ATTEMPTS;
> +	bool failed = false;
> +	const char *err_msg = "";
>  
> -	pm_runtime_get_sync(core->dev);
> +	ret = pm_runtime_get_sync(core->dev);
> +	if (ret < 0) {
> +		err_msg = "resume runtime PM\n";

Will end up with two newlines I think as %s\n" later.

> +		max_attempts = 0;
> +		failed = true;
> +	}
>  
>  	hfi_core_deinit(core, true);
>  
> -	dev_warn(core->dev, "system error has occurred, starting recovery!\n");
> -
>  	mutex_lock(&core->lock);
>  
> -	while (pm_runtime_active(core->dev_dec) || pm_runtime_active(core->dev_enc))
> +	for (i = 0; i < max_attempts; i++) {
> +		if (!pm_runtime_active(core->dev_dec) && !pm_runtime_active(core->dev_enc))
> +			break;
>  		msleep(10);
> +	}
>  
>  	venus_shutdown(core);
>  
> @@ -101,31 +111,56 @@ static void venus_sys_error_handler(struct work_struct *work)
>  
>  	pm_runtime_put_sync(core->dev);
>  
> -	while (core->pmdomains[0] && pm_runtime_active(core->pmdomains[0]))
> +	for (i = 0; i < max_attempts; i++) {
> +		if (!core->pmdomains[0] || !pm_runtime_active(core->pmdomains[0]))
> +			break;
>  		usleep_range(1000, 1500);
> +	}
>  
>  	hfi_reinit(core);
>  
> -	pm_runtime_get_sync(core->dev);
> +	ret = pm_runtime_get_sync(core->dev);
> +	if (ret < 0) {
> +		err_msg = "resume runtime PM\n";
> +		max_attempts = 0;

This is after the last use of max_attempts, so no point in setting it to zero.

> +		failed = true;
> +	}
>  
> -	ret |= venus_boot(core);
> -	ret |= hfi_core_resume(core, true);
> +	ret = venus_boot(core);
> +	if (ret && !failed) {
> +		err_msg = "boot Venus\n";
> +		failed = true;
> +	}
> +
> +	ret = hfi_core_resume(core, true);
> +	if (ret && !failed) {
> +		err_msg = "resume HFI\n";
> +		failed = true;
> +	}
>  
>  	enable_irq(core->irq);
>  
>  	mutex_unlock(&core->lock);
>  
> -	ret |= hfi_core_init(core);
> +	ret = hfi_core_init(core);
> +	if (ret && !failed) {
> +		err_msg = "init HFI\n";
> +		failed = true;
> +	}
>  
>  	pm_runtime_put_sync(core->dev);
>  
> -	if (ret) {
> +	if (failed) {
>  		disable_irq_nosync(core->irq);
> -		dev_warn(core->dev, "recovery failed (%d)\n", ret);
> +		dev_warn_ratelimited(core->dev,
> +				     "System error has occurred, recovery failed to %s\n",
> +				     err_msg);
>  		schedule_delayed_work(&core->work, msecs_to_jiffies(10));
>  		return;
>  	}
>  
> +	dev_warn(core->dev, "system error has occurred (recovered)\n");
> +
>  	mutex_lock(&core->lock);
>  	core->sys_error = false;
>  	mutex_unlock(&core->lock);

Re: [PATCH 00/25] Fix some PM runtime issues at the media subsystem

[Mauro Carvalho Chehab]

Em Wed,  5 May 2021 11:41:50 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> As part of an effort to cleanup pm_runtime*get* calls, I detected a number
> of issues at the media subsystem.
> 
> Most of the patches here were submitted previously at:
> 
> 	https://lore.kernel.org/linux-media/cover.1619621413.git.mchehab+huawei@kernel.org/
> 
> This series contain just the bug fixes and other related issues that are
> present with the current code on media.

Series merged on my stage tree, at:
	https://git.linuxtv.org/media_stage.git/log/

I'll be merging it at media_tree (either for 5.13 or 5.14) after the
end of the merge window (likely next week).

Please let me know if you find any problems on it.

PS.: please notice that my stage tree can be rebased.

Regards,
Mauro

> 
> It address the points from the existing reviews. I also did my own
> set of reviews, in order to avoid regressions.
> 
> Changes from v4 of the previous changeset:
> 
> - reworked i2c/css RPM get logic;
> - dropped two patches that could cause regressions;
> - am437x: keep using pm_runtime_get_sync on suspend/resume;
> - atmel: fix the returned code and add a print on failures at start streaming;
> - simplify some checks for return code > 0;
> - mdk-vcodec: properly handle RPM errors at device on logic;
> - venus: rework venus_sys_error_handler() logic;
> - sti/delta: fix an issue at the error checking logic.
> 
> Mauro Carvalho Chehab (25):
>   staging: media: rkvdec: fix pm_runtime_get_sync() usage count
>   staging: media: imx7-mipi-csis: fix pm_runtime_get_sync() usage count
>   media: venus: Rework error fail recover logic
>   media: s5p_cec: decrement usage count if disabled
>   media: i2c: ccs-core: return the right error code at suspend
>   media: i2c: imx334: fix the pm runtime get logic
>   media: exynos-gsc: don't resume at remove time
>   media: atmel: properly get pm_runtime
>   media: hantro: do a PM resume earlier
>   media: marvel-ccic: fix some issues when getting pm_runtime
>   media: mdk-mdp: fix pm_runtime_get_sync() usage count
>   media: rcar_fdp1: simplify error check logic at fdp_open()
>   media: rcar_fdp1: fix pm_runtime_get_sync() usage count
>   media: renesas-ceu: Properly check for PM errors
>   media: s5p: fix pm_runtime_get_sync() usage count
>   media: am437x: fix pm_runtime_get_sync() usage count
>   media: sh_vou: fix pm_runtime_get_sync() usage count
>   media: mtk-vcodec: fix PM runtime get logic
>   media: s5p-jpeg: fix pm_runtime_get_sync() usage count
>   media: sti/delta: use pm_runtime_resume_and_get()
>   media: sunxi: fix pm_runtime_get_sync() usage count
>   media: sti/bdisp: fix pm_runtime_get_sync() usage count
>   media: exynos4-is: fix pm_runtime_get_sync() usage count
>   media: exynos-gsc: fix pm_runtime_get_sync() usage count
>   media: i2c: ccs-core: fix pm_runtime_get_sync() usage count
> 
>  drivers/media/cec/platform/s5p/s5p_cec.c      |  7 ++-
>  drivers/media/i2c/ccs/ccs-core.c              | 41 ++++++++-----
>  drivers/media/i2c/imx334.c                    |  7 ++-
>  drivers/media/platform/am437x/am437x-vpfe.c   | 15 ++++-
>  drivers/media/platform/atmel/atmel-isc-base.c | 30 +++++++---
>  drivers/media/platform/atmel/atmel-isi.c      | 19 ++++--
>  drivers/media/platform/exynos-gsc/gsc-core.c  | 11 ++--
>  drivers/media/platform/exynos-gsc/gsc-m2m.c   |  4 +-
>  .../media/platform/exynos4-is/fimc-capture.c  |  6 +-
>  drivers/media/platform/exynos4-is/fimc-is.c   |  4 +-
>  .../platform/exynos4-is/fimc-isp-video.c      |  3 +-
>  drivers/media/platform/exynos4-is/fimc-isp.c  |  7 +--
>  drivers/media/platform/exynos4-is/fimc-lite.c |  5 +-
>  drivers/media/platform/exynos4-is/fimc-m2m.c  |  5 +-
>  drivers/media/platform/exynos4-is/media-dev.c |  9 +--
>  drivers/media/platform/exynos4-is/mipi-csis.c | 10 ++--
>  .../media/platform/marvell-ccic/mcam-core.c   |  9 ++-
>  drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c  |  6 +-
>  .../platform/mtk-vcodec/mtk_vcodec_dec_drv.c  |  4 +-
>  .../platform/mtk-vcodec/mtk_vcodec_dec_pm.c   |  8 ++-
>  .../platform/mtk-vcodec/mtk_vcodec_dec_pm.h   |  2 +-
>  drivers/media/platform/qcom/venus/core.c      | 59 +++++++++++++++----
>  drivers/media/platform/rcar_fdp1.c            | 28 ++++++---
>  drivers/media/platform/renesas-ceu.c          |  4 +-
>  drivers/media/platform/s5p-jpeg/jpeg-core.c   |  5 +-
>  drivers/media/platform/sh_vou.c               |  6 +-
>  drivers/media/platform/sti/bdisp/bdisp-v4l2.c |  7 ++-
>  drivers/media/platform/sti/delta/delta-v4l2.c |  8 +--
>  .../sunxi/sun8i-rotate/sun8i_rotate.c         |  2 +-
>  drivers/staging/media/hantro/hantro_drv.c     |  7 ++-
>  drivers/staging/media/imx/imx7-mipi-csis.c    |  7 +--
>  drivers/staging/media/rkvdec/rkvdec.c         |  2 +-
>  32 files changed, 220 insertions(+), 127 deletions(-)
> 



Thanks,
Mauro