linux-rockchip.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 1/3] dt-bindings: iommu: rockchip: Add rockchip,disable-device-link-resume
@ 2023-03-30 13:17 Jagan Teki
  2023-03-30 13:17 ` [PATCH 2/3] iommu/rockchip: Disable the device link during resume Jagan Teki
                   ` (2 more replies)
  0 siblings, 3 replies; 8+ messages in thread
From: Jagan Teki @ 2023-03-30 13:17 UTC (permalink / raw)
  To: Heiko Stuebner, Joerg Roedel, Will Deacon, Robin Murphy,
	Rob Herring, Krzysztof Kozlowski
  Cc: iommu, linux-rockchip, linux-arm-kernel, devicetree,
	linux-amarula, Jagan Teki

Rockchip iommu is trying to enable the associated device at runtime
resume however some devices might enable the iommu during their pm
runtime resume operation which indeed leads iommu to use the wrong
domain and this leads to device iommu page fault.

Add rockchip,disable-device-link-resume flag and give an option for
those devices to disable the device link during Rockchip iommu pm
runtime operation.

This makes the device enablement for that iommu domain ignored during
the rk_iommu_resume call as it assumes it handled iommu device
attachment in the associated device itself.

Document rockchip,disable-device-link-resume flag.

Signed-off-by: Jagan Teki <jagan@amarulasolutions.com>
---
 .../devicetree/bindings/iommu/rockchip,iommu.yaml        | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/Documentation/devicetree/bindings/iommu/rockchip,iommu.yaml b/Documentation/devicetree/bindings/iommu/rockchip,iommu.yaml
index ba9124f721f1..ac8b03806cb3 100644
--- a/Documentation/devicetree/bindings/iommu/rockchip,iommu.yaml
+++ b/Documentation/devicetree/bindings/iommu/rockchip,iommu.yaml
@@ -58,6 +58,15 @@ properties:
       Some mmu instances may produce unexpected results
       when the reset operation is used.
 
+  rockchip,disable-device-link-resume:
+    $ref: /schemas/types.yaml#/definitions/flag
+    description: |
+      Do not link the device during runtime resume operation.
+      Some devices might enable the iommu during their pm runtime
+      resume operation, so disable the device link for those devices
+      otherwise iommu use the wrong domain and that leads to device
+      iommu page fault.
+
 required:
   - compatible
   - reg
-- 
2.25.1


_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH 2/3] iommu/rockchip: Disable the device link during resume
  2023-03-30 13:17 [PATCH 1/3] dt-bindings: iommu: rockchip: Add rockchip,disable-device-link-resume Jagan Teki
@ 2023-03-30 13:17 ` Jagan Teki
  2023-03-30 13:42   ` Robin Murphy
  2023-03-30 13:17 ` [PATCH 3/3] arm64: dts: rockchip: Disable device link for RK3328 VOP Jagan Teki
  2023-03-31  8:35 ` [PATCH 1/3] dt-bindings: iommu: rockchip: Add rockchip,disable-device-link-resume Krzysztof Kozlowski
  2 siblings, 1 reply; 8+ messages in thread
From: Jagan Teki @ 2023-03-30 13:17 UTC (permalink / raw)
  To: Heiko Stuebner, Joerg Roedel, Will Deacon, Robin Murphy,
	Rob Herring, Krzysztof Kozlowski
  Cc: iommu, linux-rockchip, linux-arm-kernel, devicetree,
	linux-amarula, Jagan Teki, Simon Xue

Rockchip iommu is trying to enable the associated device at runtime
resume however some devices might enable the iommu during their pm
runtime resume operation which indeed leads iommu to use the wrong
domain and this leads to device iommu page fault.

An example of this behavior has been observed in Rockchip RK3328, where
iommu stalls request timeout dring VOP device enablement.

Here is the dmesg log for the same:

rockchip-drm display-subsystem: bound ff370000.vop (ops vop_component_ops)
dwhdmi-rockchip ff3c0000.hdmi: supply avdd-0v9 not found, using dummy regulator
rk_iommu ff373f00.iommu: Enable stall request timed out, status: 0x00004b
dwhdmi-rockchip ff3c0000.hdmi: supply avdd-1v8 not found, using dummy regulator
rk_iommu ff373f00.iommu: Disable paging request timed out, status: 0x00004b
dwhdmi-rockchip ff3c0000.hdmi: Detected HDMI TX controller v2.11a with HDCP (inno_dw_hdmi_phy2)
dwhdmi-rockchip ff3c0000.hdmi: registered DesignWare HDMI I2C bus driver
rockchip-drm display-subsystem: bound ff3c0000.hdmi (ops dw_hdmi_rockchip_ops)
[drm] Initialized rockchip 1.0.0 20140818 for display-subsystem on minor 0

This issue is reproduced if we enable the display in U-Boot however
U-Boot is not even touched any iommu register as the U-Boot display
uses the simple frame buffer like other Rockchip platforms RK3399,
and RK3328 do.

When VOP is trying to enable the iommu using runtime resume call
pm_runtime_resume_and_get from @vop_enable then the iommu runtime
resume call @rk_iommu_resume will try to attach the VOP in the wrong
domain via @rk_iommu_enable will lead to the vop iommu page fault.

vop_enable()
   pm_runtime_resume_and_get()
      rk_iommu_resume()
         rk_iommu_enable()
            ... vop iommu page fault ...
	    rk_iommu ff373f00.iommu: Enable stall request timed out, status: 0x00004b
	    rk_iommu ff373f00.iommu: Disable paging request timed out, status: 0x00004b

So, this patch is trying to disable the device link for those devices
that are enabled rockchip,disable-device-link-resume flag assumes here
VOP device.

This makes the device enablement for that iommu domain ignored during
the rk_iommu_resume call as it assumes it handled iommu device
attachment in the associated device itself.

vop_enable()
   pm_runtime_resume_and_get()
      rk_iommu_resume()
         ... ignore the device link ...
	    rockchip_drm_dma_attach_device()
	       iommu_attach_device()

Here is the downstream patch for similar issue,
https://github.com/rockchip-linux/kernel/commit/85959f645ba38617233fbf44f442f8a88875d765

Co-developed-by: Simon Xue <xxm@rock-chips.com>
Signed-off-by: Simon Xue <xxm@rock-chips.com>
Signed-off-by: Jagan Teki <jagan@amarulasolutions.com>
---
 drivers/iommu/rockchip-iommu.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c
index f30db22ea5d7..bcff0dc21223 100644
--- a/drivers/iommu/rockchip-iommu.c
+++ b/drivers/iommu/rockchip-iommu.c
@@ -111,6 +111,7 @@ struct rk_iommu {
 	struct clk_bulk_data *clocks;
 	int num_clocks;
 	bool reset_disabled;
+	bool dlr_disable; /* avoid access iommu when runtime ops called */
 	struct iommu_device iommu;
 	struct list_head node; /* entry in rk_iommu_domain.iommus */
 	struct iommu_domain *domain; /* domain to which iommu is attached */
@@ -1250,6 +1251,8 @@ static int rk_iommu_probe(struct platform_device *pdev)
 
 	iommu->reset_disabled = device_property_read_bool(dev,
 					"rockchip,disable-mmu-reset");
+	iommu->dlr_disable = device_property_read_bool(dev,
+					"rockchip,disable-device-link-resume");
 
 	iommu->num_clocks = ARRAY_SIZE(rk_iommu_clocks);
 	iommu->clocks = devm_kcalloc(iommu->dev, iommu->num_clocks,
@@ -1346,6 +1349,9 @@ static int __maybe_unused rk_iommu_suspend(struct device *dev)
 	if (!iommu->domain)
 		return 0;
 
+	if (iommu->dlr_disable)
+		return 0;
+
 	rk_iommu_disable(iommu);
 	return 0;
 }
@@ -1357,6 +1363,9 @@ static int __maybe_unused rk_iommu_resume(struct device *dev)
 	if (!iommu->domain)
 		return 0;
 
+	if (iommu->dlr_disable)
+		return 0;
+
 	return rk_iommu_enable(iommu);
 }
 
-- 
2.25.1


_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH 3/3] arm64: dts: rockchip: Disable device link for RK3328 VOP
  2023-03-30 13:17 [PATCH 1/3] dt-bindings: iommu: rockchip: Add rockchip,disable-device-link-resume Jagan Teki
  2023-03-30 13:17 ` [PATCH 2/3] iommu/rockchip: Disable the device link during resume Jagan Teki
@ 2023-03-30 13:17 ` Jagan Teki
  2023-03-31  8:35 ` [PATCH 1/3] dt-bindings: iommu: rockchip: Add rockchip,disable-device-link-resume Krzysztof Kozlowski
  2 siblings, 0 replies; 8+ messages in thread
From: Jagan Teki @ 2023-03-30 13:17 UTC (permalink / raw)
  To: Heiko Stuebner, Joerg Roedel, Will Deacon, Robin Murphy,
	Rob Herring, Krzysztof Kozlowski
  Cc: iommu, linux-rockchip, linux-arm-kernel, devicetree,
	linux-amarula, Jagan Teki

Rockchip iommu is trying to enable the VOP device at runtime resume
however VOP device might enable the iommu during their pm runtime resume
operation which indeed leads iommu to use the wrong domain and this
leads to VOP iommu page fault.

This issue is reproduced if we enable the display in U-Boot however
U-Boot is not even touched any iommu register as the U-Boot display
uses the simple frame buffer like other Rockchip platforms RK3399,
and RK3328 do.

rockchip-drm display-subsystem: bound ff370000.vop (ops vop_component_ops)
dwhdmi-rockchip ff3c0000.hdmi: supply avdd-0v9 not found, using dummy regulator
rk_iommu ff373f00.iommu: Enable stall request timed out, status: 0x00004b
dwhdmi-rockchip ff3c0000.hdmi: supply avdd-1v8 not found, using dummy regulator
rk_iommu ff373f00.iommu: Disable paging request timed out, status: 0x00004b
dwhdmi-rockchip ff3c0000.hdmi: Detected HDMI TX controller v2.11a with HDCP (inno_dw_hdmi_phy2)
dwhdmi-rockchip ff3c0000.hdmi: registered DesignWare HDMI I2C bus driver
rockchip-drm display-subsystem: bound ff3c0000.hdmi (ops dw_hdmi_rockchip_ops)
[drm] Initialized rockchip 1.0.0 20140818 for display-subsystem on minor 0

So, prevent this by enabling rockchip,disable-device-link-resume in VOP
node so that VOP enablement for that iommu domain ignored during the
rk_iommu_resume call as it assumes it handled iommu device attachment in
the VOP itself.

Signed-off-by: Jagan Teki <jagan@amarulasolutions.com>
---
 arch/arm64/boot/dts/rockchip/rk3328.dtsi | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/boot/dts/rockchip/rk3328.dtsi b/arch/arm64/boot/dts/rockchip/rk3328.dtsi
index 6d7a7bf72ac7..7ca83bc844c1 100644
--- a/arch/arm64/boot/dts/rockchip/rk3328.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3328.dtsi
@@ -718,6 +718,7 @@ vop_mmu: iommu@ff373f00 {
 		clocks = <&cru ACLK_VOP>, <&cru HCLK_VOP>;
 		clock-names = "aclk", "iface";
 		#iommu-cells = <0>;
+		rockchip,disable-device-link-resume;
 		status = "disabled";
 	};
 
-- 
2.25.1


_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: [PATCH 2/3] iommu/rockchip: Disable the device link during resume
  2023-03-30 13:17 ` [PATCH 2/3] iommu/rockchip: Disable the device link during resume Jagan Teki
@ 2023-03-30 13:42   ` Robin Murphy
  2023-04-04  7:51     ` Jagan Teki
  0 siblings, 1 reply; 8+ messages in thread
From: Robin Murphy @ 2023-03-30 13:42 UTC (permalink / raw)
  To: Jagan Teki, Heiko Stuebner, Joerg Roedel, Will Deacon,
	Rob Herring, Krzysztof Kozlowski
  Cc: iommu, linux-rockchip, linux-arm-kernel, devicetree,
	linux-amarula, Simon Xue

On 2023-03-30 14:17, Jagan Teki wrote:
> Rockchip iommu is trying to enable the associated device at runtime
> resume however some devices might enable the iommu during their pm
> runtime resume operation which indeed leads iommu to use the wrong
> domain and this leads to device iommu page fault.
> 
> An example of this behavior has been observed in Rockchip RK3328, where
> iommu stalls request timeout dring VOP device enablement.
> 
> Here is the dmesg log for the same:
> 
> rockchip-drm display-subsystem: bound ff370000.vop (ops vop_component_ops)
> dwhdmi-rockchip ff3c0000.hdmi: supply avdd-0v9 not found, using dummy regulator
> rk_iommu ff373f00.iommu: Enable stall request timed out, status: 0x00004b
> dwhdmi-rockchip ff3c0000.hdmi: supply avdd-1v8 not found, using dummy regulator
> rk_iommu ff373f00.iommu: Disable paging request timed out, status: 0x00004b
> dwhdmi-rockchip ff3c0000.hdmi: Detected HDMI TX controller v2.11a with HDCP (inno_dw_hdmi_phy2)
> dwhdmi-rockchip ff3c0000.hdmi: registered DesignWare HDMI I2C bus driver
> rockchip-drm display-subsystem: bound ff3c0000.hdmi (ops dw_hdmi_rockchip_ops)
> [drm] Initialized rockchip 1.0.0 20140818 for display-subsystem on minor 0
> 
> This issue is reproduced if we enable the display in U-Boot however
> U-Boot is not even touched any iommu register as the U-Boot display
> uses the simple frame buffer like other Rockchip platforms RK3399,
> and RK3328 do.
> 
> When VOP is trying to enable the iommu using runtime resume call
> pm_runtime_resume_and_get from @vop_enable then the iommu runtime
> resume call @rk_iommu_resume will try to attach the VOP in the wrong
> domain via @rk_iommu_enable will lead to the vop iommu page fault.

That sounds like a driver bug. The whole point of the device link is 
supposed to be that the IOMMU gets suspended after the VOP, and resumed 
before it, so it can make sure that whatever translations the VOP was 
using are restored *before* the VOP starts trying to access them again. 
If the IOMMU driver is failing to restore the correct state on resume, 
no amount of DT abuse is the right answer.

I can understand if the IOMMU itself expects to be idle for the initial 
configuration at probe time, and gets unhappy if we try to reset it 
while (bypass) VOP traffic for the bootloader framebuffer is still going 
through, but that's an entirely different issue, and again hacking 
around with runtime PM doesn't seem like the right answer.

Thanks,
Robin.

> 
> vop_enable()
>     pm_runtime_resume_and_get()
>        rk_iommu_resume()
>           rk_iommu_enable()
>              ... vop iommu page fault ...
> 	    rk_iommu ff373f00.iommu: Enable stall request timed out, status: 0x00004b
> 	    rk_iommu ff373f00.iommu: Disable paging request timed out, status: 0x00004b
> 
> So, this patch is trying to disable the device link for those devices
> that are enabled rockchip,disable-device-link-resume flag assumes here
> VOP device.
> 
> This makes the device enablement for that iommu domain ignored during
> the rk_iommu_resume call as it assumes it handled iommu device
> attachment in the associated device itself.
> 
> vop_enable()
>     pm_runtime_resume_and_get()
>        rk_iommu_resume()
>           ... ignore the device link ...
> 	    rockchip_drm_dma_attach_device()
> 	       iommu_attach_device()
> 
> Here is the downstream patch for similar issue,
> https://github.com/rockchip-linux/kernel/commit/85959f645ba38617233fbf44f442f8a88875d765
> 
> Co-developed-by: Simon Xue <xxm@rock-chips.com>
> Signed-off-by: Simon Xue <xxm@rock-chips.com>
> Signed-off-by: Jagan Teki <jagan@amarulasolutions.com>
> ---
>   drivers/iommu/rockchip-iommu.c | 9 +++++++++
>   1 file changed, 9 insertions(+)
> 
> diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c
> index f30db22ea5d7..bcff0dc21223 100644
> --- a/drivers/iommu/rockchip-iommu.c
> +++ b/drivers/iommu/rockchip-iommu.c
> @@ -111,6 +111,7 @@ struct rk_iommu {
>   	struct clk_bulk_data *clocks;
>   	int num_clocks;
>   	bool reset_disabled;
> +	bool dlr_disable; /* avoid access iommu when runtime ops called */
>   	struct iommu_device iommu;
>   	struct list_head node; /* entry in rk_iommu_domain.iommus */
>   	struct iommu_domain *domain; /* domain to which iommu is attached */
> @@ -1250,6 +1251,8 @@ static int rk_iommu_probe(struct platform_device *pdev)
>   
>   	iommu->reset_disabled = device_property_read_bool(dev,
>   					"rockchip,disable-mmu-reset");
> +	iommu->dlr_disable = device_property_read_bool(dev,
> +					"rockchip,disable-device-link-resume");
>   
>   	iommu->num_clocks = ARRAY_SIZE(rk_iommu_clocks);
>   	iommu->clocks = devm_kcalloc(iommu->dev, iommu->num_clocks,
> @@ -1346,6 +1349,9 @@ static int __maybe_unused rk_iommu_suspend(struct device *dev)
>   	if (!iommu->domain)
>   		return 0;
>   
> +	if (iommu->dlr_disable)
> +		return 0;
> +
>   	rk_iommu_disable(iommu);
>   	return 0;
>   }
> @@ -1357,6 +1363,9 @@ static int __maybe_unused rk_iommu_resume(struct device *dev)
>   	if (!iommu->domain)
>   		return 0;
>   
> +	if (iommu->dlr_disable)
> +		return 0;
> +
>   	return rk_iommu_enable(iommu);
>   }
>   

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 1/3] dt-bindings: iommu: rockchip: Add rockchip,disable-device-link-resume
  2023-03-30 13:17 [PATCH 1/3] dt-bindings: iommu: rockchip: Add rockchip,disable-device-link-resume Jagan Teki
  2023-03-30 13:17 ` [PATCH 2/3] iommu/rockchip: Disable the device link during resume Jagan Teki
  2023-03-30 13:17 ` [PATCH 3/3] arm64: dts: rockchip: Disable device link for RK3328 VOP Jagan Teki
@ 2023-03-31  8:35 ` Krzysztof Kozlowski
  2 siblings, 0 replies; 8+ messages in thread
From: Krzysztof Kozlowski @ 2023-03-31  8:35 UTC (permalink / raw)
  To: Jagan Teki, Heiko Stuebner, Joerg Roedel, Will Deacon,
	Robin Murphy, Rob Herring, Krzysztof Kozlowski
  Cc: iommu, linux-rockchip, linux-arm-kernel, devicetree, linux-amarula

On 30/03/2023 15:17, Jagan Teki wrote:
> Rockchip iommu is trying to enable the associated device at runtime
> resume however some devices might enable the iommu during their pm
> runtime resume operation which indeed leads iommu to use the wrong
> domain and this leads to device iommu page fault.

That's not a DT property. You now described system behavior and want to
encode policy in DT.



Best regards,
Krzysztof


_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 2/3] iommu/rockchip: Disable the device link during resume
  2023-03-30 13:42   ` Robin Murphy
@ 2023-04-04  7:51     ` Jagan Teki
  2023-05-18 12:15       ` Jagan Teki
  2023-05-28 10:35       ` Jagan Teki
  0 siblings, 2 replies; 8+ messages in thread
From: Jagan Teki @ 2023-04-04  7:51 UTC (permalink / raw)
  To: Robin Murphy
  Cc: Heiko Stuebner, Joerg Roedel, Will Deacon, Rob Herring,
	Krzysztof Kozlowski, iommu, linux-rockchip, linux-arm-kernel,
	devicetree, linux-amarula, Simon Xue

On Thu, Mar 30, 2023 at 7:13 PM Robin Murphy <robin.murphy@arm.com> wrote:
>
> On 2023-03-30 14:17, Jagan Teki wrote:
> > Rockchip iommu is trying to enable the associated device at runtime
> > resume however some devices might enable the iommu during their pm
> > runtime resume operation which indeed leads iommu to use the wrong
> > domain and this leads to device iommu page fault.
> >
> > An example of this behavior has been observed in Rockchip RK3328, where
> > iommu stalls request timeout dring VOP device enablement.
> >
> > Here is the dmesg log for the same:
> >
> > rockchip-drm display-subsystem: bound ff370000.vop (ops vop_component_ops)
> > dwhdmi-rockchip ff3c0000.hdmi: supply avdd-0v9 not found, using dummy regulator
> > rk_iommu ff373f00.iommu: Enable stall request timed out, status: 0x00004b
> > dwhdmi-rockchip ff3c0000.hdmi: supply avdd-1v8 not found, using dummy regulator
> > rk_iommu ff373f00.iommu: Disable paging request timed out, status: 0x00004b
> > dwhdmi-rockchip ff3c0000.hdmi: Detected HDMI TX controller v2.11a with HDCP (inno_dw_hdmi_phy2)
> > dwhdmi-rockchip ff3c0000.hdmi: registered DesignWare HDMI I2C bus driver
> > rockchip-drm display-subsystem: bound ff3c0000.hdmi (ops dw_hdmi_rockchip_ops)
> > [drm] Initialized rockchip 1.0.0 20140818 for display-subsystem on minor 0
> >
> > This issue is reproduced if we enable the display in U-Boot however
> > U-Boot is not even touched any iommu register as the U-Boot display
> > uses the simple frame buffer like other Rockchip platforms RK3399,
> > and RK3328 do.
> >
> > When VOP is trying to enable the iommu using runtime resume call
> > pm_runtime_resume_and_get from @vop_enable then the iommu runtime
> > resume call @rk_iommu_resume will try to attach the VOP in the wrong
> > domain via @rk_iommu_enable will lead to the vop iommu page fault.
>
> That sounds like a driver bug. The whole point of the device link is

Do you mean the bug in rockchip-iommu.c or vop?

> supposed to be that the IOMMU gets suspended after the VOP, and resumed
> before it, so it can make sure that whatever translations the VOP was
> using are restored *before* the VOP starts trying to access them again.
> If the IOMMU driver is failing to restore the correct state on resume,
> no amount of DT abuse is the right answer.

Then how can we handle the co-relation b/w them as VOP already
attaching the iommu and at the same time IOMMU trying to enable VOP
device but referring to the wrong domain? Any suggestions?

>
> I can understand if the IOMMU itself expects to be idle for the initial
> configuration at probe time, and gets unhappy if we try to reset it
> while (bypass) VOP traffic for the bootloader framebuffer is still going
> through, but that's an entirely different issue, and again hacking

Does it mean accessing VOP traffic at the bootloader stage effecting
iommu even though the VOP drivers in the bootloader are not using
iommu at all?

Thanks,
Jagan.

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 2/3] iommu/rockchip: Disable the device link during resume
  2023-04-04  7:51     ` Jagan Teki
@ 2023-05-18 12:15       ` Jagan Teki
  2023-05-28 10:35       ` Jagan Teki
  1 sibling, 0 replies; 8+ messages in thread
From: Jagan Teki @ 2023-05-18 12:15 UTC (permalink / raw)
  To: Heiko Stuebner, Simon Xue, Kever Yang
  Cc: Joerg Roedel, Will Deacon, Rob Herring, Krzysztof Kozlowski,
	iommu, linux-rockchip, linux-arm-kernel, devicetree,
	linux-amarula, Robin Murphy

Hi Heiko/Kever/Simon,

On Tue, Apr 4, 2023 at 1:21 PM Jagan Teki <jagan@amarulasolutions.com> wrote:
>
> On Thu, Mar 30, 2023 at 7:13 PM Robin Murphy <robin.murphy@arm.com> wrote:
> >
> > On 2023-03-30 14:17, Jagan Teki wrote:
> > > Rockchip iommu is trying to enable the associated device at runtime
> > > resume however some devices might enable the iommu during their pm
> > > runtime resume operation which indeed leads iommu to use the wrong
> > > domain and this leads to device iommu page fault.
> > >
> > > An example of this behavior has been observed in Rockchip RK3328, where
> > > iommu stalls request timeout dring VOP device enablement.
> > >
> > > Here is the dmesg log for the same:
> > >
> > > rockchip-drm display-subsystem: bound ff370000.vop (ops vop_component_ops)
> > > dwhdmi-rockchip ff3c0000.hdmi: supply avdd-0v9 not found, using dummy regulator
> > > rk_iommu ff373f00.iommu: Enable stall request timed out, status: 0x00004b
> > > dwhdmi-rockchip ff3c0000.hdmi: supply avdd-1v8 not found, using dummy regulator
> > > rk_iommu ff373f00.iommu: Disable paging request timed out, status: 0x00004b
> > > dwhdmi-rockchip ff3c0000.hdmi: Detected HDMI TX controller v2.11a with HDCP (inno_dw_hdmi_phy2)
> > > dwhdmi-rockchip ff3c0000.hdmi: registered DesignWare HDMI I2C bus driver
> > > rockchip-drm display-subsystem: bound ff3c0000.hdmi (ops dw_hdmi_rockchip_ops)
> > > [drm] Initialized rockchip 1.0.0 20140818 for display-subsystem on minor 0
> > >
> > > This issue is reproduced if we enable the display in U-Boot however
> > > U-Boot is not even touched any iommu register as the U-Boot display
> > > uses the simple frame buffer like other Rockchip platforms RK3399,
> > > and RK3328 do.
> > >
> > > When VOP is trying to enable the iommu using runtime resume call
> > > pm_runtime_resume_and_get from @vop_enable then the iommu runtime
> > > resume call @rk_iommu_resume will try to attach the VOP in the wrong
> > > domain via @rk_iommu_enable will lead to the vop iommu page fault.
> >
> > That sounds like a driver bug. The whole point of the device link is
>
> Do you mean the bug in rockchip-iommu.c or vop?
>
> > supposed to be that the IOMMU gets suspended after the VOP, and resumed
> > before it, so it can make sure that whatever translations the VOP was
> > using are restored *before* the VOP starts trying to access them again.
> > If the IOMMU driver is failing to restore the correct state on resume,
> > no amount of DT abuse is the right answer.
>
> Then how can we handle the co-relation b/w them as VOP already
> attaching the iommu and at the same time IOMMU trying to enable VOP
> device but referring to the wrong domain? Any suggestions?
>
> >
> > I can understand if the IOMMU itself expects to be idle for the initial
> > configuration at probe time, and gets unhappy if we try to reset it
> > while (bypass) VOP traffic for the bootloader framebuffer is still going
> > through, but that's an entirely different issue, and again hacking
>
> Does it mean accessing VOP traffic at the bootloader stage effecting
> iommu even though the VOP drivers in the bootloader are not using
> iommu at all?

Any suggestions on this issue? we found similar issues even with
upcoming RK SoCs - RV1126, RK3566, RK3588.

Thanks,
Jagan.

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 2/3] iommu/rockchip: Disable the device link during resume
  2023-04-04  7:51     ` Jagan Teki
  2023-05-18 12:15       ` Jagan Teki
@ 2023-05-28 10:35       ` Jagan Teki
  1 sibling, 0 replies; 8+ messages in thread
From: Jagan Teki @ 2023-05-28 10:35 UTC (permalink / raw)
  To: Robin Murphy
  Cc: Heiko Stuebner, Joerg Roedel, Will Deacon, Rob Herring,
	Krzysztof Kozlowski, iommu, linux-rockchip, linux-arm-kernel,
	devicetree, linux-amarula, Simon Xue

Hi Robin,

On Tue, Apr 4, 2023 at 1:21 PM Jagan Teki <jagan@amarulasolutions.com> wrote:
>
> On Thu, Mar 30, 2023 at 7:13 PM Robin Murphy <robin.murphy@arm.com> wrote:
> >
> > On 2023-03-30 14:17, Jagan Teki wrote:
> > > Rockchip iommu is trying to enable the associated device at runtime
> > > resume however some devices might enable the iommu during their pm
> > > runtime resume operation which indeed leads iommu to use the wrong
> > > domain and this leads to device iommu page fault.
> > >
> > > An example of this behavior has been observed in Rockchip RK3328, where
> > > iommu stalls request timeout dring VOP device enablement.
> > >
> > > Here is the dmesg log for the same:
> > >
> > > rockchip-drm display-subsystem: bound ff370000.vop (ops vop_component_ops)
> > > dwhdmi-rockchip ff3c0000.hdmi: supply avdd-0v9 not found, using dummy regulator
> > > rk_iommu ff373f00.iommu: Enable stall request timed out, status: 0x00004b
> > > dwhdmi-rockchip ff3c0000.hdmi: supply avdd-1v8 not found, using dummy regulator
> > > rk_iommu ff373f00.iommu: Disable paging request timed out, status: 0x00004b
> > > dwhdmi-rockchip ff3c0000.hdmi: Detected HDMI TX controller v2.11a with HDCP (inno_dw_hdmi_phy2)
> > > dwhdmi-rockchip ff3c0000.hdmi: registered DesignWare HDMI I2C bus driver
> > > rockchip-drm display-subsystem: bound ff3c0000.hdmi (ops dw_hdmi_rockchip_ops)
> > > [drm] Initialized rockchip 1.0.0 20140818 for display-subsystem on minor 0
> > >
> > > This issue is reproduced if we enable the display in U-Boot however
> > > U-Boot is not even touched any iommu register as the U-Boot display
> > > uses the simple frame buffer like other Rockchip platforms RK3399,
> > > and RK3328 do.
> > >
> > > When VOP is trying to enable the iommu using runtime resume call
> > > pm_runtime_resume_and_get from @vop_enable then the iommu runtime
> > > resume call @rk_iommu_resume will try to attach the VOP in the wrong
> > > domain via @rk_iommu_enable will lead to the vop iommu page fault.
> >
> > That sounds like a driver bug. The whole point of the device link is
>
> Do you mean the bug in rockchip-iommu.c or vop?

Any further comments on this? Issue seems to reproducing on latest RK SoC's.

Thanks,
Jagan.

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2023-05-28 10:36 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-30 13:17 [PATCH 1/3] dt-bindings: iommu: rockchip: Add rockchip,disable-device-link-resume Jagan Teki
2023-03-30 13:17 ` [PATCH 2/3] iommu/rockchip: Disable the device link during resume Jagan Teki
2023-03-30 13:42   ` Robin Murphy
2023-04-04  7:51     ` Jagan Teki
2023-05-18 12:15       ` Jagan Teki
2023-05-28 10:35       ` Jagan Teki
2023-03-30 13:17 ` [PATCH 3/3] arm64: dts: rockchip: Disable device link for RK3328 VOP Jagan Teki
2023-03-31  8:35 ` [PATCH 1/3] dt-bindings: iommu: rockchip: Add rockchip,disable-device-link-resume Krzysztof Kozlowski

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).