* [PATCH v7 0/4] add zynqmp TCM bindings
@ 2023-11-17 17:42 ` Tanmay Shah
0 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-17 17:42 UTC (permalink / raw)
To: andersson, mathieu.poirier, robh+dt, krzysztof.kozlowski+dt,
conor+dt, michal.simek, ben.levinsky, tanmay.shah
Cc: linux-remoteproc, devicetree, linux-arm-kernel, linux-kernel
Tightly-Coupled Memories(TCMs) are low-latency memory that provides
predictable instruction execution and predictable data load/store
timing. Each Cortex-R5F processor contains exclusive two 64 KB memory
banks on the ATCM and BTCM ports, for a total of 128 KB of memory.
In lockstep mode, both 128KB memory is accessible to the cluster.
As per ZynqMP Ultrascale+ Technical Reference Manual UG1085, following
is address space of TCM memory. The bindings in this patch series
introduces properties to accommodate following address space with
address translation between Linux and Cortex-R5 views.
| | | |
| --- | --- | --- |
| *Mode* | *R5 View* | *Linux view* | Notes |
| *Split Mode* | *start addr*| *start addr* | |
| R5_0 ATCM (64 KB) | 0x0000_0000 | 0xFFE0_0000 | |
| R5_0 BTCM (64 KB) | 0x0002_0000 | 0xFFE2_0000 | |
| R5_1 ATCM (64 KB) | 0x0000_0000 | 0xFFE9_0000 | alias of 0xFFE1_0000 |
| R5_1 BTCM (64 KB) | 0x0002_0000 | 0xFFEB_0000 | alias of 0xFFE3_0000 |
| ___ | ___ | ___ | |
| *Lockstep Mode* | | | |
| R5_0 ATCM (128 KB) | 0x0000_0000 | 0xFFE0_0000 | |
| R5_0 BTCM (128 KB) | 0x0002_0000 | 0xFFE2_0000 | |
References:
UG1085 TCM address space:
https://docs.xilinx.com/r/en-US/ug1085-zynq-ultrascale-trm/Tightly-Coupled-Memory-Address-Map
Changes in v7:
- %s/pm_dev1/pm_dev_core0/r
- %s/pm_dev_link1/pm_dev_core0_link/r
- %s/pm_dev2/pm_dev_core1/r
- %s/pm_dev_link2/pm_dev_core1_link/r
- remove pm_domain_id check to move next patch
- add comment about how 1st entry in pm domain list is used
- fix loop when jump to fail_add_pm_domains loop
- move checking of pm_domain_id from previous patch
- fix mem_bank_data memory allocation
Changes in v6:
- Introduce new node entry for r5f cluster split mode dts and
keep it disabled by default.
- Keep remoteproc lockstep mode enabled by default to maintian
back compatibility.
- Enable split mode only for zcu102 board to demo split mode use
- Remove spurious change
- Handle errors in add_pm_domains function
- Remove redundant code to handle errors from remove_pm_domains
- Missing . at the end of the commit message
- remove redundant initialization of variables
- remove fail_tcm label and relevant code to free memory
acquired using devm_* API. As this will be freed when device free it
- add extra check to see if "reg" property is supported or not
Changes in v5:
- maintain Rob's Ack on bindings patch as no changes in bindings
- split previous patch into multiple patches
- Use pm domain framework to turn on/off TCM
- Add support of parsing TCM information from device-tree
- maintain backward compatibility with previous bindings without
TCM information available in device-tree
This patch series continues previous effort to upstream ZynqMP
TCM bindings:
Previous v4 version link:
https://lore.kernel.org/all/20230829181900.2561194-1-tanmay.shah@amd.com/
Previous v3 version link:
https://lore.kernel.org/all/1689964908-22371-1-git-send-email-radhey.shyam.pandey@amd.com/
Radhey Shyam Pandey (1):
dt-bindings: remoteproc: add Tightly Coupled Memory (TCM) bindings
Radhey Shyam Pandey (1):
dt-bindings: remoteproc: add Tightly Coupled Memory (TCM) bindings
Tanmay Shah (3):
dts: zynqmp: add properties for TCM in remoteproc
remoteproc: zynqmp: add pm domains support
remoteproc: zynqmp: parse TCM from device tree
.../remoteproc/xlnx,zynqmp-r5fss.yaml | 131 ++++++-
.../boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts | 8 +
arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 60 ++-
drivers/remoteproc/xlnx_r5_remoteproc.c | 363 ++++++++++++++++--
4 files changed, 514 insertions(+), 48 deletions(-)
base-commit: 6dc66a3096730db94a89b098da66887a390cc6af
--
2.25.1
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v7 0/4] add zynqmp TCM bindings
@ 2023-11-17 17:42 ` Tanmay Shah
0 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-17 17:42 UTC (permalink / raw)
To: andersson, mathieu.poirier, robh+dt, krzysztof.kozlowski+dt,
conor+dt, michal.simek, ben.levinsky, tanmay.shah
Cc: linux-remoteproc, devicetree, linux-arm-kernel, linux-kernel
Tightly-Coupled Memories(TCMs) are low-latency memory that provides
predictable instruction execution and predictable data load/store
timing. Each Cortex-R5F processor contains exclusive two 64 KB memory
banks on the ATCM and BTCM ports, for a total of 128 KB of memory.
In lockstep mode, both 128KB memory is accessible to the cluster.
As per ZynqMP Ultrascale+ Technical Reference Manual UG1085, following
is address space of TCM memory. The bindings in this patch series
introduces properties to accommodate following address space with
address translation between Linux and Cortex-R5 views.
| | | |
| --- | --- | --- |
| *Mode* | *R5 View* | *Linux view* | Notes |
| *Split Mode* | *start addr*| *start addr* | |
| R5_0 ATCM (64 KB) | 0x0000_0000 | 0xFFE0_0000 | |
| R5_0 BTCM (64 KB) | 0x0002_0000 | 0xFFE2_0000 | |
| R5_1 ATCM (64 KB) | 0x0000_0000 | 0xFFE9_0000 | alias of 0xFFE1_0000 |
| R5_1 BTCM (64 KB) | 0x0002_0000 | 0xFFEB_0000 | alias of 0xFFE3_0000 |
| ___ | ___ | ___ | |
| *Lockstep Mode* | | | |
| R5_0 ATCM (128 KB) | 0x0000_0000 | 0xFFE0_0000 | |
| R5_0 BTCM (128 KB) | 0x0002_0000 | 0xFFE2_0000 | |
References:
UG1085 TCM address space:
https://docs.xilinx.com/r/en-US/ug1085-zynq-ultrascale-trm/Tightly-Coupled-Memory-Address-Map
Changes in v7:
- %s/pm_dev1/pm_dev_core0/r
- %s/pm_dev_link1/pm_dev_core0_link/r
- %s/pm_dev2/pm_dev_core1/r
- %s/pm_dev_link2/pm_dev_core1_link/r
- remove pm_domain_id check to move next patch
- add comment about how 1st entry in pm domain list is used
- fix loop when jump to fail_add_pm_domains loop
- move checking of pm_domain_id from previous patch
- fix mem_bank_data memory allocation
Changes in v6:
- Introduce new node entry for r5f cluster split mode dts and
keep it disabled by default.
- Keep remoteproc lockstep mode enabled by default to maintian
back compatibility.
- Enable split mode only for zcu102 board to demo split mode use
- Remove spurious change
- Handle errors in add_pm_domains function
- Remove redundant code to handle errors from remove_pm_domains
- Missing . at the end of the commit message
- remove redundant initialization of variables
- remove fail_tcm label and relevant code to free memory
acquired using devm_* API. As this will be freed when device free it
- add extra check to see if "reg" property is supported or not
Changes in v5:
- maintain Rob's Ack on bindings patch as no changes in bindings
- split previous patch into multiple patches
- Use pm domain framework to turn on/off TCM
- Add support of parsing TCM information from device-tree
- maintain backward compatibility with previous bindings without
TCM information available in device-tree
This patch series continues previous effort to upstream ZynqMP
TCM bindings:
Previous v4 version link:
https://lore.kernel.org/all/20230829181900.2561194-1-tanmay.shah@amd.com/
Previous v3 version link:
https://lore.kernel.org/all/1689964908-22371-1-git-send-email-radhey.shyam.pandey@amd.com/
Radhey Shyam Pandey (1):
dt-bindings: remoteproc: add Tightly Coupled Memory (TCM) bindings
Radhey Shyam Pandey (1):
dt-bindings: remoteproc: add Tightly Coupled Memory (TCM) bindings
Tanmay Shah (3):
dts: zynqmp: add properties for TCM in remoteproc
remoteproc: zynqmp: add pm domains support
remoteproc: zynqmp: parse TCM from device tree
.../remoteproc/xlnx,zynqmp-r5fss.yaml | 131 ++++++-
.../boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts | 8 +
arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 60 ++-
drivers/remoteproc/xlnx_r5_remoteproc.c | 363 ++++++++++++++++--
4 files changed, 514 insertions(+), 48 deletions(-)
base-commit: 6dc66a3096730db94a89b098da66887a390cc6af
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v7 1/4] dt-bindings: remoteproc: add Tightly Coupled Memory (TCM) bindings
2023-11-17 17:42 ` Tanmay Shah
@ 2023-11-17 17:42 ` Tanmay Shah
-1 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-17 17:42 UTC (permalink / raw)
To: andersson, mathieu.poirier, robh+dt, krzysztof.kozlowski+dt,
conor+dt, michal.simek, ben.levinsky, tanmay.shah
Cc: linux-remoteproc, devicetree, linux-arm-kernel, linux-kernel,
Radhey Shyam Pandey, Rob Herring
From: Radhey Shyam Pandey <radhey.shyam.pandey@amd.com>
Introduce bindings for TCM memory address space on AMD-xilinx Zynq
UltraScale+ platform. It will help in defining TCM in device-tree
and make it's access platform agnostic and data-driven.
Tightly-coupled memories(TCMs) are low-latency memory that provides
predictable instruction execution and predictable data load/store
timing. Each Cortex-R5F processor contains two 64-bit wide 64 KB memory
banks on the ATCM and BTCM ports, for a total of 128 KB of memory.
The TCM resources(reg, reg-names and power-domain) are documented for
each TCM in the R5 node. The reg and reg-names are made as required
properties as we don't want to hardcode TCM addresses for future
platforms and for zu+ legacy implementation will ensure that the
old dts w/o reg/reg-names works and stable ABI is maintained.
It also extends the examples for TCM split and lockstep modes.
Signed-off-by: Radhey Shyam Pandey <radhey.shyam.pandey@amd.com>
Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
Acked-by: Rob Herring <robh@kernel.org>
---
.../remoteproc/xlnx,zynqmp-r5fss.yaml | 131 +++++++++++++++---
1 file changed, 113 insertions(+), 18 deletions(-)
diff --git a/Documentation/devicetree/bindings/remoteproc/xlnx,zynqmp-r5fss.yaml b/Documentation/devicetree/bindings/remoteproc/xlnx,zynqmp-r5fss.yaml
index 78aac69f1060..9ecd63ea1b38 100644
--- a/Documentation/devicetree/bindings/remoteproc/xlnx,zynqmp-r5fss.yaml
+++ b/Documentation/devicetree/bindings/remoteproc/xlnx,zynqmp-r5fss.yaml
@@ -20,6 +20,17 @@ properties:
compatible:
const: xlnx,zynqmp-r5fss
+ "#address-cells":
+ const: 2
+
+ "#size-cells":
+ const: 2
+
+ ranges:
+ description: |
+ Standard ranges definition providing address translations for
+ local R5F TCM address spaces to bus addresses.
+
xlnx,cluster-mode:
$ref: /schemas/types.yaml#/definitions/uint32
enum: [0, 1, 2]
@@ -37,7 +48,7 @@ properties:
2: single cpu mode
patternProperties:
- "^r5f-[a-f0-9]+$":
+ "^r5f@[0-9a-f]+$":
type: object
description: |
The RPU is located in the Low Power Domain of the Processor Subsystem.
@@ -54,8 +65,19 @@ patternProperties:
compatible:
const: xlnx,zynqmp-r5f
+ reg:
+ items:
+ - description: ATCM internal memory region
+ - description: BTCM internal memory region
+
+ reg-names:
+ items:
+ - const: atcm
+ - const: btcm
+
power-domains:
- maxItems: 1
+ minItems: 1
+ maxItems: 3
mboxes:
minItems: 1
@@ -102,34 +124,107 @@ patternProperties:
required:
- compatible
- power-domains
+ - reg
+ - reg-names
unevaluatedProperties: false
required:
- compatible
+ - "#address-cells"
+ - "#size-cells"
+ - ranges
additionalProperties: false
examples:
- |
- remoteproc {
- compatible = "xlnx,zynqmp-r5fss";
- xlnx,cluster-mode = <1>;
-
- r5f-0 {
- compatible = "xlnx,zynqmp-r5f";
- power-domains = <&zynqmp_firmware 0x7>;
- memory-region = <&rproc_0_fw_image>, <&rpu0vdev0buffer>, <&rpu0vdev0vring0>, <&rpu0vdev0vring1>;
- mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>;
- mbox-names = "tx", "rx";
+ #include <dt-bindings/power/xlnx-zynqmp-power.h>
+
+ //Split mode configuration
+ soc {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ remoteproc@ffe00000 {
+ compatible = "xlnx,zynqmp-r5fss";
+ xlnx,cluster-mode = <0>;
+
+ #address-cells = <2>;
+ #size-cells = <2>;
+ ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x10000>,
+ <0x0 0x20000 0x0 0xffe20000 0x0 0x10000>,
+ <0x1 0x0 0x0 0xffe90000 0x0 0x10000>,
+ <0x1 0x20000 0x0 0xffeb0000 0x0 0x10000>;
+
+ r5f@0 {
+ compatible = "xlnx,zynqmp-r5f";
+ reg = <0x0 0x0 0x0 0x10000>, <0x0 0x20000 0x0 0x10000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_0>,
+ <&zynqmp_firmware PD_R5_0_ATCM>,
+ <&zynqmp_firmware PD_R5_0_BTCM>;
+ memory-region = <&rproc_0_fw_image>, <&rpu0vdev0buffer>,
+ <&rpu0vdev0vring0>, <&rpu0vdev0vring1>;
+ mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>;
+ mbox-names = "tx", "rx";
+ };
+
+ r5f@1 {
+ compatible = "xlnx,zynqmp-r5f";
+ reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_1>,
+ <&zynqmp_firmware PD_R5_1_ATCM>,
+ <&zynqmp_firmware PD_R5_1_BTCM>;
+ memory-region = <&rproc_1_fw_image>, <&rpu1vdev0buffer>,
+ <&rpu1vdev0vring0>, <&rpu1vdev0vring1>;
+ mboxes = <&ipi_mailbox_rpu1 0>, <&ipi_mailbox_rpu1 1>;
+ mbox-names = "tx", "rx";
+ };
};
+ };
- r5f-1 {
- compatible = "xlnx,zynqmp-r5f";
- power-domains = <&zynqmp_firmware 0x8>;
- memory-region = <&rproc_1_fw_image>, <&rpu1vdev0buffer>, <&rpu1vdev0vring0>, <&rpu1vdev0vring1>;
- mboxes = <&ipi_mailbox_rpu1 0>, <&ipi_mailbox_rpu1 1>;
- mbox-names = "tx", "rx";
+ - |
+ //Lockstep configuration
+ soc {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ remoteproc@ffe00000 {
+ compatible = "xlnx,zynqmp-r5fss";
+ xlnx,cluster-mode = <1>;
+
+ #address-cells = <2>;
+ #size-cells = <2>;
+ ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x20000>,
+ <0x0 0x20000 0x0 0xffe20000 0x0 0x20000>;
+
+ r5f@0 {
+ compatible = "xlnx,zynqmp-r5f";
+ reg = <0x0 0x0 0x0 0x20000>, <0x0 0x20000 0x0 0x20000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_0>,
+ <&zynqmp_firmware PD_R5_0_ATCM>,
+ <&zynqmp_firmware PD_R5_0_BTCM>;
+ memory-region = <&rproc_0_fw_image>, <&rpu0vdev0buffer>,
+ <&rpu0vdev0vring0>, <&rpu0vdev0vring1>;
+ mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>;
+ mbox-names = "tx", "rx";
+ };
+
+ r5f@1 {
+ compatible = "xlnx,zynqmp-r5f";
+ reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_1>,
+ <&zynqmp_firmware PD_R5_1_ATCM>,
+ <&zynqmp_firmware PD_R5_1_BTCM>;
+ memory-region = <&rproc_1_fw_image>, <&rpu1vdev0buffer>,
+ <&rpu1vdev0vring0>, <&rpu1vdev0vring1>;
+ mboxes = <&ipi_mailbox_rpu1 0>, <&ipi_mailbox_rpu1 1>;
+ mbox-names = "tx", "rx";
+ };
};
};
...
--
2.25.1
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 1/4] dt-bindings: remoteproc: add Tightly Coupled Memory (TCM) bindings
@ 2023-11-17 17:42 ` Tanmay Shah
0 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-17 17:42 UTC (permalink / raw)
To: andersson, mathieu.poirier, robh+dt, krzysztof.kozlowski+dt,
conor+dt, michal.simek, ben.levinsky, tanmay.shah
Cc: linux-remoteproc, devicetree, linux-arm-kernel, linux-kernel,
Radhey Shyam Pandey, Rob Herring
From: Radhey Shyam Pandey <radhey.shyam.pandey@amd.com>
Introduce bindings for TCM memory address space on AMD-xilinx Zynq
UltraScale+ platform. It will help in defining TCM in device-tree
and make it's access platform agnostic and data-driven.
Tightly-coupled memories(TCMs) are low-latency memory that provides
predictable instruction execution and predictable data load/store
timing. Each Cortex-R5F processor contains two 64-bit wide 64 KB memory
banks on the ATCM and BTCM ports, for a total of 128 KB of memory.
The TCM resources(reg, reg-names and power-domain) are documented for
each TCM in the R5 node. The reg and reg-names are made as required
properties as we don't want to hardcode TCM addresses for future
platforms and for zu+ legacy implementation will ensure that the
old dts w/o reg/reg-names works and stable ABI is maintained.
It also extends the examples for TCM split and lockstep modes.
Signed-off-by: Radhey Shyam Pandey <radhey.shyam.pandey@amd.com>
Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
Acked-by: Rob Herring <robh@kernel.org>
---
.../remoteproc/xlnx,zynqmp-r5fss.yaml | 131 +++++++++++++++---
1 file changed, 113 insertions(+), 18 deletions(-)
diff --git a/Documentation/devicetree/bindings/remoteproc/xlnx,zynqmp-r5fss.yaml b/Documentation/devicetree/bindings/remoteproc/xlnx,zynqmp-r5fss.yaml
index 78aac69f1060..9ecd63ea1b38 100644
--- a/Documentation/devicetree/bindings/remoteproc/xlnx,zynqmp-r5fss.yaml
+++ b/Documentation/devicetree/bindings/remoteproc/xlnx,zynqmp-r5fss.yaml
@@ -20,6 +20,17 @@ properties:
compatible:
const: xlnx,zynqmp-r5fss
+ "#address-cells":
+ const: 2
+
+ "#size-cells":
+ const: 2
+
+ ranges:
+ description: |
+ Standard ranges definition providing address translations for
+ local R5F TCM address spaces to bus addresses.
+
xlnx,cluster-mode:
$ref: /schemas/types.yaml#/definitions/uint32
enum: [0, 1, 2]
@@ -37,7 +48,7 @@ properties:
2: single cpu mode
patternProperties:
- "^r5f-[a-f0-9]+$":
+ "^r5f@[0-9a-f]+$":
type: object
description: |
The RPU is located in the Low Power Domain of the Processor Subsystem.
@@ -54,8 +65,19 @@ patternProperties:
compatible:
const: xlnx,zynqmp-r5f
+ reg:
+ items:
+ - description: ATCM internal memory region
+ - description: BTCM internal memory region
+
+ reg-names:
+ items:
+ - const: atcm
+ - const: btcm
+
power-domains:
- maxItems: 1
+ minItems: 1
+ maxItems: 3
mboxes:
minItems: 1
@@ -102,34 +124,107 @@ patternProperties:
required:
- compatible
- power-domains
+ - reg
+ - reg-names
unevaluatedProperties: false
required:
- compatible
+ - "#address-cells"
+ - "#size-cells"
+ - ranges
additionalProperties: false
examples:
- |
- remoteproc {
- compatible = "xlnx,zynqmp-r5fss";
- xlnx,cluster-mode = <1>;
-
- r5f-0 {
- compatible = "xlnx,zynqmp-r5f";
- power-domains = <&zynqmp_firmware 0x7>;
- memory-region = <&rproc_0_fw_image>, <&rpu0vdev0buffer>, <&rpu0vdev0vring0>, <&rpu0vdev0vring1>;
- mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>;
- mbox-names = "tx", "rx";
+ #include <dt-bindings/power/xlnx-zynqmp-power.h>
+
+ //Split mode configuration
+ soc {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ remoteproc@ffe00000 {
+ compatible = "xlnx,zynqmp-r5fss";
+ xlnx,cluster-mode = <0>;
+
+ #address-cells = <2>;
+ #size-cells = <2>;
+ ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x10000>,
+ <0x0 0x20000 0x0 0xffe20000 0x0 0x10000>,
+ <0x1 0x0 0x0 0xffe90000 0x0 0x10000>,
+ <0x1 0x20000 0x0 0xffeb0000 0x0 0x10000>;
+
+ r5f@0 {
+ compatible = "xlnx,zynqmp-r5f";
+ reg = <0x0 0x0 0x0 0x10000>, <0x0 0x20000 0x0 0x10000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_0>,
+ <&zynqmp_firmware PD_R5_0_ATCM>,
+ <&zynqmp_firmware PD_R5_0_BTCM>;
+ memory-region = <&rproc_0_fw_image>, <&rpu0vdev0buffer>,
+ <&rpu0vdev0vring0>, <&rpu0vdev0vring1>;
+ mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>;
+ mbox-names = "tx", "rx";
+ };
+
+ r5f@1 {
+ compatible = "xlnx,zynqmp-r5f";
+ reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_1>,
+ <&zynqmp_firmware PD_R5_1_ATCM>,
+ <&zynqmp_firmware PD_R5_1_BTCM>;
+ memory-region = <&rproc_1_fw_image>, <&rpu1vdev0buffer>,
+ <&rpu1vdev0vring0>, <&rpu1vdev0vring1>;
+ mboxes = <&ipi_mailbox_rpu1 0>, <&ipi_mailbox_rpu1 1>;
+ mbox-names = "tx", "rx";
+ };
};
+ };
- r5f-1 {
- compatible = "xlnx,zynqmp-r5f";
- power-domains = <&zynqmp_firmware 0x8>;
- memory-region = <&rproc_1_fw_image>, <&rpu1vdev0buffer>, <&rpu1vdev0vring0>, <&rpu1vdev0vring1>;
- mboxes = <&ipi_mailbox_rpu1 0>, <&ipi_mailbox_rpu1 1>;
- mbox-names = "tx", "rx";
+ - |
+ //Lockstep configuration
+ soc {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ remoteproc@ffe00000 {
+ compatible = "xlnx,zynqmp-r5fss";
+ xlnx,cluster-mode = <1>;
+
+ #address-cells = <2>;
+ #size-cells = <2>;
+ ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x20000>,
+ <0x0 0x20000 0x0 0xffe20000 0x0 0x20000>;
+
+ r5f@0 {
+ compatible = "xlnx,zynqmp-r5f";
+ reg = <0x0 0x0 0x0 0x20000>, <0x0 0x20000 0x0 0x20000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_0>,
+ <&zynqmp_firmware PD_R5_0_ATCM>,
+ <&zynqmp_firmware PD_R5_0_BTCM>;
+ memory-region = <&rproc_0_fw_image>, <&rpu0vdev0buffer>,
+ <&rpu0vdev0vring0>, <&rpu0vdev0vring1>;
+ mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>;
+ mbox-names = "tx", "rx";
+ };
+
+ r5f@1 {
+ compatible = "xlnx,zynqmp-r5f";
+ reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_1>,
+ <&zynqmp_firmware PD_R5_1_ATCM>,
+ <&zynqmp_firmware PD_R5_1_BTCM>;
+ memory-region = <&rproc_1_fw_image>, <&rpu1vdev0buffer>,
+ <&rpu1vdev0vring0>, <&rpu1vdev0vring1>;
+ mboxes = <&ipi_mailbox_rpu1 0>, <&ipi_mailbox_rpu1 1>;
+ mbox-names = "tx", "rx";
+ };
};
};
...
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 2/4] dts: zynqmp: add properties for TCM in remoteproc
2023-11-17 17:42 ` Tanmay Shah
@ 2023-11-17 17:42 ` Tanmay Shah
-1 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-17 17:42 UTC (permalink / raw)
To: andersson, mathieu.poirier, robh+dt, krzysztof.kozlowski+dt,
conor+dt, michal.simek, ben.levinsky, tanmay.shah
Cc: linux-remoteproc, devicetree, linux-arm-kernel, linux-kernel
Add properties as per new bindings in zynqmp remoteproc node
to represent TCM address and size.
This patch also adds alternative remoteproc node to represent
remoteproc cluster in split mode. By default lockstep mode is
enabled and users should disable it before using split mode
dts. Both device-tree nodes can't be used simultaneously one
of them must be disabled. For zcu102-1.0 and zcu102-1.1 board
remoteproc split mode dts node is enabled and lockstep mode
dts is disabled.
Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
---
.../boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts | 8 +++
arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 60 +++++++++++++++++--
2 files changed, 63 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts b/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts
index c8f71a1aec89..495ca94b45db 100644
--- a/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts
+++ b/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts
@@ -14,6 +14,14 @@ / {
compatible = "xlnx,zynqmp-zcu102-rev1.0", "xlnx,zynqmp-zcu102", "xlnx,zynqmp";
};
+&rproc_split {
+ status = "okay";
+};
+
+&rproc_lockstep {
+ status = "disabled";
+};
+
&eeprom {
#address-cells = <1>;
#size-cells = <1>;
diff --git a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
index b61fc99cd911..602e6aba7ac5 100644
--- a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
+++ b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
@@ -247,19 +247,69 @@ fpga_full: fpga-full {
ranges;
};
- remoteproc {
+ rproc_lockstep: remoteproc@ffe00000 {
compatible = "xlnx,zynqmp-r5fss";
xlnx,cluster-mode = <1>;
- r5f-0 {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x20000>,
+ <0x0 0x20000 0x0 0xffe20000 0x0 0x20000>,
+ <0x1 0x0 0x0 0xffe90000 0x0 0x10000>,
+ <0x1 0x20000 0x0 0xffeb0000 0x0 0x10000>;
+
+ r5f@0 {
+ compatible = "xlnx,zynqmp-r5f";
+ reg = <0x0 0x0 0x0 0x20000>, <0x0 0x20000 0x0 0x20000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_0>,
+ <&zynqmp_firmware PD_R5_0_ATCM>,
+ <&zynqmp_firmware PD_R5_0_BTCM>;
+ memory-region = <&rproc_0_fw_image>;
+ };
+
+ r5f@1 {
+ compatible = "xlnx,zynqmp-r5f";
+ reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_1>,
+ <&zynqmp_firmware PD_R5_1_ATCM>,
+ <&zynqmp_firmware PD_R5_1_BTCM>;
+ memory-region = <&rproc_1_fw_image>;
+ };
+ };
+
+ rproc_split: remoteproc-split@ffe00000 {
+ status = "disabled";
+ compatible = "xlnx,zynqmp-r5fss";
+ xlnx,cluster-mode = <0>;
+
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x10000>,
+ <0x0 0x20000 0x0 0xffe20000 0x0 0x10000>,
+ <0x1 0x0 0x0 0xffe90000 0x0 0x10000>,
+ <0x1 0x20000 0x0 0xffeb0000 0x0 0x10000>;
+
+ r5f@0 {
compatible = "xlnx,zynqmp-r5f";
- power-domains = <&zynqmp_firmware PD_RPU_0>;
+ reg = <0x0 0x0 0x0 0x10000>, <0x0 0x20000 0x0 0x10000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_0>,
+ <&zynqmp_firmware PD_R5_0_ATCM>,
+ <&zynqmp_firmware PD_R5_0_BTCM>;
memory-region = <&rproc_0_fw_image>;
};
- r5f-1 {
+ r5f@1 {
compatible = "xlnx,zynqmp-r5f";
- power-domains = <&zynqmp_firmware PD_RPU_1>;
+ reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_1>,
+ <&zynqmp_firmware PD_R5_1_ATCM>,
+ <&zynqmp_firmware PD_R5_1_BTCM>;
memory-region = <&rproc_1_fw_image>;
};
};
--
2.25.1
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 2/4] dts: zynqmp: add properties for TCM in remoteproc
@ 2023-11-17 17:42 ` Tanmay Shah
0 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-17 17:42 UTC (permalink / raw)
To: andersson, mathieu.poirier, robh+dt, krzysztof.kozlowski+dt,
conor+dt, michal.simek, ben.levinsky, tanmay.shah
Cc: linux-remoteproc, devicetree, linux-arm-kernel, linux-kernel
Add properties as per new bindings in zynqmp remoteproc node
to represent TCM address and size.
This patch also adds alternative remoteproc node to represent
remoteproc cluster in split mode. By default lockstep mode is
enabled and users should disable it before using split mode
dts. Both device-tree nodes can't be used simultaneously one
of them must be disabled. For zcu102-1.0 and zcu102-1.1 board
remoteproc split mode dts node is enabled and lockstep mode
dts is disabled.
Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
---
.../boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts | 8 +++
arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 60 +++++++++++++++++--
2 files changed, 63 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts b/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts
index c8f71a1aec89..495ca94b45db 100644
--- a/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts
+++ b/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts
@@ -14,6 +14,14 @@ / {
compatible = "xlnx,zynqmp-zcu102-rev1.0", "xlnx,zynqmp-zcu102", "xlnx,zynqmp";
};
+&rproc_split {
+ status = "okay";
+};
+
+&rproc_lockstep {
+ status = "disabled";
+};
+
&eeprom {
#address-cells = <1>;
#size-cells = <1>;
diff --git a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
index b61fc99cd911..602e6aba7ac5 100644
--- a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
+++ b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
@@ -247,19 +247,69 @@ fpga_full: fpga-full {
ranges;
};
- remoteproc {
+ rproc_lockstep: remoteproc@ffe00000 {
compatible = "xlnx,zynqmp-r5fss";
xlnx,cluster-mode = <1>;
- r5f-0 {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x20000>,
+ <0x0 0x20000 0x0 0xffe20000 0x0 0x20000>,
+ <0x1 0x0 0x0 0xffe90000 0x0 0x10000>,
+ <0x1 0x20000 0x0 0xffeb0000 0x0 0x10000>;
+
+ r5f@0 {
+ compatible = "xlnx,zynqmp-r5f";
+ reg = <0x0 0x0 0x0 0x20000>, <0x0 0x20000 0x0 0x20000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_0>,
+ <&zynqmp_firmware PD_R5_0_ATCM>,
+ <&zynqmp_firmware PD_R5_0_BTCM>;
+ memory-region = <&rproc_0_fw_image>;
+ };
+
+ r5f@1 {
+ compatible = "xlnx,zynqmp-r5f";
+ reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_1>,
+ <&zynqmp_firmware PD_R5_1_ATCM>,
+ <&zynqmp_firmware PD_R5_1_BTCM>;
+ memory-region = <&rproc_1_fw_image>;
+ };
+ };
+
+ rproc_split: remoteproc-split@ffe00000 {
+ status = "disabled";
+ compatible = "xlnx,zynqmp-r5fss";
+ xlnx,cluster-mode = <0>;
+
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x10000>,
+ <0x0 0x20000 0x0 0xffe20000 0x0 0x10000>,
+ <0x1 0x0 0x0 0xffe90000 0x0 0x10000>,
+ <0x1 0x20000 0x0 0xffeb0000 0x0 0x10000>;
+
+ r5f@0 {
compatible = "xlnx,zynqmp-r5f";
- power-domains = <&zynqmp_firmware PD_RPU_0>;
+ reg = <0x0 0x0 0x0 0x10000>, <0x0 0x20000 0x0 0x10000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_0>,
+ <&zynqmp_firmware PD_R5_0_ATCM>,
+ <&zynqmp_firmware PD_R5_0_BTCM>;
memory-region = <&rproc_0_fw_image>;
};
- r5f-1 {
+ r5f@1 {
compatible = "xlnx,zynqmp-r5f";
- power-domains = <&zynqmp_firmware PD_RPU_1>;
+ reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
+ reg-names = "atcm", "btcm";
+ power-domains = <&zynqmp_firmware PD_RPU_1>,
+ <&zynqmp_firmware PD_R5_1_ATCM>,
+ <&zynqmp_firmware PD_R5_1_BTCM>;
memory-region = <&rproc_1_fw_image>;
};
};
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-11-17 17:42 ` Tanmay Shah
@ 2023-11-17 17:42 ` Tanmay Shah
-1 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-17 17:42 UTC (permalink / raw)
To: andersson, mathieu.poirier, robh+dt, krzysztof.kozlowski+dt,
conor+dt, michal.simek, ben.levinsky, tanmay.shah
Cc: linux-remoteproc, devicetree, linux-arm-kernel, linux-kernel
Use TCM pm domains extracted from device-tree
to power on/off TCM using general pm domain framework.
Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
---
Changes in v7:
- %s/pm_dev1/pm_dev_core0/r
- %s/pm_dev_link1/pm_dev_core0_link/r
- %s/pm_dev2/pm_dev_core1/r
- %s/pm_dev_link2/pm_dev_core1_link/r
- remove pm_domain_id check to move next patch
- add comment about how 1st entry in pm domain list is used
- fix loop when jump to fail_add_pm_domains loop
drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
1 file changed, 212 insertions(+), 3 deletions(-)
diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
index 4395edea9a64..22bccc5075a0 100644
--- a/drivers/remoteproc/xlnx_r5_remoteproc.c
+++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
@@ -16,6 +16,7 @@
#include <linux/of_reserved_mem.h>
#include <linux/platform_device.h>
#include <linux/remoteproc.h>
+#include <linux/pm_domain.h>
#include "remoteproc_internal.h"
@@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
* @rproc: rproc handle
* @pm_domain_id: RPU CPU power domain id
* @ipi: pointer to mailbox information
+ * @num_pm_dev: number of tcm pm domain devices for this core
+ * @pm_dev_core0: pm domain virtual devices for power domain framework
+ * @pm_dev_core0_link: pm domain device links after registration
+ * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
+ * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
+ * registration
*/
struct zynqmp_r5_core {
struct device *dev;
@@ -111,6 +118,11 @@ struct zynqmp_r5_core {
struct rproc *rproc;
u32 pm_domain_id;
struct mbox_info *ipi;
+ int num_pm_dev;
+ struct device **pm_dev_core0;
+ struct device_link **pm_dev_core0_link;
+ struct device **pm_dev_core1;
+ struct device_link **pm_dev_core1_link;
};
/**
@@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
ZYNQMP_PM_CAPABILITY_ACCESS, 0,
ZYNQMP_PM_REQUEST_ACK_BLOCKING);
if (ret < 0) {
- dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
+ dev_err(dev, "failed to turn on TCM 0x%x",
+ pm_domain_id);
goto release_tcm_lockstep;
}
@@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
return ret;
}
+static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
+{
+ struct zynqmp_r5_core *r5_core = rproc->priv;
+ struct device *dev = r5_core->dev;
+ struct zynqmp_r5_cluster *cluster;
+ int i;
+
+ cluster = platform_get_drvdata(to_platform_device(dev->parent));
+
+ for (i = 1; i < r5_core->num_pm_dev; i++) {
+ device_link_del(r5_core->pm_dev_core0_link[i]);
+ dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
+ }
+
+ kfree(r5_core->pm_dev_core0);
+ r5_core->pm_dev_core0 = NULL;
+ kfree(r5_core->pm_dev_core0_link);
+ r5_core->pm_dev_core0_link = NULL;
+
+ if (cluster->mode == SPLIT_MODE) {
+ r5_core->num_pm_dev = 0;
+ return;
+ }
+
+ for (i = 1; i < r5_core->num_pm_dev; i++) {
+ device_link_del(r5_core->pm_dev_core1_link[i]);
+ dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
+ }
+
+ kfree(r5_core->pm_dev_core1);
+ r5_core->pm_dev_core1 = NULL;
+ kfree(r5_core->pm_dev_core1_link);
+ r5_core->pm_dev_core1_link = NULL;
+ r5_core->num_pm_dev = 0;
+}
+
+static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
+{
+ struct zynqmp_r5_core *r5_core = rproc->priv;
+ struct device *dev = r5_core->dev, *dev2;
+ struct zynqmp_r5_cluster *cluster;
+ struct platform_device *pdev;
+ struct device_node *np;
+ int i, j, num_pm_dev, ret;
+
+ cluster = dev_get_drvdata(dev->parent);
+
+ /* get number of power-domains */
+ num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
+ "#power-domain-cells");
+
+ if (num_pm_dev <= 0)
+ return -EINVAL;
+
+ r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
+ sizeof(struct device *),
+ GFP_KERNEL);
+ if (!r5_core->pm_dev_core0)
+ ret = -ENOMEM;
+
+ r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
+ sizeof(struct device_link *),
+ GFP_KERNEL);
+ if (!r5_core->pm_dev_core0_link) {
+ kfree(r5_core->pm_dev_core0);
+ r5_core->pm_dev_core0 = NULL;
+ return -ENOMEM;
+ }
+
+ r5_core->num_pm_dev = num_pm_dev;
+
+ /*
+ * start from 2nd entry in power-domains property list as
+ * for zynqmp we only add TCM power domains and not core's power domain.
+ * 1st entry is used to configure r5 operation mode.
+ */
+ for (i = 1; i < r5_core->num_pm_dev; i++) {
+ r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
+ if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
+ dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
+ PTR_ERR(r5_core->pm_dev_core0[i]));
+ ret = -EINVAL;
+ goto fail_add_pm_domains;
+ }
+ r5_core->pm_dev_core0_link[i] = device_link_add(dev,
+ r5_core->pm_dev_core0[i],
+ DL_FLAG_STATELESS |
+ DL_FLAG_RPM_ACTIVE |
+ DL_FLAG_PM_RUNTIME);
+ if (!r5_core->pm_dev_core0_link[i]) {
+ dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
+ r5_core->pm_dev_core0[i] = NULL;
+ ret = -EINVAL;
+ goto fail_add_pm_domains;
+ }
+ }
+
+ if (cluster->mode == SPLIT_MODE)
+ return 0;
+
+ r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
+ sizeof(struct device *),
+ GFP_KERNEL);
+ if (!r5_core->pm_dev_core1) {
+ ret = -ENOMEM;
+ goto fail_add_pm_domains;
+ }
+
+ r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
+ sizeof(struct device_link *),
+ GFP_KERNEL);
+ if (!r5_core->pm_dev_core1_link) {
+ kfree(r5_core->pm_dev_core1);
+ r5_core->pm_dev_core1 = NULL;
+ ret = -ENOMEM;
+ goto fail_add_pm_domains;
+ }
+
+ /* get second core's device to detach its power-domains */
+ np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
+
+ pdev = of_find_device_by_node(np);
+ if (!pdev) {
+ dev_err(cluster->dev, "core1 platform device not available\n");
+ kfree(r5_core->pm_dev_core1);
+ kfree(r5_core->pm_dev_core1_link);
+ r5_core->pm_dev_core1 = NULL;
+ r5_core->pm_dev_core1_link = NULL;
+ ret = -EINVAL;
+ goto fail_add_pm_domains;
+ }
+
+ dev2 = &pdev->dev;
+
+ /* for zynqmp we only add TCM power domains and not core's power domain */
+ for (j = 1; j < r5_core->num_pm_dev; j++) {
+ r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
+ if (!r5_core->pm_dev_core1[j]) {
+ dev_dbg(dev, "can't attach to pm domain %d\n", j);
+ ret = -EINVAL;
+ goto fail_add_pm_domains_lockstep;
+ } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
+ dev_dbg(dev, "can't attach to pm domain %d\n", j);
+ ret = PTR_ERR(r5_core->pm_dev_core1[j]);
+ goto fail_add_pm_domains_lockstep;
+ }
+
+ r5_core->pm_dev_core1_link[j] = device_link_add(dev,
+ r5_core->pm_dev_core1[j],
+ DL_FLAG_STATELESS |
+ DL_FLAG_RPM_ACTIVE |
+ DL_FLAG_PM_RUNTIME);
+ if (!r5_core->pm_dev_core1_link[j]) {
+ dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
+ r5_core->pm_dev_core1[j] = NULL;
+ ret = -ENODEV;
+ goto fail_add_pm_domains_lockstep;
+ }
+ }
+
+fail_add_pm_domains_lockstep:
+ while (--j >= 0) {
+ device_link_del(r5_core->pm_dev_core1_link[j]);
+ dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
+ }
+ kfree(r5_core->pm_dev_core1);
+ r5_core->pm_dev_core1 = NULL;
+ kfree(r5_core->pm_dev_core1_link);
+ r5_core->pm_dev_core1_link = NULL;
+
+fail_add_pm_domains:
+ while (--i >= 0) {
+ device_link_del(r5_core->pm_dev_core0_link[i]);
+ dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
+ }
+ kfree(r5_core->pm_dev_core0);
+ r5_core->pm_dev_core0 = NULL;
+ kfree(r5_core->pm_dev_core0_link);
+ r5_core->pm_dev_core0_link = NULL;
+
+ return ret;
+}
+
/**
* zynqmp_r5_rproc_prepare()
* adds carveouts for TCM bank and reserved memory regions
@@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
{
int ret;
+ ret = zynqmp_r5_add_pm_domains(rproc);
+ if (ret) {
+ dev_err(&rproc->dev, "failed to add pm domains\n");
+ return ret;
+ }
+
ret = add_tcm_banks(rproc);
if (ret) {
dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
- return ret;
+ goto fail_prepare;
}
ret = add_mem_regions_carveout(rproc);
if (ret) {
dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
- return ret;
+ goto fail_prepare;
}
return 0;
+
+fail_prepare:
+ zynqmp_r5_remove_pm_domains(rproc);
+
+ return ret;
}
/**
@@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
r5_core = rproc->priv;
+ zynqmp_r5_remove_pm_domains(rproc);
+
for (i = 0; i < r5_core->tcm_bank_count; i++) {
pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
if (zynqmp_pm_release_node(pm_domain_id))
--
2.25.1
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-11-17 17:42 ` Tanmay Shah
0 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-17 17:42 UTC (permalink / raw)
To: andersson, mathieu.poirier, robh+dt, krzysztof.kozlowski+dt,
conor+dt, michal.simek, ben.levinsky, tanmay.shah
Cc: linux-remoteproc, devicetree, linux-arm-kernel, linux-kernel
Use TCM pm domains extracted from device-tree
to power on/off TCM using general pm domain framework.
Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
---
Changes in v7:
- %s/pm_dev1/pm_dev_core0/r
- %s/pm_dev_link1/pm_dev_core0_link/r
- %s/pm_dev2/pm_dev_core1/r
- %s/pm_dev_link2/pm_dev_core1_link/r
- remove pm_domain_id check to move next patch
- add comment about how 1st entry in pm domain list is used
- fix loop when jump to fail_add_pm_domains loop
drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
1 file changed, 212 insertions(+), 3 deletions(-)
diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
index 4395edea9a64..22bccc5075a0 100644
--- a/drivers/remoteproc/xlnx_r5_remoteproc.c
+++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
@@ -16,6 +16,7 @@
#include <linux/of_reserved_mem.h>
#include <linux/platform_device.h>
#include <linux/remoteproc.h>
+#include <linux/pm_domain.h>
#include "remoteproc_internal.h"
@@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
* @rproc: rproc handle
* @pm_domain_id: RPU CPU power domain id
* @ipi: pointer to mailbox information
+ * @num_pm_dev: number of tcm pm domain devices for this core
+ * @pm_dev_core0: pm domain virtual devices for power domain framework
+ * @pm_dev_core0_link: pm domain device links after registration
+ * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
+ * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
+ * registration
*/
struct zynqmp_r5_core {
struct device *dev;
@@ -111,6 +118,11 @@ struct zynqmp_r5_core {
struct rproc *rproc;
u32 pm_domain_id;
struct mbox_info *ipi;
+ int num_pm_dev;
+ struct device **pm_dev_core0;
+ struct device_link **pm_dev_core0_link;
+ struct device **pm_dev_core1;
+ struct device_link **pm_dev_core1_link;
};
/**
@@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
ZYNQMP_PM_CAPABILITY_ACCESS, 0,
ZYNQMP_PM_REQUEST_ACK_BLOCKING);
if (ret < 0) {
- dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
+ dev_err(dev, "failed to turn on TCM 0x%x",
+ pm_domain_id);
goto release_tcm_lockstep;
}
@@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
return ret;
}
+static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
+{
+ struct zynqmp_r5_core *r5_core = rproc->priv;
+ struct device *dev = r5_core->dev;
+ struct zynqmp_r5_cluster *cluster;
+ int i;
+
+ cluster = platform_get_drvdata(to_platform_device(dev->parent));
+
+ for (i = 1; i < r5_core->num_pm_dev; i++) {
+ device_link_del(r5_core->pm_dev_core0_link[i]);
+ dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
+ }
+
+ kfree(r5_core->pm_dev_core0);
+ r5_core->pm_dev_core0 = NULL;
+ kfree(r5_core->pm_dev_core0_link);
+ r5_core->pm_dev_core0_link = NULL;
+
+ if (cluster->mode == SPLIT_MODE) {
+ r5_core->num_pm_dev = 0;
+ return;
+ }
+
+ for (i = 1; i < r5_core->num_pm_dev; i++) {
+ device_link_del(r5_core->pm_dev_core1_link[i]);
+ dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
+ }
+
+ kfree(r5_core->pm_dev_core1);
+ r5_core->pm_dev_core1 = NULL;
+ kfree(r5_core->pm_dev_core1_link);
+ r5_core->pm_dev_core1_link = NULL;
+ r5_core->num_pm_dev = 0;
+}
+
+static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
+{
+ struct zynqmp_r5_core *r5_core = rproc->priv;
+ struct device *dev = r5_core->dev, *dev2;
+ struct zynqmp_r5_cluster *cluster;
+ struct platform_device *pdev;
+ struct device_node *np;
+ int i, j, num_pm_dev, ret;
+
+ cluster = dev_get_drvdata(dev->parent);
+
+ /* get number of power-domains */
+ num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
+ "#power-domain-cells");
+
+ if (num_pm_dev <= 0)
+ return -EINVAL;
+
+ r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
+ sizeof(struct device *),
+ GFP_KERNEL);
+ if (!r5_core->pm_dev_core0)
+ ret = -ENOMEM;
+
+ r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
+ sizeof(struct device_link *),
+ GFP_KERNEL);
+ if (!r5_core->pm_dev_core0_link) {
+ kfree(r5_core->pm_dev_core0);
+ r5_core->pm_dev_core0 = NULL;
+ return -ENOMEM;
+ }
+
+ r5_core->num_pm_dev = num_pm_dev;
+
+ /*
+ * start from 2nd entry in power-domains property list as
+ * for zynqmp we only add TCM power domains and not core's power domain.
+ * 1st entry is used to configure r5 operation mode.
+ */
+ for (i = 1; i < r5_core->num_pm_dev; i++) {
+ r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
+ if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
+ dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
+ PTR_ERR(r5_core->pm_dev_core0[i]));
+ ret = -EINVAL;
+ goto fail_add_pm_domains;
+ }
+ r5_core->pm_dev_core0_link[i] = device_link_add(dev,
+ r5_core->pm_dev_core0[i],
+ DL_FLAG_STATELESS |
+ DL_FLAG_RPM_ACTIVE |
+ DL_FLAG_PM_RUNTIME);
+ if (!r5_core->pm_dev_core0_link[i]) {
+ dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
+ r5_core->pm_dev_core0[i] = NULL;
+ ret = -EINVAL;
+ goto fail_add_pm_domains;
+ }
+ }
+
+ if (cluster->mode == SPLIT_MODE)
+ return 0;
+
+ r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
+ sizeof(struct device *),
+ GFP_KERNEL);
+ if (!r5_core->pm_dev_core1) {
+ ret = -ENOMEM;
+ goto fail_add_pm_domains;
+ }
+
+ r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
+ sizeof(struct device_link *),
+ GFP_KERNEL);
+ if (!r5_core->pm_dev_core1_link) {
+ kfree(r5_core->pm_dev_core1);
+ r5_core->pm_dev_core1 = NULL;
+ ret = -ENOMEM;
+ goto fail_add_pm_domains;
+ }
+
+ /* get second core's device to detach its power-domains */
+ np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
+
+ pdev = of_find_device_by_node(np);
+ if (!pdev) {
+ dev_err(cluster->dev, "core1 platform device not available\n");
+ kfree(r5_core->pm_dev_core1);
+ kfree(r5_core->pm_dev_core1_link);
+ r5_core->pm_dev_core1 = NULL;
+ r5_core->pm_dev_core1_link = NULL;
+ ret = -EINVAL;
+ goto fail_add_pm_domains;
+ }
+
+ dev2 = &pdev->dev;
+
+ /* for zynqmp we only add TCM power domains and not core's power domain */
+ for (j = 1; j < r5_core->num_pm_dev; j++) {
+ r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
+ if (!r5_core->pm_dev_core1[j]) {
+ dev_dbg(dev, "can't attach to pm domain %d\n", j);
+ ret = -EINVAL;
+ goto fail_add_pm_domains_lockstep;
+ } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
+ dev_dbg(dev, "can't attach to pm domain %d\n", j);
+ ret = PTR_ERR(r5_core->pm_dev_core1[j]);
+ goto fail_add_pm_domains_lockstep;
+ }
+
+ r5_core->pm_dev_core1_link[j] = device_link_add(dev,
+ r5_core->pm_dev_core1[j],
+ DL_FLAG_STATELESS |
+ DL_FLAG_RPM_ACTIVE |
+ DL_FLAG_PM_RUNTIME);
+ if (!r5_core->pm_dev_core1_link[j]) {
+ dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
+ r5_core->pm_dev_core1[j] = NULL;
+ ret = -ENODEV;
+ goto fail_add_pm_domains_lockstep;
+ }
+ }
+
+fail_add_pm_domains_lockstep:
+ while (--j >= 0) {
+ device_link_del(r5_core->pm_dev_core1_link[j]);
+ dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
+ }
+ kfree(r5_core->pm_dev_core1);
+ r5_core->pm_dev_core1 = NULL;
+ kfree(r5_core->pm_dev_core1_link);
+ r5_core->pm_dev_core1_link = NULL;
+
+fail_add_pm_domains:
+ while (--i >= 0) {
+ device_link_del(r5_core->pm_dev_core0_link[i]);
+ dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
+ }
+ kfree(r5_core->pm_dev_core0);
+ r5_core->pm_dev_core0 = NULL;
+ kfree(r5_core->pm_dev_core0_link);
+ r5_core->pm_dev_core0_link = NULL;
+
+ return ret;
+}
+
/**
* zynqmp_r5_rproc_prepare()
* adds carveouts for TCM bank and reserved memory regions
@@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
{
int ret;
+ ret = zynqmp_r5_add_pm_domains(rproc);
+ if (ret) {
+ dev_err(&rproc->dev, "failed to add pm domains\n");
+ return ret;
+ }
+
ret = add_tcm_banks(rproc);
if (ret) {
dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
- return ret;
+ goto fail_prepare;
}
ret = add_mem_regions_carveout(rproc);
if (ret) {
dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
- return ret;
+ goto fail_prepare;
}
return 0;
+
+fail_prepare:
+ zynqmp_r5_remove_pm_domains(rproc);
+
+ return ret;
}
/**
@@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
r5_core = rproc->priv;
+ zynqmp_r5_remove_pm_domains(rproc);
+
for (i = 0; i < r5_core->tcm_bank_count; i++) {
pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
if (zynqmp_pm_release_node(pm_domain_id))
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 4/4] remoteproc: zynqmp: parse TCM from device tree
2023-11-17 17:42 ` Tanmay Shah
@ 2023-11-17 17:42 ` Tanmay Shah
-1 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-17 17:42 UTC (permalink / raw)
To: andersson, mathieu.poirier, robh+dt, krzysztof.kozlowski+dt,
conor+dt, michal.simek, ben.levinsky, tanmay.shah
Cc: linux-remoteproc, devicetree, linux-arm-kernel, linux-kernel
ZynqMP TCM information is fixed in driver. Now ZynqMP TCM information
is available in device-tree. Parse TCM information in driver
as per new bindings.
Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
---
Changes in v7:
- move checking of pm_domain_id from previous patch
- fix mem_bank_data memory allocation
drivers/remoteproc/xlnx_r5_remoteproc.c | 152 ++++++++++++++++++++----
1 file changed, 128 insertions(+), 24 deletions(-)
diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
index 22bccc5075a0..270af73344ef 100644
--- a/drivers/remoteproc/xlnx_r5_remoteproc.c
+++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
@@ -75,8 +75,8 @@ struct mbox_info {
};
/*
- * Hardcoded TCM bank values. This will be removed once TCM bindings are
- * accepted for system-dt specifications and upstreamed in linux kernel
+ * Hardcoded TCM bank values. This will stay in driver to maintain backward
+ * compatibility with device-tree that does not have TCM information.
*/
static const struct mem_bank_data zynqmp_tcm_banks_split[] = {
{0xffe00000UL, 0x0, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
@@ -587,12 +587,21 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
bank_size = r5_core->tcm_banks[i]->size;
pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
- ret = zynqmp_pm_request_node(pm_domain_id,
- ZYNQMP_PM_CAPABILITY_ACCESS, 0,
- ZYNQMP_PM_REQUEST_ACK_BLOCKING);
- if (ret < 0) {
- dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
- goto release_tcm_split;
+ /*
+ * If TCM information is available in device-tree then
+ * in that case, pm domain framework will power on/off TCM.
+ * In that case pm_domain_id is set to 0. If hardcode
+ * bindings from driver is used, then only this driver will
+ * use pm_domain_id.
+ */
+ if (pm_domain_id) {
+ ret = zynqmp_pm_request_node(pm_domain_id,
+ ZYNQMP_PM_CAPABILITY_ACCESS, 0,
+ ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+ if (ret < 0) {
+ dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
+ goto release_tcm_split;
+ }
}
dev_dbg(dev, "TCM carveout split mode %s addr=%llx, da=0x%x, size=0x%lx",
@@ -604,7 +613,8 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
bank_name);
if (!rproc_mem) {
ret = -ENOMEM;
- zynqmp_pm_release_node(pm_domain_id);
+ if (pm_domain_id)
+ zynqmp_pm_release_node(pm_domain_id);
goto release_tcm_split;
}
@@ -617,7 +627,8 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
/* If failed, Turn off all TCM banks turned on before */
for (i--; i >= 0; i--) {
pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
- zynqmp_pm_release_node(pm_domain_id);
+ if (pm_domain_id)
+ zynqmp_pm_release_node(pm_domain_id);
}
return ret;
}
@@ -659,13 +670,16 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
/* Turn on each TCM bank individually */
- ret = zynqmp_pm_request_node(pm_domain_id,
- ZYNQMP_PM_CAPABILITY_ACCESS, 0,
- ZYNQMP_PM_REQUEST_ACK_BLOCKING);
- if (ret < 0) {
- dev_err(dev, "failed to turn on TCM 0x%x",
- pm_domain_id);
- goto release_tcm_lockstep;
+
+ if (pm_domain_id) {
+ ret = zynqmp_pm_request_node(pm_domain_id,
+ ZYNQMP_PM_CAPABILITY_ACCESS, 0,
+ ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+ if (ret < 0) {
+ dev_err(dev, "failed to turn on TCM 0x%x",
+ pm_domain_id);
+ goto release_tcm_lockstep;
+ }
}
bank_size = r5_core->tcm_banks[i]->size;
@@ -683,7 +697,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
bank_name);
if (!rproc_mem) {
ret = -ENOMEM;
- zynqmp_pm_release_node(pm_domain_id);
+ if (pm_domain_id)
+ zynqmp_pm_release_node(pm_domain_id);
goto release_tcm_lockstep;
}
@@ -700,7 +715,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
/* If failed, Turn off all TCM banks turned on before */
for (i--; i >= 0; i--) {
pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
- zynqmp_pm_release_node(pm_domain_id);
+ if (pm_domain_id)
+ zynqmp_pm_release_node(pm_domain_id);
}
return ret;
}
@@ -931,6 +947,8 @@ static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
}
}
+ return 0;
+
fail_add_pm_domains_lockstep:
while (--j >= 0) {
device_link_del(r5_core->pm_dev_core1_link[j]);
@@ -1012,7 +1030,7 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
for (i = 0; i < r5_core->tcm_bank_count; i++) {
pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
- if (zynqmp_pm_release_node(pm_domain_id))
+ if (pm_domain_id && zynqmp_pm_release_node(pm_domain_id))
dev_warn(r5_core->dev,
"can't turn off TCM bank 0x%x", pm_domain_id);
}
@@ -1087,6 +1105,83 @@ static struct zynqmp_r5_core *zynqmp_r5_add_rproc_core(struct device *cdev)
return ERR_PTR(ret);
}
+static int zynqmp_r5_get_tcm_node_from_dt(struct zynqmp_r5_cluster *cluster)
+{
+ struct zynqmp_r5_core *r5_core;
+ int i, j, tcm_bank_count, ret;
+ struct platform_device *cpdev;
+ struct mem_bank_data *tcm;
+ struct device_node *np;
+ struct resource *res;
+ u64 abs_addr, size;
+ struct device *dev;
+
+ for (i = 0; i < cluster->core_count; i++) {
+ r5_core = cluster->r5_cores[i];
+ dev = r5_core->dev;
+ np = dev_of_node(dev);
+
+ /* we have address cell 2 and size cell as 2 */
+ ret = of_property_count_elems_of_size(np, "reg",
+ 4 * sizeof(u32));
+ if (ret <= 0) {
+ dev_err(dev, "can't get reg property err %d\n", ret);
+ return -EINVAL;
+ }
+
+ tcm_bank_count = ret;
+
+ r5_core->tcm_banks = devm_kcalloc(dev, tcm_bank_count,
+ sizeof(struct mem_bank_data *),
+ GFP_KERNEL);
+ if (!r5_core->tcm_banks)
+ ret = -ENOMEM;
+
+ r5_core->tcm_bank_count = tcm_bank_count;
+ for (j = 0; j < tcm_bank_count; j++) {
+ tcm = devm_kzalloc(dev, sizeof(struct mem_bank_data),
+ GFP_KERNEL);
+ if (!tcm)
+ return -ENOMEM;
+
+ r5_core->tcm_banks[j] = tcm;
+
+ /* get tcm address without translation */
+ ret = of_property_read_reg(np, j, &abs_addr, &size);
+ if (ret) {
+ dev_err(dev, "failed to get reg property\n");
+ return ret;
+ }
+
+ /*
+ * remote processor can address only 32 bits
+ * so convert 64-bits into 32-bits. This will discard
+ * any unwanted upper 32-bits.
+ */
+ tcm->da = (u32)abs_addr;
+ tcm->size = (u32)size;
+
+ cpdev = to_platform_device(dev);
+ res = platform_get_resource(cpdev, IORESOURCE_MEM, j);
+ if (!res) {
+ dev_err(dev, "failed to get tcm resource\n");
+ return -EINVAL;
+ }
+
+ tcm->addr = (u32)res->start;
+ tcm->bank_name = (char *)res->name;
+ res = devm_request_mem_region(dev, tcm->addr, tcm->size,
+ tcm->bank_name);
+ if (!res) {
+ dev_err(dev, "failed to request tcm resource\n");
+ return -EINVAL;
+ }
+ }
+ }
+
+ return 0;
+}
+
/**
* zynqmp_r5_get_tcm_node()
* Ideally this function should parse tcm node and store information
@@ -1165,10 +1260,19 @@ static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster,
struct zynqmp_r5_core *r5_core;
int ret, i;
- ret = zynqmp_r5_get_tcm_node(cluster);
- if (ret < 0) {
- dev_err(dev, "can't get tcm node, err %d\n", ret);
- return ret;
+ r5_core = cluster->r5_cores[0];
+ if (of_find_property(r5_core->np, "reg", NULL)) {
+ ret = zynqmp_r5_get_tcm_node_from_dt(cluster);
+ if (ret) {
+ dev_err(dev, "can't get tcm node from dt, err %d\n", ret);
+ return ret;
+ }
+ } else {
+ ret = zynqmp_r5_get_tcm_node(cluster);
+ if (ret < 0) {
+ dev_err(dev, "can't get tcm node, err %d\n", ret);
+ return ret;
+ }
}
for (i = 0; i < cluster->core_count; i++) {
--
2.25.1
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 4/4] remoteproc: zynqmp: parse TCM from device tree
@ 2023-11-17 17:42 ` Tanmay Shah
0 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-17 17:42 UTC (permalink / raw)
To: andersson, mathieu.poirier, robh+dt, krzysztof.kozlowski+dt,
conor+dt, michal.simek, ben.levinsky, tanmay.shah
Cc: linux-remoteproc, devicetree, linux-arm-kernel, linux-kernel
ZynqMP TCM information is fixed in driver. Now ZynqMP TCM information
is available in device-tree. Parse TCM information in driver
as per new bindings.
Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
---
Changes in v7:
- move checking of pm_domain_id from previous patch
- fix mem_bank_data memory allocation
drivers/remoteproc/xlnx_r5_remoteproc.c | 152 ++++++++++++++++++++----
1 file changed, 128 insertions(+), 24 deletions(-)
diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
index 22bccc5075a0..270af73344ef 100644
--- a/drivers/remoteproc/xlnx_r5_remoteproc.c
+++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
@@ -75,8 +75,8 @@ struct mbox_info {
};
/*
- * Hardcoded TCM bank values. This will be removed once TCM bindings are
- * accepted for system-dt specifications and upstreamed in linux kernel
+ * Hardcoded TCM bank values. This will stay in driver to maintain backward
+ * compatibility with device-tree that does not have TCM information.
*/
static const struct mem_bank_data zynqmp_tcm_banks_split[] = {
{0xffe00000UL, 0x0, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
@@ -587,12 +587,21 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
bank_size = r5_core->tcm_banks[i]->size;
pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
- ret = zynqmp_pm_request_node(pm_domain_id,
- ZYNQMP_PM_CAPABILITY_ACCESS, 0,
- ZYNQMP_PM_REQUEST_ACK_BLOCKING);
- if (ret < 0) {
- dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
- goto release_tcm_split;
+ /*
+ * If TCM information is available in device-tree then
+ * in that case, pm domain framework will power on/off TCM.
+ * In that case pm_domain_id is set to 0. If hardcode
+ * bindings from driver is used, then only this driver will
+ * use pm_domain_id.
+ */
+ if (pm_domain_id) {
+ ret = zynqmp_pm_request_node(pm_domain_id,
+ ZYNQMP_PM_CAPABILITY_ACCESS, 0,
+ ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+ if (ret < 0) {
+ dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
+ goto release_tcm_split;
+ }
}
dev_dbg(dev, "TCM carveout split mode %s addr=%llx, da=0x%x, size=0x%lx",
@@ -604,7 +613,8 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
bank_name);
if (!rproc_mem) {
ret = -ENOMEM;
- zynqmp_pm_release_node(pm_domain_id);
+ if (pm_domain_id)
+ zynqmp_pm_release_node(pm_domain_id);
goto release_tcm_split;
}
@@ -617,7 +627,8 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
/* If failed, Turn off all TCM banks turned on before */
for (i--; i >= 0; i--) {
pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
- zynqmp_pm_release_node(pm_domain_id);
+ if (pm_domain_id)
+ zynqmp_pm_release_node(pm_domain_id);
}
return ret;
}
@@ -659,13 +670,16 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
/* Turn on each TCM bank individually */
- ret = zynqmp_pm_request_node(pm_domain_id,
- ZYNQMP_PM_CAPABILITY_ACCESS, 0,
- ZYNQMP_PM_REQUEST_ACK_BLOCKING);
- if (ret < 0) {
- dev_err(dev, "failed to turn on TCM 0x%x",
- pm_domain_id);
- goto release_tcm_lockstep;
+
+ if (pm_domain_id) {
+ ret = zynqmp_pm_request_node(pm_domain_id,
+ ZYNQMP_PM_CAPABILITY_ACCESS, 0,
+ ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+ if (ret < 0) {
+ dev_err(dev, "failed to turn on TCM 0x%x",
+ pm_domain_id);
+ goto release_tcm_lockstep;
+ }
}
bank_size = r5_core->tcm_banks[i]->size;
@@ -683,7 +697,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
bank_name);
if (!rproc_mem) {
ret = -ENOMEM;
- zynqmp_pm_release_node(pm_domain_id);
+ if (pm_domain_id)
+ zynqmp_pm_release_node(pm_domain_id);
goto release_tcm_lockstep;
}
@@ -700,7 +715,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
/* If failed, Turn off all TCM banks turned on before */
for (i--; i >= 0; i--) {
pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
- zynqmp_pm_release_node(pm_domain_id);
+ if (pm_domain_id)
+ zynqmp_pm_release_node(pm_domain_id);
}
return ret;
}
@@ -931,6 +947,8 @@ static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
}
}
+ return 0;
+
fail_add_pm_domains_lockstep:
while (--j >= 0) {
device_link_del(r5_core->pm_dev_core1_link[j]);
@@ -1012,7 +1030,7 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
for (i = 0; i < r5_core->tcm_bank_count; i++) {
pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
- if (zynqmp_pm_release_node(pm_domain_id))
+ if (pm_domain_id && zynqmp_pm_release_node(pm_domain_id))
dev_warn(r5_core->dev,
"can't turn off TCM bank 0x%x", pm_domain_id);
}
@@ -1087,6 +1105,83 @@ static struct zynqmp_r5_core *zynqmp_r5_add_rproc_core(struct device *cdev)
return ERR_PTR(ret);
}
+static int zynqmp_r5_get_tcm_node_from_dt(struct zynqmp_r5_cluster *cluster)
+{
+ struct zynqmp_r5_core *r5_core;
+ int i, j, tcm_bank_count, ret;
+ struct platform_device *cpdev;
+ struct mem_bank_data *tcm;
+ struct device_node *np;
+ struct resource *res;
+ u64 abs_addr, size;
+ struct device *dev;
+
+ for (i = 0; i < cluster->core_count; i++) {
+ r5_core = cluster->r5_cores[i];
+ dev = r5_core->dev;
+ np = dev_of_node(dev);
+
+ /* we have address cell 2 and size cell as 2 */
+ ret = of_property_count_elems_of_size(np, "reg",
+ 4 * sizeof(u32));
+ if (ret <= 0) {
+ dev_err(dev, "can't get reg property err %d\n", ret);
+ return -EINVAL;
+ }
+
+ tcm_bank_count = ret;
+
+ r5_core->tcm_banks = devm_kcalloc(dev, tcm_bank_count,
+ sizeof(struct mem_bank_data *),
+ GFP_KERNEL);
+ if (!r5_core->tcm_banks)
+ ret = -ENOMEM;
+
+ r5_core->tcm_bank_count = tcm_bank_count;
+ for (j = 0; j < tcm_bank_count; j++) {
+ tcm = devm_kzalloc(dev, sizeof(struct mem_bank_data),
+ GFP_KERNEL);
+ if (!tcm)
+ return -ENOMEM;
+
+ r5_core->tcm_banks[j] = tcm;
+
+ /* get tcm address without translation */
+ ret = of_property_read_reg(np, j, &abs_addr, &size);
+ if (ret) {
+ dev_err(dev, "failed to get reg property\n");
+ return ret;
+ }
+
+ /*
+ * remote processor can address only 32 bits
+ * so convert 64-bits into 32-bits. This will discard
+ * any unwanted upper 32-bits.
+ */
+ tcm->da = (u32)abs_addr;
+ tcm->size = (u32)size;
+
+ cpdev = to_platform_device(dev);
+ res = platform_get_resource(cpdev, IORESOURCE_MEM, j);
+ if (!res) {
+ dev_err(dev, "failed to get tcm resource\n");
+ return -EINVAL;
+ }
+
+ tcm->addr = (u32)res->start;
+ tcm->bank_name = (char *)res->name;
+ res = devm_request_mem_region(dev, tcm->addr, tcm->size,
+ tcm->bank_name);
+ if (!res) {
+ dev_err(dev, "failed to request tcm resource\n");
+ return -EINVAL;
+ }
+ }
+ }
+
+ return 0;
+}
+
/**
* zynqmp_r5_get_tcm_node()
* Ideally this function should parse tcm node and store information
@@ -1165,10 +1260,19 @@ static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster,
struct zynqmp_r5_core *r5_core;
int ret, i;
- ret = zynqmp_r5_get_tcm_node(cluster);
- if (ret < 0) {
- dev_err(dev, "can't get tcm node, err %d\n", ret);
- return ret;
+ r5_core = cluster->r5_cores[0];
+ if (of_find_property(r5_core->np, "reg", NULL)) {
+ ret = zynqmp_r5_get_tcm_node_from_dt(cluster);
+ if (ret) {
+ dev_err(dev, "can't get tcm node from dt, err %d\n", ret);
+ return ret;
+ }
+ } else {
+ ret = zynqmp_r5_get_tcm_node(cluster);
+ if (ret < 0) {
+ dev_err(dev, "can't get tcm node, err %d\n", ret);
+ return ret;
+ }
}
for (i = 0; i < cluster->core_count; i++) {
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-11-17 17:42 ` Tanmay Shah
@ 2023-11-21 22:59 ` Mathieu Poirier
-1 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-11-21 22:59 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
Hi,
On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> Use TCM pm domains extracted from device-tree
> to power on/off TCM using general pm domain framework.
>
> Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> ---
>
> Changes in v7:
> - %s/pm_dev1/pm_dev_core0/r
> - %s/pm_dev_link1/pm_dev_core0_link/r
> - %s/pm_dev2/pm_dev_core1/r
> - %s/pm_dev_link2/pm_dev_core1_link/r
> - remove pm_domain_id check to move next patch
> - add comment about how 1st entry in pm domain list is used
> - fix loop when jump to fail_add_pm_domains loop
>
> drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> 1 file changed, 212 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> index 4395edea9a64..22bccc5075a0 100644
> --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -16,6 +16,7 @@
> #include <linux/of_reserved_mem.h>
> #include <linux/platform_device.h>
> #include <linux/remoteproc.h>
> +#include <linux/pm_domain.h>
>
> #include "remoteproc_internal.h"
>
> @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> * @rproc: rproc handle
> * @pm_domain_id: RPU CPU power domain id
> * @ipi: pointer to mailbox information
> + * @num_pm_dev: number of tcm pm domain devices for this core
> + * @pm_dev_core0: pm domain virtual devices for power domain framework
> + * @pm_dev_core0_link: pm domain device links after registration
> + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> + * registration
> */
> struct zynqmp_r5_core {
> struct device *dev;
> @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> struct rproc *rproc;
> u32 pm_domain_id;
> struct mbox_info *ipi;
> + int num_pm_dev;
> + struct device **pm_dev_core0;
> + struct device_link **pm_dev_core0_link;
> + struct device **pm_dev_core1;
> + struct device_link **pm_dev_core1_link;
> };
>
> /**
> @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> if (ret < 0) {
> - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> + dev_err(dev, "failed to turn on TCM 0x%x",
> + pm_domain_id);
Spurious change, you should have caught that.
> goto release_tcm_lockstep;
> }
>
> @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> return ret;
> }
>
> +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + struct device *dev = r5_core->dev;
> + struct zynqmp_r5_cluster *cluster;
> + int i;
> +
> + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> +
> + for (i = 1; i < r5_core->num_pm_dev; i++) {
> + device_link_del(r5_core->pm_dev_core0_link[i]);
> + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> + }
> +
> + kfree(r5_core->pm_dev_core0);
> + r5_core->pm_dev_core0 = NULL;
> + kfree(r5_core->pm_dev_core0_link);
> + r5_core->pm_dev_core0_link = NULL;
> +
> + if (cluster->mode == SPLIT_MODE) {
> + r5_core->num_pm_dev = 0;
> + return;
> + }
> +
> + for (i = 1; i < r5_core->num_pm_dev; i++) {
> + device_link_del(r5_core->pm_dev_core1_link[i]);
> + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> + }
> +
> + kfree(r5_core->pm_dev_core1);
> + r5_core->pm_dev_core1 = NULL;
> + kfree(r5_core->pm_dev_core1_link);
> + r5_core->pm_dev_core1_link = NULL;
> + r5_core->num_pm_dev = 0;
> +}
> +
> +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + struct device *dev = r5_core->dev, *dev2;
> + struct zynqmp_r5_cluster *cluster;
> + struct platform_device *pdev;
> + struct device_node *np;
> + int i, j, num_pm_dev, ret;
> +
> + cluster = dev_get_drvdata(dev->parent);
> +
> + /* get number of power-domains */
> + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> + "#power-domain-cells");
> +
> + if (num_pm_dev <= 0)
> + return -EINVAL;
> +
> + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> + sizeof(struct device *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core0)
> + ret = -ENOMEM;
> +
> + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> + sizeof(struct device_link *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core0_link) {
> + kfree(r5_core->pm_dev_core0);
> + r5_core->pm_dev_core0 = NULL;
> + return -ENOMEM;
> + }
> +
> + r5_core->num_pm_dev = num_pm_dev;
> +
> + /*
> + * start from 2nd entry in power-domains property list as
> + * for zynqmp we only add TCM power domains and not core's power domain.
> + * 1st entry is used to configure r5 operation mode.
You are still not saying _where_ ->pm_dev_core0[0] gets added.
> + */
> + for (i = 1; i < r5_core->num_pm_dev; i++) {
> + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
code are used in the loop for the lockstep mode. Please pick one heuristic and
stick with it. I have no preference on which one.
> + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> + PTR_ERR(r5_core->pm_dev_core0[i]));
> + ret = -EINVAL;
> + goto fail_add_pm_domains;
> + }
> + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> + r5_core->pm_dev_core0[i],
> + DL_FLAG_STATELESS |
> + DL_FLAG_RPM_ACTIVE |
> + DL_FLAG_PM_RUNTIME);
> + if (!r5_core->pm_dev_core0_link[i]) {
> + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> + r5_core->pm_dev_core0[i] = NULL;
> + ret = -EINVAL;
> + goto fail_add_pm_domains;
> + }
> + }
> +
> + if (cluster->mode == SPLIT_MODE)
> + return 0;
> +
> + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> + sizeof(struct device *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core1) {
> + ret = -ENOMEM;
> + goto fail_add_pm_domains;
> + }
> +
> + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> + sizeof(struct device_link *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core1_link) {
> + kfree(r5_core->pm_dev_core1);
> + r5_core->pm_dev_core1 = NULL;
> + ret = -ENOMEM;
> + goto fail_add_pm_domains;
> + }
> +
> + /* get second core's device to detach its power-domains */
> + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> +
> + pdev = of_find_device_by_node(np);
> + if (!pdev) {
> + dev_err(cluster->dev, "core1 platform device not available\n");
> + kfree(r5_core->pm_dev_core1);
> + kfree(r5_core->pm_dev_core1_link);
> + r5_core->pm_dev_core1 = NULL;
> + r5_core->pm_dev_core1_link = NULL;
> + ret = -EINVAL;
> + goto fail_add_pm_domains;
> + }
> +
> + dev2 = &pdev->dev;
> +
> + /* for zynqmp we only add TCM power domains and not core's power domain */
> + for (j = 1; j < r5_core->num_pm_dev; j++) {
> + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> + if (!r5_core->pm_dev_core1[j]) {
> + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> + ret = -EINVAL;
> + goto fail_add_pm_domains_lockstep;
> + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> + goto fail_add_pm_domains_lockstep;
> + }
> +
> + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> + r5_core->pm_dev_core1[j],
> + DL_FLAG_STATELESS |
> + DL_FLAG_RPM_ACTIVE |
> + DL_FLAG_PM_RUNTIME);
> + if (!r5_core->pm_dev_core1_link[j]) {
> + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> + r5_core->pm_dev_core1[j] = NULL;
> + ret = -ENODEV;
> + goto fail_add_pm_domains_lockstep;
> + }
> + }
> +
> +fail_add_pm_domains_lockstep:
> + while (--j >= 0) {
> + device_link_del(r5_core->pm_dev_core1_link[j]);
> + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> + }
> + kfree(r5_core->pm_dev_core1);
> + r5_core->pm_dev_core1 = NULL;
> + kfree(r5_core->pm_dev_core1_link);
> + r5_core->pm_dev_core1_link = NULL;
> +
> +fail_add_pm_domains:
> + while (--i >= 0) {
> + device_link_del(r5_core->pm_dev_core0_link[i]);
> + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> + }
> + kfree(r5_core->pm_dev_core0);
> + r5_core->pm_dev_core0 = NULL;
> + kfree(r5_core->pm_dev_core0_link);
> + r5_core->pm_dev_core0_link = NULL;
> +
The error path is much cleaner and readable now.
I will continue tomorrow.
Mathieu
> + return ret;
> +}
> +
> /**
> * zynqmp_r5_rproc_prepare()
> * adds carveouts for TCM bank and reserved memory regions
> @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> {
> int ret;
>
> + ret = zynqmp_r5_add_pm_domains(rproc);
> + if (ret) {
> + dev_err(&rproc->dev, "failed to add pm domains\n");
> + return ret;
> + }
> +
> ret = add_tcm_banks(rproc);
> if (ret) {
> dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> - return ret;
> + goto fail_prepare;
> }
>
> ret = add_mem_regions_carveout(rproc);
> if (ret) {
> dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> - return ret;
> + goto fail_prepare;
> }
>
> return 0;
> +
> +fail_prepare:
> + zynqmp_r5_remove_pm_domains(rproc);
> +
> + return ret;
> }
>
> /**
> @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
>
> r5_core = rproc->priv;
>
> + zynqmp_r5_remove_pm_domains(rproc);
> +
> for (i = 0; i < r5_core->tcm_bank_count; i++) {
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> if (zynqmp_pm_release_node(pm_domain_id))
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-11-21 22:59 ` Mathieu Poirier
0 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-11-21 22:59 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
Hi,
On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> Use TCM pm domains extracted from device-tree
> to power on/off TCM using general pm domain framework.
>
> Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> ---
>
> Changes in v7:
> - %s/pm_dev1/pm_dev_core0/r
> - %s/pm_dev_link1/pm_dev_core0_link/r
> - %s/pm_dev2/pm_dev_core1/r
> - %s/pm_dev_link2/pm_dev_core1_link/r
> - remove pm_domain_id check to move next patch
> - add comment about how 1st entry in pm domain list is used
> - fix loop when jump to fail_add_pm_domains loop
>
> drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> 1 file changed, 212 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> index 4395edea9a64..22bccc5075a0 100644
> --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -16,6 +16,7 @@
> #include <linux/of_reserved_mem.h>
> #include <linux/platform_device.h>
> #include <linux/remoteproc.h>
> +#include <linux/pm_domain.h>
>
> #include "remoteproc_internal.h"
>
> @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> * @rproc: rproc handle
> * @pm_domain_id: RPU CPU power domain id
> * @ipi: pointer to mailbox information
> + * @num_pm_dev: number of tcm pm domain devices for this core
> + * @pm_dev_core0: pm domain virtual devices for power domain framework
> + * @pm_dev_core0_link: pm domain device links after registration
> + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> + * registration
> */
> struct zynqmp_r5_core {
> struct device *dev;
> @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> struct rproc *rproc;
> u32 pm_domain_id;
> struct mbox_info *ipi;
> + int num_pm_dev;
> + struct device **pm_dev_core0;
> + struct device_link **pm_dev_core0_link;
> + struct device **pm_dev_core1;
> + struct device_link **pm_dev_core1_link;
> };
>
> /**
> @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> if (ret < 0) {
> - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> + dev_err(dev, "failed to turn on TCM 0x%x",
> + pm_domain_id);
Spurious change, you should have caught that.
> goto release_tcm_lockstep;
> }
>
> @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> return ret;
> }
>
> +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + struct device *dev = r5_core->dev;
> + struct zynqmp_r5_cluster *cluster;
> + int i;
> +
> + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> +
> + for (i = 1; i < r5_core->num_pm_dev; i++) {
> + device_link_del(r5_core->pm_dev_core0_link[i]);
> + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> + }
> +
> + kfree(r5_core->pm_dev_core0);
> + r5_core->pm_dev_core0 = NULL;
> + kfree(r5_core->pm_dev_core0_link);
> + r5_core->pm_dev_core0_link = NULL;
> +
> + if (cluster->mode == SPLIT_MODE) {
> + r5_core->num_pm_dev = 0;
> + return;
> + }
> +
> + for (i = 1; i < r5_core->num_pm_dev; i++) {
> + device_link_del(r5_core->pm_dev_core1_link[i]);
> + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> + }
> +
> + kfree(r5_core->pm_dev_core1);
> + r5_core->pm_dev_core1 = NULL;
> + kfree(r5_core->pm_dev_core1_link);
> + r5_core->pm_dev_core1_link = NULL;
> + r5_core->num_pm_dev = 0;
> +}
> +
> +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + struct device *dev = r5_core->dev, *dev2;
> + struct zynqmp_r5_cluster *cluster;
> + struct platform_device *pdev;
> + struct device_node *np;
> + int i, j, num_pm_dev, ret;
> +
> + cluster = dev_get_drvdata(dev->parent);
> +
> + /* get number of power-domains */
> + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> + "#power-domain-cells");
> +
> + if (num_pm_dev <= 0)
> + return -EINVAL;
> +
> + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> + sizeof(struct device *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core0)
> + ret = -ENOMEM;
> +
> + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> + sizeof(struct device_link *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core0_link) {
> + kfree(r5_core->pm_dev_core0);
> + r5_core->pm_dev_core0 = NULL;
> + return -ENOMEM;
> + }
> +
> + r5_core->num_pm_dev = num_pm_dev;
> +
> + /*
> + * start from 2nd entry in power-domains property list as
> + * for zynqmp we only add TCM power domains and not core's power domain.
> + * 1st entry is used to configure r5 operation mode.
You are still not saying _where_ ->pm_dev_core0[0] gets added.
> + */
> + for (i = 1; i < r5_core->num_pm_dev; i++) {
> + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
code are used in the loop for the lockstep mode. Please pick one heuristic and
stick with it. I have no preference on which one.
> + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> + PTR_ERR(r5_core->pm_dev_core0[i]));
> + ret = -EINVAL;
> + goto fail_add_pm_domains;
> + }
> + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> + r5_core->pm_dev_core0[i],
> + DL_FLAG_STATELESS |
> + DL_FLAG_RPM_ACTIVE |
> + DL_FLAG_PM_RUNTIME);
> + if (!r5_core->pm_dev_core0_link[i]) {
> + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> + r5_core->pm_dev_core0[i] = NULL;
> + ret = -EINVAL;
> + goto fail_add_pm_domains;
> + }
> + }
> +
> + if (cluster->mode == SPLIT_MODE)
> + return 0;
> +
> + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> + sizeof(struct device *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core1) {
> + ret = -ENOMEM;
> + goto fail_add_pm_domains;
> + }
> +
> + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> + sizeof(struct device_link *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core1_link) {
> + kfree(r5_core->pm_dev_core1);
> + r5_core->pm_dev_core1 = NULL;
> + ret = -ENOMEM;
> + goto fail_add_pm_domains;
> + }
> +
> + /* get second core's device to detach its power-domains */
> + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> +
> + pdev = of_find_device_by_node(np);
> + if (!pdev) {
> + dev_err(cluster->dev, "core1 platform device not available\n");
> + kfree(r5_core->pm_dev_core1);
> + kfree(r5_core->pm_dev_core1_link);
> + r5_core->pm_dev_core1 = NULL;
> + r5_core->pm_dev_core1_link = NULL;
> + ret = -EINVAL;
> + goto fail_add_pm_domains;
> + }
> +
> + dev2 = &pdev->dev;
> +
> + /* for zynqmp we only add TCM power domains and not core's power domain */
> + for (j = 1; j < r5_core->num_pm_dev; j++) {
> + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> + if (!r5_core->pm_dev_core1[j]) {
> + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> + ret = -EINVAL;
> + goto fail_add_pm_domains_lockstep;
> + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> + goto fail_add_pm_domains_lockstep;
> + }
> +
> + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> + r5_core->pm_dev_core1[j],
> + DL_FLAG_STATELESS |
> + DL_FLAG_RPM_ACTIVE |
> + DL_FLAG_PM_RUNTIME);
> + if (!r5_core->pm_dev_core1_link[j]) {
> + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> + r5_core->pm_dev_core1[j] = NULL;
> + ret = -ENODEV;
> + goto fail_add_pm_domains_lockstep;
> + }
> + }
> +
> +fail_add_pm_domains_lockstep:
> + while (--j >= 0) {
> + device_link_del(r5_core->pm_dev_core1_link[j]);
> + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> + }
> + kfree(r5_core->pm_dev_core1);
> + r5_core->pm_dev_core1 = NULL;
> + kfree(r5_core->pm_dev_core1_link);
> + r5_core->pm_dev_core1_link = NULL;
> +
> +fail_add_pm_domains:
> + while (--i >= 0) {
> + device_link_del(r5_core->pm_dev_core0_link[i]);
> + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> + }
> + kfree(r5_core->pm_dev_core0);
> + r5_core->pm_dev_core0 = NULL;
> + kfree(r5_core->pm_dev_core0_link);
> + r5_core->pm_dev_core0_link = NULL;
> +
The error path is much cleaner and readable now.
I will continue tomorrow.
Mathieu
> + return ret;
> +}
> +
> /**
> * zynqmp_r5_rproc_prepare()
> * adds carveouts for TCM bank and reserved memory regions
> @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> {
> int ret;
>
> + ret = zynqmp_r5_add_pm_domains(rproc);
> + if (ret) {
> + dev_err(&rproc->dev, "failed to add pm domains\n");
> + return ret;
> + }
> +
> ret = add_tcm_banks(rproc);
> if (ret) {
> dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> - return ret;
> + goto fail_prepare;
> }
>
> ret = add_mem_regions_carveout(rproc);
> if (ret) {
> dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> - return ret;
> + goto fail_prepare;
> }
>
> return 0;
> +
> +fail_prepare:
> + zynqmp_r5_remove_pm_domains(rproc);
> +
> + return ret;
> }
>
> /**
> @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
>
> r5_core = rproc->priv;
>
> + zynqmp_r5_remove_pm_domains(rproc);
> +
> for (i = 0; i < r5_core->tcm_bank_count; i++) {
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> if (zynqmp_pm_release_node(pm_domain_id))
> --
> 2.25.1
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-11-17 17:42 ` Tanmay Shah
@ 2023-11-22 17:12 ` Mathieu Poirier
-1 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-11-22 17:12 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> Use TCM pm domains extracted from device-tree
> to power on/off TCM using general pm domain framework.
>
> Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> ---
>
> Changes in v7:
> - %s/pm_dev1/pm_dev_core0/r
> - %s/pm_dev_link1/pm_dev_core0_link/r
> - %s/pm_dev2/pm_dev_core1/r
> - %s/pm_dev_link2/pm_dev_core1_link/r
> - remove pm_domain_id check to move next patch
> - add comment about how 1st entry in pm domain list is used
> - fix loop when jump to fail_add_pm_domains loop
>
> drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> 1 file changed, 212 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> index 4395edea9a64..22bccc5075a0 100644
> --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -16,6 +16,7 @@
> #include <linux/of_reserved_mem.h>
> #include <linux/platform_device.h>
> #include <linux/remoteproc.h>
> +#include <linux/pm_domain.h>
>
> #include "remoteproc_internal.h"
>
> @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> * @rproc: rproc handle
> * @pm_domain_id: RPU CPU power domain id
> * @ipi: pointer to mailbox information
> + * @num_pm_dev: number of tcm pm domain devices for this core
> + * @pm_dev_core0: pm domain virtual devices for power domain framework
> + * @pm_dev_core0_link: pm domain device links after registration
> + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> + * registration
> */
> struct zynqmp_r5_core {
> struct device *dev;
> @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> struct rproc *rproc;
> u32 pm_domain_id;
> struct mbox_info *ipi;
> + int num_pm_dev;
> + struct device **pm_dev_core0;
> + struct device_link **pm_dev_core0_link;
> + struct device **pm_dev_core1;
> + struct device_link **pm_dev_core1_link;
> };
>
> /**
> @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> if (ret < 0) {
> - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> + dev_err(dev, "failed to turn on TCM 0x%x",
> + pm_domain_id);
> goto release_tcm_lockstep;
> }
>
> @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> return ret;
> }
>
> +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + struct device *dev = r5_core->dev;
> + struct zynqmp_r5_cluster *cluster;
> + int i;
> +
> + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> +
> + for (i = 1; i < r5_core->num_pm_dev; i++) {
> + device_link_del(r5_core->pm_dev_core0_link[i]);
> + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> + }
> +
> + kfree(r5_core->pm_dev_core0);
> + r5_core->pm_dev_core0 = NULL;
> + kfree(r5_core->pm_dev_core0_link);
> + r5_core->pm_dev_core0_link = NULL;
> +
> + if (cluster->mode == SPLIT_MODE) {
> + r5_core->num_pm_dev = 0;
> + return;
> + }
> +
> + for (i = 1; i < r5_core->num_pm_dev; i++) {
> + device_link_del(r5_core->pm_dev_core1_link[i]);
> + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> + }
> +
> + kfree(r5_core->pm_dev_core1);
> + r5_core->pm_dev_core1 = NULL;
> + kfree(r5_core->pm_dev_core1_link);
> + r5_core->pm_dev_core1_link = NULL;
> + r5_core->num_pm_dev = 0;
> +}
> +
> +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + struct device *dev = r5_core->dev, *dev2;
> + struct zynqmp_r5_cluster *cluster;
> + struct platform_device *pdev;
> + struct device_node *np;
> + int i, j, num_pm_dev, ret;
> +
> + cluster = dev_get_drvdata(dev->parent);
> +
> + /* get number of power-domains */
> + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> + "#power-domain-cells");
> +
> + if (num_pm_dev <= 0)
> + return -EINVAL;
> +
> + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> + sizeof(struct device *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core0)
> + ret = -ENOMEM;
> +
> + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> + sizeof(struct device_link *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core0_link) {
> + kfree(r5_core->pm_dev_core0);
> + r5_core->pm_dev_core0 = NULL;
> + return -ENOMEM;
> + }
> +
> + r5_core->num_pm_dev = num_pm_dev;
> +
> + /*
> + * start from 2nd entry in power-domains property list as
> + * for zynqmp we only add TCM power domains and not core's power domain.
> + * 1st entry is used to configure r5 operation mode.
> + */
> + for (i = 1; i < r5_core->num_pm_dev; i++) {
> + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> + PTR_ERR(r5_core->pm_dev_core0[i]));
> + ret = -EINVAL;
> + goto fail_add_pm_domains;
> + }
> + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> + r5_core->pm_dev_core0[i],
> + DL_FLAG_STATELESS |
> + DL_FLAG_RPM_ACTIVE |
> + DL_FLAG_PM_RUNTIME);
> + if (!r5_core->pm_dev_core0_link[i]) {
> + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> + r5_core->pm_dev_core0[i] = NULL;
> + ret = -EINVAL;
> + goto fail_add_pm_domains;
> + }
> + }
> +
> + if (cluster->mode == SPLIT_MODE)
> + return 0;
> +
> + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> + sizeof(struct device *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core1) {
> + ret = -ENOMEM;
> + goto fail_add_pm_domains;
> + }
> +
> + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> + sizeof(struct device_link *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core1_link) {
> + kfree(r5_core->pm_dev_core1);
> + r5_core->pm_dev_core1 = NULL;
> + ret = -ENOMEM;
> + goto fail_add_pm_domains;
> + }
> +
> + /* get second core's device to detach its power-domains */
Attach or detach?
> + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> +
> + pdev = of_find_device_by_node(np);
> + if (!pdev) {
> + dev_err(cluster->dev, "core1 platform device not available\n");
> + kfree(r5_core->pm_dev_core1);
> + kfree(r5_core->pm_dev_core1_link);
> + r5_core->pm_dev_core1 = NULL;
> + r5_core->pm_dev_core1_link = NULL;
> + ret = -EINVAL;
> + goto fail_add_pm_domains;
> + }
> +
> + dev2 = &pdev->dev;
> +
> + /* for zynqmp we only add TCM power domains and not core's power domain */
> + for (j = 1; j < r5_core->num_pm_dev; j++) {
> + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> + if (!r5_core->pm_dev_core1[j]) {
> + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> + ret = -EINVAL;
> + goto fail_add_pm_domains_lockstep;
> + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> + goto fail_add_pm_domains_lockstep;
> + }
> +
> + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> + r5_core->pm_dev_core1[j],
> + DL_FLAG_STATELESS |
> + DL_FLAG_RPM_ACTIVE |
> + DL_FLAG_PM_RUNTIME);
> + if (!r5_core->pm_dev_core1_link[j]) {
> + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> + r5_core->pm_dev_core1[j] = NULL;
> + ret = -ENODEV;
> + goto fail_add_pm_domains_lockstep;
> + }
> + }
> +
> +fail_add_pm_domains_lockstep:
> + while (--j >= 0) {
> + device_link_del(r5_core->pm_dev_core1_link[j]);
> + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> + }
> + kfree(r5_core->pm_dev_core1);
> + r5_core->pm_dev_core1 = NULL;
> + kfree(r5_core->pm_dev_core1_link);
> + r5_core->pm_dev_core1_link = NULL;
> +
> +fail_add_pm_domains:
> + while (--i >= 0) {
> + device_link_del(r5_core->pm_dev_core0_link[i]);
> + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> + }
> + kfree(r5_core->pm_dev_core0);
> + r5_core->pm_dev_core0 = NULL;
> + kfree(r5_core->pm_dev_core0_link);
> + r5_core->pm_dev_core0_link = NULL;
> +
> + return ret;
> +}
> +
> /**
> * zynqmp_r5_rproc_prepare()
> * adds carveouts for TCM bank and reserved memory regions
> @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> {
> int ret;
>
> + ret = zynqmp_r5_add_pm_domains(rproc);
> + if (ret) {
> + dev_err(&rproc->dev, "failed to add pm domains\n");
> + return ret;
> + }
> +
> ret = add_tcm_banks(rproc);
> if (ret) {
> dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> - return ret;
> + goto fail_prepare;
> }
>
> ret = add_mem_regions_carveout(rproc);
> if (ret) {
> dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> - return ret;
> + goto fail_prepare;
> }
>
> return 0;
> +
> +fail_prepare:
> + zynqmp_r5_remove_pm_domains(rproc);
> +
> + return ret;
> }
>
> /**
> @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
>
> r5_core = rproc->priv;
>
> + zynqmp_r5_remove_pm_domains(rproc);
> +
> for (i = 0; i < r5_core->tcm_bank_count; i++) {
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> if (zynqmp_pm_release_node(pm_domain_id))
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-11-22 17:12 ` Mathieu Poirier
0 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-11-22 17:12 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> Use TCM pm domains extracted from device-tree
> to power on/off TCM using general pm domain framework.
>
> Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> ---
>
> Changes in v7:
> - %s/pm_dev1/pm_dev_core0/r
> - %s/pm_dev_link1/pm_dev_core0_link/r
> - %s/pm_dev2/pm_dev_core1/r
> - %s/pm_dev_link2/pm_dev_core1_link/r
> - remove pm_domain_id check to move next patch
> - add comment about how 1st entry in pm domain list is used
> - fix loop when jump to fail_add_pm_domains loop
>
> drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> 1 file changed, 212 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> index 4395edea9a64..22bccc5075a0 100644
> --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -16,6 +16,7 @@
> #include <linux/of_reserved_mem.h>
> #include <linux/platform_device.h>
> #include <linux/remoteproc.h>
> +#include <linux/pm_domain.h>
>
> #include "remoteproc_internal.h"
>
> @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> * @rproc: rproc handle
> * @pm_domain_id: RPU CPU power domain id
> * @ipi: pointer to mailbox information
> + * @num_pm_dev: number of tcm pm domain devices for this core
> + * @pm_dev_core0: pm domain virtual devices for power domain framework
> + * @pm_dev_core0_link: pm domain device links after registration
> + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> + * registration
> */
> struct zynqmp_r5_core {
> struct device *dev;
> @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> struct rproc *rproc;
> u32 pm_domain_id;
> struct mbox_info *ipi;
> + int num_pm_dev;
> + struct device **pm_dev_core0;
> + struct device_link **pm_dev_core0_link;
> + struct device **pm_dev_core1;
> + struct device_link **pm_dev_core1_link;
> };
>
> /**
> @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> if (ret < 0) {
> - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> + dev_err(dev, "failed to turn on TCM 0x%x",
> + pm_domain_id);
> goto release_tcm_lockstep;
> }
>
> @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> return ret;
> }
>
> +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + struct device *dev = r5_core->dev;
> + struct zynqmp_r5_cluster *cluster;
> + int i;
> +
> + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> +
> + for (i = 1; i < r5_core->num_pm_dev; i++) {
> + device_link_del(r5_core->pm_dev_core0_link[i]);
> + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> + }
> +
> + kfree(r5_core->pm_dev_core0);
> + r5_core->pm_dev_core0 = NULL;
> + kfree(r5_core->pm_dev_core0_link);
> + r5_core->pm_dev_core0_link = NULL;
> +
> + if (cluster->mode == SPLIT_MODE) {
> + r5_core->num_pm_dev = 0;
> + return;
> + }
> +
> + for (i = 1; i < r5_core->num_pm_dev; i++) {
> + device_link_del(r5_core->pm_dev_core1_link[i]);
> + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> + }
> +
> + kfree(r5_core->pm_dev_core1);
> + r5_core->pm_dev_core1 = NULL;
> + kfree(r5_core->pm_dev_core1_link);
> + r5_core->pm_dev_core1_link = NULL;
> + r5_core->num_pm_dev = 0;
> +}
> +
> +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + struct device *dev = r5_core->dev, *dev2;
> + struct zynqmp_r5_cluster *cluster;
> + struct platform_device *pdev;
> + struct device_node *np;
> + int i, j, num_pm_dev, ret;
> +
> + cluster = dev_get_drvdata(dev->parent);
> +
> + /* get number of power-domains */
> + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> + "#power-domain-cells");
> +
> + if (num_pm_dev <= 0)
> + return -EINVAL;
> +
> + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> + sizeof(struct device *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core0)
> + ret = -ENOMEM;
> +
> + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> + sizeof(struct device_link *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core0_link) {
> + kfree(r5_core->pm_dev_core0);
> + r5_core->pm_dev_core0 = NULL;
> + return -ENOMEM;
> + }
> +
> + r5_core->num_pm_dev = num_pm_dev;
> +
> + /*
> + * start from 2nd entry in power-domains property list as
> + * for zynqmp we only add TCM power domains and not core's power domain.
> + * 1st entry is used to configure r5 operation mode.
> + */
> + for (i = 1; i < r5_core->num_pm_dev; i++) {
> + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> + PTR_ERR(r5_core->pm_dev_core0[i]));
> + ret = -EINVAL;
> + goto fail_add_pm_domains;
> + }
> + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> + r5_core->pm_dev_core0[i],
> + DL_FLAG_STATELESS |
> + DL_FLAG_RPM_ACTIVE |
> + DL_FLAG_PM_RUNTIME);
> + if (!r5_core->pm_dev_core0_link[i]) {
> + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> + r5_core->pm_dev_core0[i] = NULL;
> + ret = -EINVAL;
> + goto fail_add_pm_domains;
> + }
> + }
> +
> + if (cluster->mode == SPLIT_MODE)
> + return 0;
> +
> + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> + sizeof(struct device *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core1) {
> + ret = -ENOMEM;
> + goto fail_add_pm_domains;
> + }
> +
> + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> + sizeof(struct device_link *),
> + GFP_KERNEL);
> + if (!r5_core->pm_dev_core1_link) {
> + kfree(r5_core->pm_dev_core1);
> + r5_core->pm_dev_core1 = NULL;
> + ret = -ENOMEM;
> + goto fail_add_pm_domains;
> + }
> +
> + /* get second core's device to detach its power-domains */
Attach or detach?
> + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> +
> + pdev = of_find_device_by_node(np);
> + if (!pdev) {
> + dev_err(cluster->dev, "core1 platform device not available\n");
> + kfree(r5_core->pm_dev_core1);
> + kfree(r5_core->pm_dev_core1_link);
> + r5_core->pm_dev_core1 = NULL;
> + r5_core->pm_dev_core1_link = NULL;
> + ret = -EINVAL;
> + goto fail_add_pm_domains;
> + }
> +
> + dev2 = &pdev->dev;
> +
> + /* for zynqmp we only add TCM power domains and not core's power domain */
> + for (j = 1; j < r5_core->num_pm_dev; j++) {
> + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> + if (!r5_core->pm_dev_core1[j]) {
> + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> + ret = -EINVAL;
> + goto fail_add_pm_domains_lockstep;
> + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> + goto fail_add_pm_domains_lockstep;
> + }
> +
> + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> + r5_core->pm_dev_core1[j],
> + DL_FLAG_STATELESS |
> + DL_FLAG_RPM_ACTIVE |
> + DL_FLAG_PM_RUNTIME);
> + if (!r5_core->pm_dev_core1_link[j]) {
> + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> + r5_core->pm_dev_core1[j] = NULL;
> + ret = -ENODEV;
> + goto fail_add_pm_domains_lockstep;
> + }
> + }
> +
> +fail_add_pm_domains_lockstep:
> + while (--j >= 0) {
> + device_link_del(r5_core->pm_dev_core1_link[j]);
> + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> + }
> + kfree(r5_core->pm_dev_core1);
> + r5_core->pm_dev_core1 = NULL;
> + kfree(r5_core->pm_dev_core1_link);
> + r5_core->pm_dev_core1_link = NULL;
> +
> +fail_add_pm_domains:
> + while (--i >= 0) {
> + device_link_del(r5_core->pm_dev_core0_link[i]);
> + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> + }
> + kfree(r5_core->pm_dev_core0);
> + r5_core->pm_dev_core0 = NULL;
> + kfree(r5_core->pm_dev_core0_link);
> + r5_core->pm_dev_core0_link = NULL;
> +
> + return ret;
> +}
> +
> /**
> * zynqmp_r5_rproc_prepare()
> * adds carveouts for TCM bank and reserved memory regions
> @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> {
> int ret;
>
> + ret = zynqmp_r5_add_pm_domains(rproc);
> + if (ret) {
> + dev_err(&rproc->dev, "failed to add pm domains\n");
> + return ret;
> + }
> +
> ret = add_tcm_banks(rproc);
> if (ret) {
> dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> - return ret;
> + goto fail_prepare;
> }
>
> ret = add_mem_regions_carveout(rproc);
> if (ret) {
> dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> - return ret;
> + goto fail_prepare;
> }
>
> return 0;
> +
> +fail_prepare:
> + zynqmp_r5_remove_pm_domains(rproc);
> +
> + return ret;
> }
>
> /**
> @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
>
> r5_core = rproc->priv;
>
> + zynqmp_r5_remove_pm_domains(rproc);
> +
> for (i = 0; i < r5_core->tcm_bank_count; i++) {
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> if (zynqmp_pm_release_node(pm_domain_id))
> --
> 2.25.1
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 4/4] remoteproc: zynqmp: parse TCM from device tree
2023-11-17 17:42 ` Tanmay Shah
@ 2023-11-22 17:51 ` Mathieu Poirier
-1 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-11-22 17:51 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Fri, Nov 17, 2023 at 09:42:38AM -0800, Tanmay Shah wrote:
> ZynqMP TCM information is fixed in driver. Now ZynqMP TCM information
> is available in device-tree. Parse TCM information in driver
> as per new bindings.
>
> Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> ---
>
> Changes in v7:
> - move checking of pm_domain_id from previous patch
> - fix mem_bank_data memory allocation
>
> drivers/remoteproc/xlnx_r5_remoteproc.c | 152 ++++++++++++++++++++----
> 1 file changed, 128 insertions(+), 24 deletions(-)
>
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> index 22bccc5075a0..270af73344ef 100644
> --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -75,8 +75,8 @@ struct mbox_info {
> };
>
> /*
> - * Hardcoded TCM bank values. This will be removed once TCM bindings are
> - * accepted for system-dt specifications and upstreamed in linux kernel
> + * Hardcoded TCM bank values. This will stay in driver to maintain backward
> + * compatibility with device-tree that does not have TCM information.
> */
> static const struct mem_bank_data zynqmp_tcm_banks_split[] = {
> {0xffe00000UL, 0x0, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> @@ -587,12 +587,21 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> bank_size = r5_core->tcm_banks[i]->size;
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
>
> - ret = zynqmp_pm_request_node(pm_domain_id,
> - ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> - ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> - if (ret < 0) {
> - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> - goto release_tcm_split;
> + /*
> + * If TCM information is available in device-tree then
> + * in that case, pm domain framework will power on/off TCM.
> + * In that case pm_domain_id is set to 0. If hardcode
> + * bindings from driver is used, then only this driver will
> + * use pm_domain_id.
> + */
> + if (pm_domain_id) {
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> + goto release_tcm_split;
> + }
> }
>
> dev_dbg(dev, "TCM carveout split mode %s addr=%llx, da=0x%x, size=0x%lx",
> @@ -604,7 +613,8 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> bank_name);
> if (!rproc_mem) {
> ret = -ENOMEM;
> - zynqmp_pm_release_node(pm_domain_id);
> + if (pm_domain_id)
> + zynqmp_pm_release_node(pm_domain_id);
> goto release_tcm_split;
> }
>
> @@ -617,7 +627,8 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> /* If failed, Turn off all TCM banks turned on before */
> for (i--; i >= 0; i--) {
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> - zynqmp_pm_release_node(pm_domain_id);
> + if (pm_domain_id)
> + zynqmp_pm_release_node(pm_domain_id);
> }
> return ret;
> }
> @@ -659,13 +670,16 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
>
> /* Turn on each TCM bank individually */
> - ret = zynqmp_pm_request_node(pm_domain_id,
> - ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> - ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> - if (ret < 0) {
> - dev_err(dev, "failed to turn on TCM 0x%x",
> - pm_domain_id);
> - goto release_tcm_lockstep;
> +
> + if (pm_domain_id) {
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM 0x%x",
> + pm_domain_id);
> + goto release_tcm_lockstep;
> + }
> }
>
> bank_size = r5_core->tcm_banks[i]->size;
> @@ -683,7 +697,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> bank_name);
> if (!rproc_mem) {
> ret = -ENOMEM;
> - zynqmp_pm_release_node(pm_domain_id);
> + if (pm_domain_id)
> + zynqmp_pm_release_node(pm_domain_id);
> goto release_tcm_lockstep;
> }
>
> @@ -700,7 +715,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> /* If failed, Turn off all TCM banks turned on before */
> for (i--; i >= 0; i--) {
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> - zynqmp_pm_release_node(pm_domain_id);
> + if (pm_domain_id)
> + zynqmp_pm_release_node(pm_domain_id);
> }
> return ret;
> }
> @@ -931,6 +947,8 @@ static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> }
> }
>
> + return 0;
> +
> fail_add_pm_domains_lockstep:
> while (--j >= 0) {
> device_link_del(r5_core->pm_dev_core1_link[j]);
> @@ -1012,7 +1030,7 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
>
> for (i = 0; i < r5_core->tcm_bank_count; i++) {
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> - if (zynqmp_pm_release_node(pm_domain_id))
> + if (pm_domain_id && zynqmp_pm_release_node(pm_domain_id))
> dev_warn(r5_core->dev,
> "can't turn off TCM bank 0x%x", pm_domain_id);
> }
> @@ -1087,6 +1105,83 @@ static struct zynqmp_r5_core *zynqmp_r5_add_rproc_core(struct device *cdev)
> return ERR_PTR(ret);
> }
>
> +static int zynqmp_r5_get_tcm_node_from_dt(struct zynqmp_r5_cluster *cluster)
> +{
> + struct zynqmp_r5_core *r5_core;
> + int i, j, tcm_bank_count, ret;
> + struct platform_device *cpdev;
> + struct mem_bank_data *tcm;
> + struct device_node *np;
> + struct resource *res;
> + u64 abs_addr, size;
> + struct device *dev;
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = cluster->r5_cores[i];
> + dev = r5_core->dev;
> + np = dev_of_node(dev);
> +
> + /* we have address cell 2 and size cell as 2 */
> + ret = of_property_count_elems_of_size(np, "reg",
> + if (ret <= 0) {
> + dev_err(dev, "can't get reg property err %d\n", ret);
> + return -EINVAL;
> + }
> +
> + tcm_bank_count = ret;
This is useless - please use @tcm_bank_count instead of @ret above.
> +
> + r5_core->tcm_banks = devm_kcalloc(dev, tcm_bank_count,
> + sizeof(struct mem_bank_data *),
> + GFP_KERNEL);
> + if (!r5_core->tcm_banks)
> + ret = -ENOMEM;
> +
> + r5_core->tcm_bank_count = tcm_bank_count;
> + for (j = 0; j < tcm_bank_count; j++) {
> + tcm = devm_kzalloc(dev, sizeof(struct mem_bank_data),
> + GFP_KERNEL);
> + if (!tcm)
> + return -ENOMEM;
> +
> + r5_core->tcm_banks[j] = tcm;
> +
> + /* get tcm address without translation */
> + ret = of_property_read_reg(np, j, &abs_addr, &size);
> + if (ret) {
> + dev_err(dev, "failed to get reg property\n");
> + return ret;
> + }
> +
> + /*
> + * remote processor can address only 32 bits
> + * so convert 64-bits into 32-bits. This will discard
> + * any unwanted upper 32-bits.
> + */
> + tcm->da = (u32)abs_addr;
> + tcm->size = (u32)size;
> +
> + cpdev = to_platform_device(dev);
> + res = platform_get_resource(cpdev, IORESOURCE_MEM, j);
> + if (!res) {
> + dev_err(dev, "failed to get tcm resource\n");
> + return -EINVAL;
> + }
> +
> + tcm->addr = (u32)res->start;
> + tcm->bank_name = (char *)res->name;
> + res = devm_request_mem_region(dev, tcm->addr, tcm->size,
> + tcm->bank_name);
> + if (!res) {
> + dev_err(dev, "failed to request tcm resource\n");
> + return -EINVAL;
> + }
> + }
> + }
> +
> + return 0;
> +}
> +
> /**
> * zynqmp_r5_get_tcm_node()
> * Ideally this function should parse tcm node and store information
> @@ -1165,10 +1260,19 @@ static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster,
> struct zynqmp_r5_core *r5_core;
> int ret, i;
>
> - ret = zynqmp_r5_get_tcm_node(cluster);
> - if (ret < 0) {
> - dev_err(dev, "can't get tcm node, err %d\n", ret);
> - return ret;
> + r5_core = cluster->r5_cores[0];
> + if (of_find_property(r5_core->np, "reg", NULL)) {
> + ret = zynqmp_r5_get_tcm_node_from_dt(cluster);
> + if (ret) {
> + dev_err(dev, "can't get tcm node from dt, err %d\n", ret);
> + return ret;
> + }
> + } else {
> + ret = zynqmp_r5_get_tcm_node(cluster);
> + if (ret < 0) {
> + dev_err(dev, "can't get tcm node, err %d\n", ret);
> + return ret;
> + }
> }
>
> for (i = 0; i < cluster->core_count; i++) {
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 4/4] remoteproc: zynqmp: parse TCM from device tree
@ 2023-11-22 17:51 ` Mathieu Poirier
0 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-11-22 17:51 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Fri, Nov 17, 2023 at 09:42:38AM -0800, Tanmay Shah wrote:
> ZynqMP TCM information is fixed in driver. Now ZynqMP TCM information
> is available in device-tree. Parse TCM information in driver
> as per new bindings.
>
> Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> ---
>
> Changes in v7:
> - move checking of pm_domain_id from previous patch
> - fix mem_bank_data memory allocation
>
> drivers/remoteproc/xlnx_r5_remoteproc.c | 152 ++++++++++++++++++++----
> 1 file changed, 128 insertions(+), 24 deletions(-)
>
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> index 22bccc5075a0..270af73344ef 100644
> --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -75,8 +75,8 @@ struct mbox_info {
> };
>
> /*
> - * Hardcoded TCM bank values. This will be removed once TCM bindings are
> - * accepted for system-dt specifications and upstreamed in linux kernel
> + * Hardcoded TCM bank values. This will stay in driver to maintain backward
> + * compatibility with device-tree that does not have TCM information.
> */
> static const struct mem_bank_data zynqmp_tcm_banks_split[] = {
> {0xffe00000UL, 0x0, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> @@ -587,12 +587,21 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> bank_size = r5_core->tcm_banks[i]->size;
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
>
> - ret = zynqmp_pm_request_node(pm_domain_id,
> - ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> - ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> - if (ret < 0) {
> - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> - goto release_tcm_split;
> + /*
> + * If TCM information is available in device-tree then
> + * in that case, pm domain framework will power on/off TCM.
> + * In that case pm_domain_id is set to 0. If hardcode
> + * bindings from driver is used, then only this driver will
> + * use pm_domain_id.
> + */
> + if (pm_domain_id) {
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> + goto release_tcm_split;
> + }
> }
>
> dev_dbg(dev, "TCM carveout split mode %s addr=%llx, da=0x%x, size=0x%lx",
> @@ -604,7 +613,8 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> bank_name);
> if (!rproc_mem) {
> ret = -ENOMEM;
> - zynqmp_pm_release_node(pm_domain_id);
> + if (pm_domain_id)
> + zynqmp_pm_release_node(pm_domain_id);
> goto release_tcm_split;
> }
>
> @@ -617,7 +627,8 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> /* If failed, Turn off all TCM banks turned on before */
> for (i--; i >= 0; i--) {
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> - zynqmp_pm_release_node(pm_domain_id);
> + if (pm_domain_id)
> + zynqmp_pm_release_node(pm_domain_id);
> }
> return ret;
> }
> @@ -659,13 +670,16 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
>
> /* Turn on each TCM bank individually */
> - ret = zynqmp_pm_request_node(pm_domain_id,
> - ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> - ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> - if (ret < 0) {
> - dev_err(dev, "failed to turn on TCM 0x%x",
> - pm_domain_id);
> - goto release_tcm_lockstep;
> +
> + if (pm_domain_id) {
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM 0x%x",
> + pm_domain_id);
> + goto release_tcm_lockstep;
> + }
> }
>
> bank_size = r5_core->tcm_banks[i]->size;
> @@ -683,7 +697,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> bank_name);
> if (!rproc_mem) {
> ret = -ENOMEM;
> - zynqmp_pm_release_node(pm_domain_id);
> + if (pm_domain_id)
> + zynqmp_pm_release_node(pm_domain_id);
> goto release_tcm_lockstep;
> }
>
> @@ -700,7 +715,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> /* If failed, Turn off all TCM banks turned on before */
> for (i--; i >= 0; i--) {
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> - zynqmp_pm_release_node(pm_domain_id);
> + if (pm_domain_id)
> + zynqmp_pm_release_node(pm_domain_id);
> }
> return ret;
> }
> @@ -931,6 +947,8 @@ static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> }
> }
>
> + return 0;
> +
> fail_add_pm_domains_lockstep:
> while (--j >= 0) {
> device_link_del(r5_core->pm_dev_core1_link[j]);
> @@ -1012,7 +1030,7 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
>
> for (i = 0; i < r5_core->tcm_bank_count; i++) {
> pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> - if (zynqmp_pm_release_node(pm_domain_id))
> + if (pm_domain_id && zynqmp_pm_release_node(pm_domain_id))
> dev_warn(r5_core->dev,
> "can't turn off TCM bank 0x%x", pm_domain_id);
> }
> @@ -1087,6 +1105,83 @@ static struct zynqmp_r5_core *zynqmp_r5_add_rproc_core(struct device *cdev)
> return ERR_PTR(ret);
> }
>
> +static int zynqmp_r5_get_tcm_node_from_dt(struct zynqmp_r5_cluster *cluster)
> +{
> + struct zynqmp_r5_core *r5_core;
> + int i, j, tcm_bank_count, ret;
> + struct platform_device *cpdev;
> + struct mem_bank_data *tcm;
> + struct device_node *np;
> + struct resource *res;
> + u64 abs_addr, size;
> + struct device *dev;
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = cluster->r5_cores[i];
> + dev = r5_core->dev;
> + np = dev_of_node(dev);
> +
> + /* we have address cell 2 and size cell as 2 */
> + ret = of_property_count_elems_of_size(np, "reg",
> + if (ret <= 0) {
> + dev_err(dev, "can't get reg property err %d\n", ret);
> + return -EINVAL;
> + }
> +
> + tcm_bank_count = ret;
This is useless - please use @tcm_bank_count instead of @ret above.
> +
> + r5_core->tcm_banks = devm_kcalloc(dev, tcm_bank_count,
> + sizeof(struct mem_bank_data *),
> + GFP_KERNEL);
> + if (!r5_core->tcm_banks)
> + ret = -ENOMEM;
> +
> + r5_core->tcm_bank_count = tcm_bank_count;
> + for (j = 0; j < tcm_bank_count; j++) {
> + tcm = devm_kzalloc(dev, sizeof(struct mem_bank_data),
> + GFP_KERNEL);
> + if (!tcm)
> + return -ENOMEM;
> +
> + r5_core->tcm_banks[j] = tcm;
> +
> + /* get tcm address without translation */
> + ret = of_property_read_reg(np, j, &abs_addr, &size);
> + if (ret) {
> + dev_err(dev, "failed to get reg property\n");
> + return ret;
> + }
> +
> + /*
> + * remote processor can address only 32 bits
> + * so convert 64-bits into 32-bits. This will discard
> + * any unwanted upper 32-bits.
> + */
> + tcm->da = (u32)abs_addr;
> + tcm->size = (u32)size;
> +
> + cpdev = to_platform_device(dev);
> + res = platform_get_resource(cpdev, IORESOURCE_MEM, j);
> + if (!res) {
> + dev_err(dev, "failed to get tcm resource\n");
> + return -EINVAL;
> + }
> +
> + tcm->addr = (u32)res->start;
> + tcm->bank_name = (char *)res->name;
> + res = devm_request_mem_region(dev, tcm->addr, tcm->size,
> + tcm->bank_name);
> + if (!res) {
> + dev_err(dev, "failed to request tcm resource\n");
> + return -EINVAL;
> + }
> + }
> + }
> +
> + return 0;
> +}
> +
> /**
> * zynqmp_r5_get_tcm_node()
> * Ideally this function should parse tcm node and store information
> @@ -1165,10 +1260,19 @@ static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster,
> struct zynqmp_r5_core *r5_core;
> int ret, i;
>
> - ret = zynqmp_r5_get_tcm_node(cluster);
> - if (ret < 0) {
> - dev_err(dev, "can't get tcm node, err %d\n", ret);
> - return ret;
> + r5_core = cluster->r5_cores[0];
> + if (of_find_property(r5_core->np, "reg", NULL)) {
> + ret = zynqmp_r5_get_tcm_node_from_dt(cluster);
> + if (ret) {
> + dev_err(dev, "can't get tcm node from dt, err %d\n", ret);
> + return ret;
> + }
> + } else {
> + ret = zynqmp_r5_get_tcm_node(cluster);
> + if (ret < 0) {
> + dev_err(dev, "can't get tcm node, err %d\n", ret);
> + return ret;
> + }
> }
>
> for (i = 0; i < cluster->core_count; i++) {
> --
> 2.25.1
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 4/4] remoteproc: zynqmp: parse TCM from device tree
2023-11-22 17:51 ` Mathieu Poirier
@ 2023-11-22 20:27 ` Mathieu Poirier
-1 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-11-22 20:27 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Wed, Nov 22, 2023 at 10:51:08AM -0700, Mathieu Poirier wrote:
> On Fri, Nov 17, 2023 at 09:42:38AM -0800, Tanmay Shah wrote:
> > ZynqMP TCM information is fixed in driver. Now ZynqMP TCM information
> > is available in device-tree. Parse TCM information in driver
> > as per new bindings.
> >
> > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > ---
> >
> > Changes in v7:
> > - move checking of pm_domain_id from previous patch
> > - fix mem_bank_data memory allocation
> >
> > drivers/remoteproc/xlnx_r5_remoteproc.c | 152 ++++++++++++++++++++----
> > 1 file changed, 128 insertions(+), 24 deletions(-)
> >
> > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > index 22bccc5075a0..270af73344ef 100644
> > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > @@ -75,8 +75,8 @@ struct mbox_info {
> > };
> >
> > /*
> > - * Hardcoded TCM bank values. This will be removed once TCM bindings are
> > - * accepted for system-dt specifications and upstreamed in linux kernel
> > + * Hardcoded TCM bank values. This will stay in driver to maintain backward
> > + * compatibility with device-tree that does not have TCM information.
> > */
> > static const struct mem_bank_data zynqmp_tcm_banks_split[] = {
> > {0xffe00000UL, 0x0, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> > @@ -587,12 +587,21 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> > bank_size = r5_core->tcm_banks[i]->size;
> > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> >
> > - ret = zynqmp_pm_request_node(pm_domain_id,
> > - ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > - ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > - if (ret < 0) {
> > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > - goto release_tcm_split;
> > + /*
> > + * If TCM information is available in device-tree then
> > + * in that case, pm domain framework will power on/off TCM.
> > + * In that case pm_domain_id is set to 0. If hardcode
> > + * bindings from driver is used, then only this driver will
> > + * use pm_domain_id.
> > + */
> > + if (pm_domain_id) {
> > + ret = zynqmp_pm_request_node(pm_domain_id,
> > + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > + if (ret < 0) {
> > + dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > + goto release_tcm_split;
> > + }
> > }
> >
> > dev_dbg(dev, "TCM carveout split mode %s addr=%llx, da=0x%x, size=0x%lx",
> > @@ -604,7 +613,8 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> > bank_name);
> > if (!rproc_mem) {
> > ret = -ENOMEM;
> > - zynqmp_pm_release_node(pm_domain_id);
> > + if (pm_domain_id)
> > + zynqmp_pm_release_node(pm_domain_id);
> > goto release_tcm_split;
> > }
> >
> > @@ -617,7 +627,8 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> > /* If failed, Turn off all TCM banks turned on before */
> > for (i--; i >= 0; i--) {
> > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > - zynqmp_pm_release_node(pm_domain_id);
> > + if (pm_domain_id)
> > + zynqmp_pm_release_node(pm_domain_id);
> > }
> > return ret;
> > }
> > @@ -659,13 +670,16 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> >
> > /* Turn on each TCM bank individually */
> > - ret = zynqmp_pm_request_node(pm_domain_id,
> > - ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > - ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > - if (ret < 0) {
> > - dev_err(dev, "failed to turn on TCM 0x%x",
> > - pm_domain_id);
> > - goto release_tcm_lockstep;
> > +
> > + if (pm_domain_id) {
> > + ret = zynqmp_pm_request_node(pm_domain_id,
> > + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > + if (ret < 0) {
> > + dev_err(dev, "failed to turn on TCM 0x%x",
> > + pm_domain_id);
> > + goto release_tcm_lockstep;
> > + }
> > }
> >
> > bank_size = r5_core->tcm_banks[i]->size;
> > @@ -683,7 +697,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > bank_name);
> > if (!rproc_mem) {
> > ret = -ENOMEM;
> > - zynqmp_pm_release_node(pm_domain_id);
> > + if (pm_domain_id)
> > + zynqmp_pm_release_node(pm_domain_id);
> > goto release_tcm_lockstep;
> > }
> >
> > @@ -700,7 +715,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > /* If failed, Turn off all TCM banks turned on before */
> > for (i--; i >= 0; i--) {
> > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > - zynqmp_pm_release_node(pm_domain_id);
> > + if (pm_domain_id)
> > + zynqmp_pm_release_node(pm_domain_id);
> > }
> > return ret;
> > }
> > @@ -931,6 +947,8 @@ static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > }
> > }
> >
> > + return 0;
> > +
> > fail_add_pm_domains_lockstep:
> > while (--j >= 0) {
> > device_link_del(r5_core->pm_dev_core1_link[j]);
> > @@ -1012,7 +1030,7 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> >
> > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > - if (zynqmp_pm_release_node(pm_domain_id))
> > + if (pm_domain_id && zynqmp_pm_release_node(pm_domain_id))
> > dev_warn(r5_core->dev,
> > "can't turn off TCM bank 0x%x", pm_domain_id);
> > }
> > @@ -1087,6 +1105,83 @@ static struct zynqmp_r5_core *zynqmp_r5_add_rproc_core(struct device *cdev)
> > return ERR_PTR(ret);
> > }
> >
> > +static int zynqmp_r5_get_tcm_node_from_dt(struct zynqmp_r5_cluster *cluster)
> > +{
> > + struct zynqmp_r5_core *r5_core;
> > + int i, j, tcm_bank_count, ret;
> > + struct platform_device *cpdev;
> > + struct mem_bank_data *tcm;
> > + struct device_node *np;
> > + struct resource *res;
> > + u64 abs_addr, size;
> > + struct device *dev;
> > +
> > + for (i = 0; i < cluster->core_count; i++) {
> > + r5_core = cluster->r5_cores[i];
> > + dev = r5_core->dev;
> > + np = dev_of_node(dev);
> > +
> > + /* we have address cell 2 and size cell as 2 */
> > + ret = of_property_count_elems_of_size(np, "reg",
> > + if (ret <= 0) {
> > + dev_err(dev, "can't get reg property err %d\n", ret);
> > + return -EINVAL;
> > + }
> > +
> > + tcm_bank_count = ret;
>
> This is useless - please use @tcm_bank_count instead of @ret above.
>
I am done reviewing this set.
Thanks,
Mathieu
> > +
> > + r5_core->tcm_banks = devm_kcalloc(dev, tcm_bank_count,
> > + sizeof(struct mem_bank_data *),
> > + GFP_KERNEL);
> > + if (!r5_core->tcm_banks)
> > + ret = -ENOMEM;
> > +
> > + r5_core->tcm_bank_count = tcm_bank_count;
> > + for (j = 0; j < tcm_bank_count; j++) {
> > + tcm = devm_kzalloc(dev, sizeof(struct mem_bank_data),
> > + GFP_KERNEL);
> > + if (!tcm)
> > + return -ENOMEM;
> > +
> > + r5_core->tcm_banks[j] = tcm;
> > +
> > + /* get tcm address without translation */
> > + ret = of_property_read_reg(np, j, &abs_addr, &size);
> > + if (ret) {
> > + dev_err(dev, "failed to get reg property\n");
> > + return ret;
> > + }
> > +
> > + /*
> > + * remote processor can address only 32 bits
> > + * so convert 64-bits into 32-bits. This will discard
> > + * any unwanted upper 32-bits.
> > + */
> > + tcm->da = (u32)abs_addr;
> > + tcm->size = (u32)size;
> > +
> > + cpdev = to_platform_device(dev);
> > + res = platform_get_resource(cpdev, IORESOURCE_MEM, j);
> > + if (!res) {
> > + dev_err(dev, "failed to get tcm resource\n");
> > + return -EINVAL;
> > + }
> > +
> > + tcm->addr = (u32)res->start;
> > + tcm->bank_name = (char *)res->name;
> > + res = devm_request_mem_region(dev, tcm->addr, tcm->size,
> > + tcm->bank_name);
> > + if (!res) {
> > + dev_err(dev, "failed to request tcm resource\n");
> > + return -EINVAL;
> > + }
> > + }
> > + }
> > +
> > + return 0;
> > +}
> > +
> > /**
> > * zynqmp_r5_get_tcm_node()
> > * Ideally this function should parse tcm node and store information
> > @@ -1165,10 +1260,19 @@ static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster,
> > struct zynqmp_r5_core *r5_core;
> > int ret, i;
> >
> > - ret = zynqmp_r5_get_tcm_node(cluster);
> > - if (ret < 0) {
> > - dev_err(dev, "can't get tcm node, err %d\n", ret);
> > - return ret;
> > + r5_core = cluster->r5_cores[0];
> > + if (of_find_property(r5_core->np, "reg", NULL)) {
> > + ret = zynqmp_r5_get_tcm_node_from_dt(cluster);
> > + if (ret) {
> > + dev_err(dev, "can't get tcm node from dt, err %d\n", ret);
> > + return ret;
> > + }
> > + } else {
> > + ret = zynqmp_r5_get_tcm_node(cluster);
> > + if (ret < 0) {
> > + dev_err(dev, "can't get tcm node, err %d\n", ret);
> > + return ret;
> > + }
> > }
> >
> > for (i = 0; i < cluster->core_count; i++) {
> > --
> > 2.25.1
> >
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 4/4] remoteproc: zynqmp: parse TCM from device tree
@ 2023-11-22 20:27 ` Mathieu Poirier
0 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-11-22 20:27 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Wed, Nov 22, 2023 at 10:51:08AM -0700, Mathieu Poirier wrote:
> On Fri, Nov 17, 2023 at 09:42:38AM -0800, Tanmay Shah wrote:
> > ZynqMP TCM information is fixed in driver. Now ZynqMP TCM information
> > is available in device-tree. Parse TCM information in driver
> > as per new bindings.
> >
> > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > ---
> >
> > Changes in v7:
> > - move checking of pm_domain_id from previous patch
> > - fix mem_bank_data memory allocation
> >
> > drivers/remoteproc/xlnx_r5_remoteproc.c | 152 ++++++++++++++++++++----
> > 1 file changed, 128 insertions(+), 24 deletions(-)
> >
> > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > index 22bccc5075a0..270af73344ef 100644
> > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > @@ -75,8 +75,8 @@ struct mbox_info {
> > };
> >
> > /*
> > - * Hardcoded TCM bank values. This will be removed once TCM bindings are
> > - * accepted for system-dt specifications and upstreamed in linux kernel
> > + * Hardcoded TCM bank values. This will stay in driver to maintain backward
> > + * compatibility with device-tree that does not have TCM information.
> > */
> > static const struct mem_bank_data zynqmp_tcm_banks_split[] = {
> > {0xffe00000UL, 0x0, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> > @@ -587,12 +587,21 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> > bank_size = r5_core->tcm_banks[i]->size;
> > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> >
> > - ret = zynqmp_pm_request_node(pm_domain_id,
> > - ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > - ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > - if (ret < 0) {
> > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > - goto release_tcm_split;
> > + /*
> > + * If TCM information is available in device-tree then
> > + * in that case, pm domain framework will power on/off TCM.
> > + * In that case pm_domain_id is set to 0. If hardcode
> > + * bindings from driver is used, then only this driver will
> > + * use pm_domain_id.
> > + */
> > + if (pm_domain_id) {
> > + ret = zynqmp_pm_request_node(pm_domain_id,
> > + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > + if (ret < 0) {
> > + dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > + goto release_tcm_split;
> > + }
> > }
> >
> > dev_dbg(dev, "TCM carveout split mode %s addr=%llx, da=0x%x, size=0x%lx",
> > @@ -604,7 +613,8 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> > bank_name);
> > if (!rproc_mem) {
> > ret = -ENOMEM;
> > - zynqmp_pm_release_node(pm_domain_id);
> > + if (pm_domain_id)
> > + zynqmp_pm_release_node(pm_domain_id);
> > goto release_tcm_split;
> > }
> >
> > @@ -617,7 +627,8 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> > /* If failed, Turn off all TCM banks turned on before */
> > for (i--; i >= 0; i--) {
> > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > - zynqmp_pm_release_node(pm_domain_id);
> > + if (pm_domain_id)
> > + zynqmp_pm_release_node(pm_domain_id);
> > }
> > return ret;
> > }
> > @@ -659,13 +670,16 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> >
> > /* Turn on each TCM bank individually */
> > - ret = zynqmp_pm_request_node(pm_domain_id,
> > - ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > - ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > - if (ret < 0) {
> > - dev_err(dev, "failed to turn on TCM 0x%x",
> > - pm_domain_id);
> > - goto release_tcm_lockstep;
> > +
> > + if (pm_domain_id) {
> > + ret = zynqmp_pm_request_node(pm_domain_id,
> > + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > + if (ret < 0) {
> > + dev_err(dev, "failed to turn on TCM 0x%x",
> > + pm_domain_id);
> > + goto release_tcm_lockstep;
> > + }
> > }
> >
> > bank_size = r5_core->tcm_banks[i]->size;
> > @@ -683,7 +697,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > bank_name);
> > if (!rproc_mem) {
> > ret = -ENOMEM;
> > - zynqmp_pm_release_node(pm_domain_id);
> > + if (pm_domain_id)
> > + zynqmp_pm_release_node(pm_domain_id);
> > goto release_tcm_lockstep;
> > }
> >
> > @@ -700,7 +715,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > /* If failed, Turn off all TCM banks turned on before */
> > for (i--; i >= 0; i--) {
> > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > - zynqmp_pm_release_node(pm_domain_id);
> > + if (pm_domain_id)
> > + zynqmp_pm_release_node(pm_domain_id);
> > }
> > return ret;
> > }
> > @@ -931,6 +947,8 @@ static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > }
> > }
> >
> > + return 0;
> > +
> > fail_add_pm_domains_lockstep:
> > while (--j >= 0) {
> > device_link_del(r5_core->pm_dev_core1_link[j]);
> > @@ -1012,7 +1030,7 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> >
> > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > - if (zynqmp_pm_release_node(pm_domain_id))
> > + if (pm_domain_id && zynqmp_pm_release_node(pm_domain_id))
> > dev_warn(r5_core->dev,
> > "can't turn off TCM bank 0x%x", pm_domain_id);
> > }
> > @@ -1087,6 +1105,83 @@ static struct zynqmp_r5_core *zynqmp_r5_add_rproc_core(struct device *cdev)
> > return ERR_PTR(ret);
> > }
> >
> > +static int zynqmp_r5_get_tcm_node_from_dt(struct zynqmp_r5_cluster *cluster)
> > +{
> > + struct zynqmp_r5_core *r5_core;
> > + int i, j, tcm_bank_count, ret;
> > + struct platform_device *cpdev;
> > + struct mem_bank_data *tcm;
> > + struct device_node *np;
> > + struct resource *res;
> > + u64 abs_addr, size;
> > + struct device *dev;
> > +
> > + for (i = 0; i < cluster->core_count; i++) {
> > + r5_core = cluster->r5_cores[i];
> > + dev = r5_core->dev;
> > + np = dev_of_node(dev);
> > +
> > + /* we have address cell 2 and size cell as 2 */
> > + ret = of_property_count_elems_of_size(np, "reg",
> > + if (ret <= 0) {
> > + dev_err(dev, "can't get reg property err %d\n", ret);
> > + return -EINVAL;
> > + }
> > +
> > + tcm_bank_count = ret;
>
> This is useless - please use @tcm_bank_count instead of @ret above.
>
I am done reviewing this set.
Thanks,
Mathieu
> > +
> > + r5_core->tcm_banks = devm_kcalloc(dev, tcm_bank_count,
> > + sizeof(struct mem_bank_data *),
> > + GFP_KERNEL);
> > + if (!r5_core->tcm_banks)
> > + ret = -ENOMEM;
> > +
> > + r5_core->tcm_bank_count = tcm_bank_count;
> > + for (j = 0; j < tcm_bank_count; j++) {
> > + tcm = devm_kzalloc(dev, sizeof(struct mem_bank_data),
> > + GFP_KERNEL);
> > + if (!tcm)
> > + return -ENOMEM;
> > +
> > + r5_core->tcm_banks[j] = tcm;
> > +
> > + /* get tcm address without translation */
> > + ret = of_property_read_reg(np, j, &abs_addr, &size);
> > + if (ret) {
> > + dev_err(dev, "failed to get reg property\n");
> > + return ret;
> > + }
> > +
> > + /*
> > + * remote processor can address only 32 bits
> > + * so convert 64-bits into 32-bits. This will discard
> > + * any unwanted upper 32-bits.
> > + */
> > + tcm->da = (u32)abs_addr;
> > + tcm->size = (u32)size;
> > +
> > + cpdev = to_platform_device(dev);
> > + res = platform_get_resource(cpdev, IORESOURCE_MEM, j);
> > + if (!res) {
> > + dev_err(dev, "failed to get tcm resource\n");
> > + return -EINVAL;
> > + }
> > +
> > + tcm->addr = (u32)res->start;
> > + tcm->bank_name = (char *)res->name;
> > + res = devm_request_mem_region(dev, tcm->addr, tcm->size,
> > + tcm->bank_name);
> > + if (!res) {
> > + dev_err(dev, "failed to request tcm resource\n");
> > + return -EINVAL;
> > + }
> > + }
> > + }
> > +
> > + return 0;
> > +}
> > +
> > /**
> > * zynqmp_r5_get_tcm_node()
> > * Ideally this function should parse tcm node and store information
> > @@ -1165,10 +1260,19 @@ static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster,
> > struct zynqmp_r5_core *r5_core;
> > int ret, i;
> >
> > - ret = zynqmp_r5_get_tcm_node(cluster);
> > - if (ret < 0) {
> > - dev_err(dev, "can't get tcm node, err %d\n", ret);
> > - return ret;
> > + r5_core = cluster->r5_cores[0];
> > + if (of_find_property(r5_core->np, "reg", NULL)) {
> > + ret = zynqmp_r5_get_tcm_node_from_dt(cluster);
> > + if (ret) {
> > + dev_err(dev, "can't get tcm node from dt, err %d\n", ret);
> > + return ret;
> > + }
> > + } else {
> > + ret = zynqmp_r5_get_tcm_node(cluster);
> > + if (ret < 0) {
> > + dev_err(dev, "can't get tcm node, err %d\n", ret);
> > + return ret;
> > + }
> > }
> >
> > for (i = 0; i < cluster->core_count; i++) {
> > --
> > 2.25.1
> >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-11-21 22:59 ` Mathieu Poirier
@ 2023-11-22 21:00 ` Tanmay Shah
-1 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-22 21:00 UTC (permalink / raw)
To: Mathieu Poirier
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
Hi Mathieu,
Please find my comments below.
On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> Hi,
>
> On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > Use TCM pm domains extracted from device-tree
> > to power on/off TCM using general pm domain framework.
> >
> > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > ---
> >
> > Changes in v7:
> > - %s/pm_dev1/pm_dev_core0/r
> > - %s/pm_dev_link1/pm_dev_core0_link/r
> > - %s/pm_dev2/pm_dev_core1/r
> > - %s/pm_dev_link2/pm_dev_core1_link/r
> > - remove pm_domain_id check to move next patch
> > - add comment about how 1st entry in pm domain list is used
> > - fix loop when jump to fail_add_pm_domains loop
> >
> > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > 1 file changed, 212 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > index 4395edea9a64..22bccc5075a0 100644
> > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > @@ -16,6 +16,7 @@
> > #include <linux/of_reserved_mem.h>
> > #include <linux/platform_device.h>
> > #include <linux/remoteproc.h>
> > +#include <linux/pm_domain.h>
> >
> > #include "remoteproc_internal.h"
> >
> > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > * @rproc: rproc handle
> > * @pm_domain_id: RPU CPU power domain id
> > * @ipi: pointer to mailbox information
> > + * @num_pm_dev: number of tcm pm domain devices for this core
> > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > + * @pm_dev_core0_link: pm domain device links after registration
> > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > + * registration
> > */
> > struct zynqmp_r5_core {
> > struct device *dev;
> > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > struct rproc *rproc;
> > u32 pm_domain_id;
> > struct mbox_info *ipi;
> > + int num_pm_dev;
> > + struct device **pm_dev_core0;
> > + struct device_link **pm_dev_core0_link;
> > + struct device **pm_dev_core1;
> > + struct device_link **pm_dev_core1_link;
> > };
> >
> > /**
> > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > if (ret < 0) {
> > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > + dev_err(dev, "failed to turn on TCM 0x%x",
> > + pm_domain_id);
>
> Spurious change, you should have caught that.
Ack, need to observe changes more closely before sending them.
>
> > goto release_tcm_lockstep;
> > }
> >
> > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > return ret;
> > }
> >
> > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > +{
> > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > + struct device *dev = r5_core->dev;
> > + struct zynqmp_r5_cluster *cluster;
> > + int i;
> > +
> > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > +
> > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > + }
> > +
> > + kfree(r5_core->pm_dev_core0);
> > + r5_core->pm_dev_core0 = NULL;
> > + kfree(r5_core->pm_dev_core0_link);
> > + r5_core->pm_dev_core0_link = NULL;
> > +
> > + if (cluster->mode == SPLIT_MODE) {
> > + r5_core->num_pm_dev = 0;
> > + return;
> > + }
> > +
> > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > + }
> > +
> > + kfree(r5_core->pm_dev_core1);
> > + r5_core->pm_dev_core1 = NULL;
> > + kfree(r5_core->pm_dev_core1_link);
> > + r5_core->pm_dev_core1_link = NULL;
> > + r5_core->num_pm_dev = 0;
> > +}
> > +
> > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > +{
> > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > + struct device *dev = r5_core->dev, *dev2;
> > + struct zynqmp_r5_cluster *cluster;
> > + struct platform_device *pdev;
> > + struct device_node *np;
> > + int i, j, num_pm_dev, ret;
> > +
> > + cluster = dev_get_drvdata(dev->parent);
> > +
> > + /* get number of power-domains */
> > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > + "#power-domain-cells");
> > +
> > + if (num_pm_dev <= 0)
> > + return -EINVAL;
> > +
> > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > + sizeof(struct device *),
> > + GFP_KERNEL);
> > + if (!r5_core->pm_dev_core0)
> > + ret = -ENOMEM;
> > +
> > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > + sizeof(struct device_link *),
> > + GFP_KERNEL);
> > + if (!r5_core->pm_dev_core0_link) {
> > + kfree(r5_core->pm_dev_core0);
> > + r5_core->pm_dev_core0 = NULL;
> > + return -ENOMEM;
> > + }
> > +
> > + r5_core->num_pm_dev = num_pm_dev;
> > +
> > + /*
> > + * start from 2nd entry in power-domains property list as
> > + * for zynqmp we only add TCM power domains and not core's power domain.
> > + * 1st entry is used to configure r5 operation mode.
>
> You are still not saying _where_ ->pm_dev_core0[0] gets added.
So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
zynqmp_pm_request_wake during rproc_start callback. I will document this in next
revision. For new platforms pm_dev_core0[0] will be added in future.
I hope this meets expectations.
>
> > + */
> > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
>
> Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> code are used in the loop for the lockstep mode. Please pick one heuristic and
> stick with it. I have no preference on which one.
Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
>
> > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > + ret = -EINVAL;
> > + goto fail_add_pm_domains;
> > + }
> > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > + r5_core->pm_dev_core0[i],
> > + DL_FLAG_STATELESS |
> > + DL_FLAG_RPM_ACTIVE |
> > + DL_FLAG_PM_RUNTIME);
> > + if (!r5_core->pm_dev_core0_link[i]) {
> > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > + r5_core->pm_dev_core0[i] = NULL;
> > + ret = -EINVAL;
> > + goto fail_add_pm_domains;
> > + }
> > + }
> > +
> > + if (cluster->mode == SPLIT_MODE)
> > + return 0;
> > +
> > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > + sizeof(struct device *),
> > + GFP_KERNEL);
> > + if (!r5_core->pm_dev_core1) {
> > + ret = -ENOMEM;
> > + goto fail_add_pm_domains;
> > + }
> > +
> > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > + sizeof(struct device_link *),
> > + GFP_KERNEL);
> > + if (!r5_core->pm_dev_core1_link) {
> > + kfree(r5_core->pm_dev_core1);
> > + r5_core->pm_dev_core1 = NULL;
> > + ret = -ENOMEM;
> > + goto fail_add_pm_domains;
> > + }
> > +
> > + /* get second core's device to detach its power-domains */
> > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > +
> > + pdev = of_find_device_by_node(np);
> > + if (!pdev) {
> > + dev_err(cluster->dev, "core1 platform device not available\n");
> > + kfree(r5_core->pm_dev_core1);
> > + kfree(r5_core->pm_dev_core1_link);
> > + r5_core->pm_dev_core1 = NULL;
> > + r5_core->pm_dev_core1_link = NULL;
> > + ret = -EINVAL;
> > + goto fail_add_pm_domains;
> > + }
> > +
> > + dev2 = &pdev->dev;
> > +
> > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > + if (!r5_core->pm_dev_core1[j]) {
> > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > + ret = -EINVAL;
> > + goto fail_add_pm_domains_lockstep;
> > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > + goto fail_add_pm_domains_lockstep;
> > + }
> > +
> > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > + r5_core->pm_dev_core1[j],
> > + DL_FLAG_STATELESS |
> > + DL_FLAG_RPM_ACTIVE |
> > + DL_FLAG_PM_RUNTIME);
> > + if (!r5_core->pm_dev_core1_link[j]) {
> > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > + r5_core->pm_dev_core1[j] = NULL;
> > + ret = -ENODEV;
> > + goto fail_add_pm_domains_lockstep;
> > + }
> > + }
> > +
> > +fail_add_pm_domains_lockstep:
> > + while (--j >= 0) {
> > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > + }
> > + kfree(r5_core->pm_dev_core1);
> > + r5_core->pm_dev_core1 = NULL;
> > + kfree(r5_core->pm_dev_core1_link);
> > + r5_core->pm_dev_core1_link = NULL;
> > +
> > +fail_add_pm_domains:
> > + while (--i >= 0) {
> > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > + }
> > + kfree(r5_core->pm_dev_core0);
> > + r5_core->pm_dev_core0 = NULL;
> > + kfree(r5_core->pm_dev_core0_link);
> > + r5_core->pm_dev_core0_link = NULL;
> > +
>
> The error path is much cleaner and readable now.
>
> I will continue tomorrow.
>
> Mathieu
>
> > + return ret;
> > +}
> > +
> > /**
> > * zynqmp_r5_rproc_prepare()
> > * adds carveouts for TCM bank and reserved memory regions
> > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > {
> > int ret;
> >
> > + ret = zynqmp_r5_add_pm_domains(rproc);
> > + if (ret) {
> > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > + return ret;
> > + }
> > +
> > ret = add_tcm_banks(rproc);
> > if (ret) {
> > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > - return ret;
> > + goto fail_prepare;
> > }
> >
> > ret = add_mem_regions_carveout(rproc);
> > if (ret) {
> > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > - return ret;
> > + goto fail_prepare;
> > }
> >
> > return 0;
> > +
> > +fail_prepare:
> > + zynqmp_r5_remove_pm_domains(rproc);
> > +
> > + return ret;
> > }
> >
> > /**
> > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> >
> > r5_core = rproc->priv;
> >
> > + zynqmp_r5_remove_pm_domains(rproc);
> > +
> > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > if (zynqmp_pm_release_node(pm_domain_id))
> > --
> > 2.25.1
> >
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-11-22 21:00 ` Tanmay Shah
0 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-22 21:00 UTC (permalink / raw)
To: Mathieu Poirier
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
Hi Mathieu,
Please find my comments below.
On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> Hi,
>
> On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > Use TCM pm domains extracted from device-tree
> > to power on/off TCM using general pm domain framework.
> >
> > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > ---
> >
> > Changes in v7:
> > - %s/pm_dev1/pm_dev_core0/r
> > - %s/pm_dev_link1/pm_dev_core0_link/r
> > - %s/pm_dev2/pm_dev_core1/r
> > - %s/pm_dev_link2/pm_dev_core1_link/r
> > - remove pm_domain_id check to move next patch
> > - add comment about how 1st entry in pm domain list is used
> > - fix loop when jump to fail_add_pm_domains loop
> >
> > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > 1 file changed, 212 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > index 4395edea9a64..22bccc5075a0 100644
> > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > @@ -16,6 +16,7 @@
> > #include <linux/of_reserved_mem.h>
> > #include <linux/platform_device.h>
> > #include <linux/remoteproc.h>
> > +#include <linux/pm_domain.h>
> >
> > #include "remoteproc_internal.h"
> >
> > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > * @rproc: rproc handle
> > * @pm_domain_id: RPU CPU power domain id
> > * @ipi: pointer to mailbox information
> > + * @num_pm_dev: number of tcm pm domain devices for this core
> > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > + * @pm_dev_core0_link: pm domain device links after registration
> > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > + * registration
> > */
> > struct zynqmp_r5_core {
> > struct device *dev;
> > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > struct rproc *rproc;
> > u32 pm_domain_id;
> > struct mbox_info *ipi;
> > + int num_pm_dev;
> > + struct device **pm_dev_core0;
> > + struct device_link **pm_dev_core0_link;
> > + struct device **pm_dev_core1;
> > + struct device_link **pm_dev_core1_link;
> > };
> >
> > /**
> > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > if (ret < 0) {
> > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > + dev_err(dev, "failed to turn on TCM 0x%x",
> > + pm_domain_id);
>
> Spurious change, you should have caught that.
Ack, need to observe changes more closely before sending them.
>
> > goto release_tcm_lockstep;
> > }
> >
> > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > return ret;
> > }
> >
> > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > +{
> > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > + struct device *dev = r5_core->dev;
> > + struct zynqmp_r5_cluster *cluster;
> > + int i;
> > +
> > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > +
> > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > + }
> > +
> > + kfree(r5_core->pm_dev_core0);
> > + r5_core->pm_dev_core0 = NULL;
> > + kfree(r5_core->pm_dev_core0_link);
> > + r5_core->pm_dev_core0_link = NULL;
> > +
> > + if (cluster->mode == SPLIT_MODE) {
> > + r5_core->num_pm_dev = 0;
> > + return;
> > + }
> > +
> > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > + }
> > +
> > + kfree(r5_core->pm_dev_core1);
> > + r5_core->pm_dev_core1 = NULL;
> > + kfree(r5_core->pm_dev_core1_link);
> > + r5_core->pm_dev_core1_link = NULL;
> > + r5_core->num_pm_dev = 0;
> > +}
> > +
> > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > +{
> > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > + struct device *dev = r5_core->dev, *dev2;
> > + struct zynqmp_r5_cluster *cluster;
> > + struct platform_device *pdev;
> > + struct device_node *np;
> > + int i, j, num_pm_dev, ret;
> > +
> > + cluster = dev_get_drvdata(dev->parent);
> > +
> > + /* get number of power-domains */
> > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > + "#power-domain-cells");
> > +
> > + if (num_pm_dev <= 0)
> > + return -EINVAL;
> > +
> > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > + sizeof(struct device *),
> > + GFP_KERNEL);
> > + if (!r5_core->pm_dev_core0)
> > + ret = -ENOMEM;
> > +
> > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > + sizeof(struct device_link *),
> > + GFP_KERNEL);
> > + if (!r5_core->pm_dev_core0_link) {
> > + kfree(r5_core->pm_dev_core0);
> > + r5_core->pm_dev_core0 = NULL;
> > + return -ENOMEM;
> > + }
> > +
> > + r5_core->num_pm_dev = num_pm_dev;
> > +
> > + /*
> > + * start from 2nd entry in power-domains property list as
> > + * for zynqmp we only add TCM power domains and not core's power domain.
> > + * 1st entry is used to configure r5 operation mode.
>
> You are still not saying _where_ ->pm_dev_core0[0] gets added.
So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
zynqmp_pm_request_wake during rproc_start callback. I will document this in next
revision. For new platforms pm_dev_core0[0] will be added in future.
I hope this meets expectations.
>
> > + */
> > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
>
> Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> code are used in the loop for the lockstep mode. Please pick one heuristic and
> stick with it. I have no preference on which one.
Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
>
> > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > + ret = -EINVAL;
> > + goto fail_add_pm_domains;
> > + }
> > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > + r5_core->pm_dev_core0[i],
> > + DL_FLAG_STATELESS |
> > + DL_FLAG_RPM_ACTIVE |
> > + DL_FLAG_PM_RUNTIME);
> > + if (!r5_core->pm_dev_core0_link[i]) {
> > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > + r5_core->pm_dev_core0[i] = NULL;
> > + ret = -EINVAL;
> > + goto fail_add_pm_domains;
> > + }
> > + }
> > +
> > + if (cluster->mode == SPLIT_MODE)
> > + return 0;
> > +
> > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > + sizeof(struct device *),
> > + GFP_KERNEL);
> > + if (!r5_core->pm_dev_core1) {
> > + ret = -ENOMEM;
> > + goto fail_add_pm_domains;
> > + }
> > +
> > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > + sizeof(struct device_link *),
> > + GFP_KERNEL);
> > + if (!r5_core->pm_dev_core1_link) {
> > + kfree(r5_core->pm_dev_core1);
> > + r5_core->pm_dev_core1 = NULL;
> > + ret = -ENOMEM;
> > + goto fail_add_pm_domains;
> > + }
> > +
> > + /* get second core's device to detach its power-domains */
> > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > +
> > + pdev = of_find_device_by_node(np);
> > + if (!pdev) {
> > + dev_err(cluster->dev, "core1 platform device not available\n");
> > + kfree(r5_core->pm_dev_core1);
> > + kfree(r5_core->pm_dev_core1_link);
> > + r5_core->pm_dev_core1 = NULL;
> > + r5_core->pm_dev_core1_link = NULL;
> > + ret = -EINVAL;
> > + goto fail_add_pm_domains;
> > + }
> > +
> > + dev2 = &pdev->dev;
> > +
> > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > + if (!r5_core->pm_dev_core1[j]) {
> > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > + ret = -EINVAL;
> > + goto fail_add_pm_domains_lockstep;
> > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > + goto fail_add_pm_domains_lockstep;
> > + }
> > +
> > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > + r5_core->pm_dev_core1[j],
> > + DL_FLAG_STATELESS |
> > + DL_FLAG_RPM_ACTIVE |
> > + DL_FLAG_PM_RUNTIME);
> > + if (!r5_core->pm_dev_core1_link[j]) {
> > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > + r5_core->pm_dev_core1[j] = NULL;
> > + ret = -ENODEV;
> > + goto fail_add_pm_domains_lockstep;
> > + }
> > + }
> > +
> > +fail_add_pm_domains_lockstep:
> > + while (--j >= 0) {
> > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > + }
> > + kfree(r5_core->pm_dev_core1);
> > + r5_core->pm_dev_core1 = NULL;
> > + kfree(r5_core->pm_dev_core1_link);
> > + r5_core->pm_dev_core1_link = NULL;
> > +
> > +fail_add_pm_domains:
> > + while (--i >= 0) {
> > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > + }
> > + kfree(r5_core->pm_dev_core0);
> > + r5_core->pm_dev_core0 = NULL;
> > + kfree(r5_core->pm_dev_core0_link);
> > + r5_core->pm_dev_core0_link = NULL;
> > +
>
> The error path is much cleaner and readable now.
>
> I will continue tomorrow.
>
> Mathieu
>
> > + return ret;
> > +}
> > +
> > /**
> > * zynqmp_r5_rproc_prepare()
> > * adds carveouts for TCM bank and reserved memory regions
> > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > {
> > int ret;
> >
> > + ret = zynqmp_r5_add_pm_domains(rproc);
> > + if (ret) {
> > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > + return ret;
> > + }
> > +
> > ret = add_tcm_banks(rproc);
> > if (ret) {
> > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > - return ret;
> > + goto fail_prepare;
> > }
> >
> > ret = add_mem_regions_carveout(rproc);
> > if (ret) {
> > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > - return ret;
> > + goto fail_prepare;
> > }
> >
> > return 0;
> > +
> > +fail_prepare:
> > + zynqmp_r5_remove_pm_domains(rproc);
> > +
> > + return ret;
> > }
> >
> > /**
> > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> >
> > r5_core = rproc->priv;
> >
> > + zynqmp_r5_remove_pm_domains(rproc);
> > +
> > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > if (zynqmp_pm_release_node(pm_domain_id))
> > --
> > 2.25.1
> >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-11-22 21:00 ` Tanmay Shah
@ 2023-11-23 18:11 ` Mathieu Poirier
-1 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-11-23 18:11 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> Hi Mathieu,
>
> Please find my comments below.
>
> On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > Hi,
> >
> > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > Use TCM pm domains extracted from device-tree
> > > to power on/off TCM using general pm domain framework.
> > >
> > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > ---
> > >
> > > Changes in v7:
> > > - %s/pm_dev1/pm_dev_core0/r
> > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > - %s/pm_dev2/pm_dev_core1/r
> > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > - remove pm_domain_id check to move next patch
> > > - add comment about how 1st entry in pm domain list is used
> > > - fix loop when jump to fail_add_pm_domains loop
> > >
> > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > index 4395edea9a64..22bccc5075a0 100644
> > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > @@ -16,6 +16,7 @@
> > > #include <linux/of_reserved_mem.h>
> > > #include <linux/platform_device.h>
> > > #include <linux/remoteproc.h>
> > > +#include <linux/pm_domain.h>
> > >
> > > #include "remoteproc_internal.h"
> > >
> > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > * @rproc: rproc handle
> > > * @pm_domain_id: RPU CPU power domain id
> > > * @ipi: pointer to mailbox information
> > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > + * @pm_dev_core0_link: pm domain device links after registration
> > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > + * registration
> > > */
> > > struct zynqmp_r5_core {
> > > struct device *dev;
> > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > struct rproc *rproc;
> > > u32 pm_domain_id;
> > > struct mbox_info *ipi;
> > > + int num_pm_dev;
> > > + struct device **pm_dev_core0;
> > > + struct device_link **pm_dev_core0_link;
> > > + struct device **pm_dev_core1;
> > > + struct device_link **pm_dev_core1_link;
> > > };
> > >
> > > /**
> > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > if (ret < 0) {
> > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > + pm_domain_id);
> >
> > Spurious change, you should have caught that.
>
> Ack, need to observe changes more closely before sending them.
>
> >
> > > goto release_tcm_lockstep;
> > > }
> > >
> > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > return ret;
> > > }
> > >
> > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > +{
> > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > + struct device *dev = r5_core->dev;
> > > + struct zynqmp_r5_cluster *cluster;
> > > + int i;
> > > +
> > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > +
> > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > + }
> > > +
> > > + kfree(r5_core->pm_dev_core0);
> > > + r5_core->pm_dev_core0 = NULL;
> > > + kfree(r5_core->pm_dev_core0_link);
> > > + r5_core->pm_dev_core0_link = NULL;
> > > +
> > > + if (cluster->mode == SPLIT_MODE) {
> > > + r5_core->num_pm_dev = 0;
> > > + return;
> > > + }
> > > +
> > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > + }
> > > +
> > > + kfree(r5_core->pm_dev_core1);
> > > + r5_core->pm_dev_core1 = NULL;
> > > + kfree(r5_core->pm_dev_core1_link);
> > > + r5_core->pm_dev_core1_link = NULL;
> > > + r5_core->num_pm_dev = 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > +{
> > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > + struct device *dev = r5_core->dev, *dev2;
> > > + struct zynqmp_r5_cluster *cluster;
> > > + struct platform_device *pdev;
> > > + struct device_node *np;
> > > + int i, j, num_pm_dev, ret;
> > > +
> > > + cluster = dev_get_drvdata(dev->parent);
> > > +
> > > + /* get number of power-domains */
> > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > + "#power-domain-cells");
> > > +
> > > + if (num_pm_dev <= 0)
> > > + return -EINVAL;
> > > +
> > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > + sizeof(struct device *),
> > > + GFP_KERNEL);
> > > + if (!r5_core->pm_dev_core0)
> > > + ret = -ENOMEM;
> > > +
> > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > + sizeof(struct device_link *),
> > > + GFP_KERNEL);
> > > + if (!r5_core->pm_dev_core0_link) {
> > > + kfree(r5_core->pm_dev_core0);
> > > + r5_core->pm_dev_core0 = NULL;
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + r5_core->num_pm_dev = num_pm_dev;
> > > +
> > > + /*
> > > + * start from 2nd entry in power-domains property list as
> > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > + * 1st entry is used to configure r5 operation mode.
> >
> > You are still not saying _where_ ->pm_dev_core0[0] gets added.
>
> So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
>
> zynqmp_pm_request_wake during rproc_start callback. I will document this in next
>
That is exactly what I am looking for. That way people don't have to go through
the entire driver trying to figure out what is happening with pm_dev_core[0].
I'm also not sure about the power-up order. Logically the TCMs should be
powered up before the R5 in order to put code in them. The R5s are powered in
zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
TCMs are powered - can you expand on that?
> revision. For new platforms pm_dev_core0[0] will be added in future.
Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
future"?
>
> I hope this meets expectations.
>
>
> >
> > > + */
> > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> >
> > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > stick with it. I have no preference on which one.
>
> Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
>
>
> >
> > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > + ret = -EINVAL;
> > > + goto fail_add_pm_domains;
> > > + }
> > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > + r5_core->pm_dev_core0[i],
> > > + DL_FLAG_STATELESS |
> > > + DL_FLAG_RPM_ACTIVE |
> > > + DL_FLAG_PM_RUNTIME);
> > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > + r5_core->pm_dev_core0[i] = NULL;
> > > + ret = -EINVAL;
> > > + goto fail_add_pm_domains;
> > > + }
> > > + }
> > > +
> > > + if (cluster->mode == SPLIT_MODE)
> > > + return 0;
> > > +
> > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > + sizeof(struct device *),
> > > + GFP_KERNEL);
> > > + if (!r5_core->pm_dev_core1) {
> > > + ret = -ENOMEM;
> > > + goto fail_add_pm_domains;
> > > + }
> > > +
> > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > + sizeof(struct device_link *),
> > > + GFP_KERNEL);
> > > + if (!r5_core->pm_dev_core1_link) {
> > > + kfree(r5_core->pm_dev_core1);
> > > + r5_core->pm_dev_core1 = NULL;
> > > + ret = -ENOMEM;
> > > + goto fail_add_pm_domains;
> > > + }
> > > +
> > > + /* get second core's device to detach its power-domains */
> > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > +
> > > + pdev = of_find_device_by_node(np);
> > > + if (!pdev) {
> > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > + kfree(r5_core->pm_dev_core1);
> > > + kfree(r5_core->pm_dev_core1_link);
> > > + r5_core->pm_dev_core1 = NULL;
> > > + r5_core->pm_dev_core1_link = NULL;
> > > + ret = -EINVAL;
> > > + goto fail_add_pm_domains;
> > > + }
> > > +
> > > + dev2 = &pdev->dev;
> > > +
> > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > + if (!r5_core->pm_dev_core1[j]) {
> > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > + ret = -EINVAL;
> > > + goto fail_add_pm_domains_lockstep;
> > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > + goto fail_add_pm_domains_lockstep;
> > > + }
> > > +
> > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > + r5_core->pm_dev_core1[j],
> > > + DL_FLAG_STATELESS |
> > > + DL_FLAG_RPM_ACTIVE |
> > > + DL_FLAG_PM_RUNTIME);
> > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > + r5_core->pm_dev_core1[j] = NULL;
> > > + ret = -ENODEV;
> > > + goto fail_add_pm_domains_lockstep;
> > > + }
> > > + }
> > > +
> > > +fail_add_pm_domains_lockstep:
> > > + while (--j >= 0) {
> > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > + }
> > > + kfree(r5_core->pm_dev_core1);
> > > + r5_core->pm_dev_core1 = NULL;
> > > + kfree(r5_core->pm_dev_core1_link);
> > > + r5_core->pm_dev_core1_link = NULL;
> > > +
> > > +fail_add_pm_domains:
> > > + while (--i >= 0) {
> > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > + }
> > > + kfree(r5_core->pm_dev_core0);
> > > + r5_core->pm_dev_core0 = NULL;
> > > + kfree(r5_core->pm_dev_core0_link);
> > > + r5_core->pm_dev_core0_link = NULL;
> > > +
> >
> > The error path is much cleaner and readable now.
> >
> > I will continue tomorrow.
> >
> > Mathieu
> >
> > > + return ret;
> > > +}
> > > +
> > > /**
> > > * zynqmp_r5_rproc_prepare()
> > > * adds carveouts for TCM bank and reserved memory regions
> > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > {
> > > int ret;
> > >
> > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > + if (ret) {
> > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > + return ret;
> > > + }
> > > +
> > > ret = add_tcm_banks(rproc);
> > > if (ret) {
> > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > - return ret;
> > > + goto fail_prepare;
> > > }
> > >
> > > ret = add_mem_regions_carveout(rproc);
> > > if (ret) {
> > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > - return ret;
> > > + goto fail_prepare;
> > > }
> > >
> > > return 0;
> > > +
> > > +fail_prepare:
> > > + zynqmp_r5_remove_pm_domains(rproc);
> > > +
> > > + return ret;
> > > }
> > >
> > > /**
> > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > >
> > > r5_core = rproc->priv;
> > >
> > > + zynqmp_r5_remove_pm_domains(rproc);
> > > +
> > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > if (zynqmp_pm_release_node(pm_domain_id))
> > > --
> > > 2.25.1
> > >
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-11-23 18:11 ` Mathieu Poirier
0 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-11-23 18:11 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> Hi Mathieu,
>
> Please find my comments below.
>
> On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > Hi,
> >
> > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > Use TCM pm domains extracted from device-tree
> > > to power on/off TCM using general pm domain framework.
> > >
> > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > ---
> > >
> > > Changes in v7:
> > > - %s/pm_dev1/pm_dev_core0/r
> > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > - %s/pm_dev2/pm_dev_core1/r
> > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > - remove pm_domain_id check to move next patch
> > > - add comment about how 1st entry in pm domain list is used
> > > - fix loop when jump to fail_add_pm_domains loop
> > >
> > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > index 4395edea9a64..22bccc5075a0 100644
> > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > @@ -16,6 +16,7 @@
> > > #include <linux/of_reserved_mem.h>
> > > #include <linux/platform_device.h>
> > > #include <linux/remoteproc.h>
> > > +#include <linux/pm_domain.h>
> > >
> > > #include "remoteproc_internal.h"
> > >
> > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > * @rproc: rproc handle
> > > * @pm_domain_id: RPU CPU power domain id
> > > * @ipi: pointer to mailbox information
> > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > + * @pm_dev_core0_link: pm domain device links after registration
> > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > + * registration
> > > */
> > > struct zynqmp_r5_core {
> > > struct device *dev;
> > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > struct rproc *rproc;
> > > u32 pm_domain_id;
> > > struct mbox_info *ipi;
> > > + int num_pm_dev;
> > > + struct device **pm_dev_core0;
> > > + struct device_link **pm_dev_core0_link;
> > > + struct device **pm_dev_core1;
> > > + struct device_link **pm_dev_core1_link;
> > > };
> > >
> > > /**
> > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > if (ret < 0) {
> > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > + pm_domain_id);
> >
> > Spurious change, you should have caught that.
>
> Ack, need to observe changes more closely before sending them.
>
> >
> > > goto release_tcm_lockstep;
> > > }
> > >
> > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > return ret;
> > > }
> > >
> > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > +{
> > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > + struct device *dev = r5_core->dev;
> > > + struct zynqmp_r5_cluster *cluster;
> > > + int i;
> > > +
> > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > +
> > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > + }
> > > +
> > > + kfree(r5_core->pm_dev_core0);
> > > + r5_core->pm_dev_core0 = NULL;
> > > + kfree(r5_core->pm_dev_core0_link);
> > > + r5_core->pm_dev_core0_link = NULL;
> > > +
> > > + if (cluster->mode == SPLIT_MODE) {
> > > + r5_core->num_pm_dev = 0;
> > > + return;
> > > + }
> > > +
> > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > + }
> > > +
> > > + kfree(r5_core->pm_dev_core1);
> > > + r5_core->pm_dev_core1 = NULL;
> > > + kfree(r5_core->pm_dev_core1_link);
> > > + r5_core->pm_dev_core1_link = NULL;
> > > + r5_core->num_pm_dev = 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > +{
> > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > + struct device *dev = r5_core->dev, *dev2;
> > > + struct zynqmp_r5_cluster *cluster;
> > > + struct platform_device *pdev;
> > > + struct device_node *np;
> > > + int i, j, num_pm_dev, ret;
> > > +
> > > + cluster = dev_get_drvdata(dev->parent);
> > > +
> > > + /* get number of power-domains */
> > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > + "#power-domain-cells");
> > > +
> > > + if (num_pm_dev <= 0)
> > > + return -EINVAL;
> > > +
> > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > + sizeof(struct device *),
> > > + GFP_KERNEL);
> > > + if (!r5_core->pm_dev_core0)
> > > + ret = -ENOMEM;
> > > +
> > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > + sizeof(struct device_link *),
> > > + GFP_KERNEL);
> > > + if (!r5_core->pm_dev_core0_link) {
> > > + kfree(r5_core->pm_dev_core0);
> > > + r5_core->pm_dev_core0 = NULL;
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + r5_core->num_pm_dev = num_pm_dev;
> > > +
> > > + /*
> > > + * start from 2nd entry in power-domains property list as
> > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > + * 1st entry is used to configure r5 operation mode.
> >
> > You are still not saying _where_ ->pm_dev_core0[0] gets added.
>
> So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
>
> zynqmp_pm_request_wake during rproc_start callback. I will document this in next
>
That is exactly what I am looking for. That way people don't have to go through
the entire driver trying to figure out what is happening with pm_dev_core[0].
I'm also not sure about the power-up order. Logically the TCMs should be
powered up before the R5 in order to put code in them. The R5s are powered in
zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
TCMs are powered - can you expand on that?
> revision. For new platforms pm_dev_core0[0] will be added in future.
Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
future"?
>
> I hope this meets expectations.
>
>
> >
> > > + */
> > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> >
> > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > stick with it. I have no preference on which one.
>
> Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
>
>
> >
> > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > + ret = -EINVAL;
> > > + goto fail_add_pm_domains;
> > > + }
> > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > + r5_core->pm_dev_core0[i],
> > > + DL_FLAG_STATELESS |
> > > + DL_FLAG_RPM_ACTIVE |
> > > + DL_FLAG_PM_RUNTIME);
> > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > + r5_core->pm_dev_core0[i] = NULL;
> > > + ret = -EINVAL;
> > > + goto fail_add_pm_domains;
> > > + }
> > > + }
> > > +
> > > + if (cluster->mode == SPLIT_MODE)
> > > + return 0;
> > > +
> > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > + sizeof(struct device *),
> > > + GFP_KERNEL);
> > > + if (!r5_core->pm_dev_core1) {
> > > + ret = -ENOMEM;
> > > + goto fail_add_pm_domains;
> > > + }
> > > +
> > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > + sizeof(struct device_link *),
> > > + GFP_KERNEL);
> > > + if (!r5_core->pm_dev_core1_link) {
> > > + kfree(r5_core->pm_dev_core1);
> > > + r5_core->pm_dev_core1 = NULL;
> > > + ret = -ENOMEM;
> > > + goto fail_add_pm_domains;
> > > + }
> > > +
> > > + /* get second core's device to detach its power-domains */
> > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > +
> > > + pdev = of_find_device_by_node(np);
> > > + if (!pdev) {
> > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > + kfree(r5_core->pm_dev_core1);
> > > + kfree(r5_core->pm_dev_core1_link);
> > > + r5_core->pm_dev_core1 = NULL;
> > > + r5_core->pm_dev_core1_link = NULL;
> > > + ret = -EINVAL;
> > > + goto fail_add_pm_domains;
> > > + }
> > > +
> > > + dev2 = &pdev->dev;
> > > +
> > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > + if (!r5_core->pm_dev_core1[j]) {
> > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > + ret = -EINVAL;
> > > + goto fail_add_pm_domains_lockstep;
> > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > + goto fail_add_pm_domains_lockstep;
> > > + }
> > > +
> > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > + r5_core->pm_dev_core1[j],
> > > + DL_FLAG_STATELESS |
> > > + DL_FLAG_RPM_ACTIVE |
> > > + DL_FLAG_PM_RUNTIME);
> > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > + r5_core->pm_dev_core1[j] = NULL;
> > > + ret = -ENODEV;
> > > + goto fail_add_pm_domains_lockstep;
> > > + }
> > > + }
> > > +
> > > +fail_add_pm_domains_lockstep:
> > > + while (--j >= 0) {
> > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > + }
> > > + kfree(r5_core->pm_dev_core1);
> > > + r5_core->pm_dev_core1 = NULL;
> > > + kfree(r5_core->pm_dev_core1_link);
> > > + r5_core->pm_dev_core1_link = NULL;
> > > +
> > > +fail_add_pm_domains:
> > > + while (--i >= 0) {
> > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > + }
> > > + kfree(r5_core->pm_dev_core0);
> > > + r5_core->pm_dev_core0 = NULL;
> > > + kfree(r5_core->pm_dev_core0_link);
> > > + r5_core->pm_dev_core0_link = NULL;
> > > +
> >
> > The error path is much cleaner and readable now.
> >
> > I will continue tomorrow.
> >
> > Mathieu
> >
> > > + return ret;
> > > +}
> > > +
> > > /**
> > > * zynqmp_r5_rproc_prepare()
> > > * adds carveouts for TCM bank and reserved memory regions
> > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > {
> > > int ret;
> > >
> > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > + if (ret) {
> > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > + return ret;
> > > + }
> > > +
> > > ret = add_tcm_banks(rproc);
> > > if (ret) {
> > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > - return ret;
> > > + goto fail_prepare;
> > > }
> > >
> > > ret = add_mem_regions_carveout(rproc);
> > > if (ret) {
> > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > - return ret;
> > > + goto fail_prepare;
> > > }
> > >
> > > return 0;
> > > +
> > > +fail_prepare:
> > > + zynqmp_r5_remove_pm_domains(rproc);
> > > +
> > > + return ret;
> > > }
> > >
> > > /**
> > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > >
> > > r5_core = rproc->priv;
> > >
> > > + zynqmp_r5_remove_pm_domains(rproc);
> > > +
> > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > if (zynqmp_pm_release_node(pm_domain_id))
> > > --
> > > 2.25.1
> > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-11-23 18:11 ` Mathieu Poirier
@ 2023-11-27 16:33 ` Tanmay Shah
-1 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-27 16:33 UTC (permalink / raw)
To: Mathieu Poirier
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > Hi Mathieu,
> >
> > Please find my comments below.
> >
> > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > Hi,
> > >
> > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > Use TCM pm domains extracted from device-tree
> > > > to power on/off TCM using general pm domain framework.
> > > >
> > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > ---
> > > >
> > > > Changes in v7:
> > > > - %s/pm_dev1/pm_dev_core0/r
> > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > - %s/pm_dev2/pm_dev_core1/r
> > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > - remove pm_domain_id check to move next patch
> > > > - add comment about how 1st entry in pm domain list is used
> > > > - fix loop when jump to fail_add_pm_domains loop
> > > >
> > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > index 4395edea9a64..22bccc5075a0 100644
> > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > @@ -16,6 +16,7 @@
> > > > #include <linux/of_reserved_mem.h>
> > > > #include <linux/platform_device.h>
> > > > #include <linux/remoteproc.h>
> > > > +#include <linux/pm_domain.h>
> > > >
> > > > #include "remoteproc_internal.h"
> > > >
> > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > * @rproc: rproc handle
> > > > * @pm_domain_id: RPU CPU power domain id
> > > > * @ipi: pointer to mailbox information
> > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > + * registration
> > > > */
> > > > struct zynqmp_r5_core {
> > > > struct device *dev;
> > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > struct rproc *rproc;
> > > > u32 pm_domain_id;
> > > > struct mbox_info *ipi;
> > > > + int num_pm_dev;
> > > > + struct device **pm_dev_core0;
> > > > + struct device_link **pm_dev_core0_link;
> > > > + struct device **pm_dev_core1;
> > > > + struct device_link **pm_dev_core1_link;
> > > > };
> > > >
> > > > /**
> > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > if (ret < 0) {
> > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > + pm_domain_id);
> > >
> > > Spurious change, you should have caught that.
> >
> > Ack, need to observe changes more closely before sending them.
> >
> > >
> > > > goto release_tcm_lockstep;
> > > > }
> > > >
> > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > return ret;
> > > > }
> > > >
> > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > +{
> > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > + struct device *dev = r5_core->dev;
> > > > + struct zynqmp_r5_cluster *cluster;
> > > > + int i;
> > > > +
> > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > +
> > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > + }
> > > > +
> > > > + kfree(r5_core->pm_dev_core0);
> > > > + r5_core->pm_dev_core0 = NULL;
> > > > + kfree(r5_core->pm_dev_core0_link);
> > > > + r5_core->pm_dev_core0_link = NULL;
> > > > +
> > > > + if (cluster->mode == SPLIT_MODE) {
> > > > + r5_core->num_pm_dev = 0;
> > > > + return;
> > > > + }
> > > > +
> > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > + }
> > > > +
> > > > + kfree(r5_core->pm_dev_core1);
> > > > + r5_core->pm_dev_core1 = NULL;
> > > > + kfree(r5_core->pm_dev_core1_link);
> > > > + r5_core->pm_dev_core1_link = NULL;
> > > > + r5_core->num_pm_dev = 0;
> > > > +}
> > > > +
> > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > +{
> > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > + struct device *dev = r5_core->dev, *dev2;
> > > > + struct zynqmp_r5_cluster *cluster;
> > > > + struct platform_device *pdev;
> > > > + struct device_node *np;
> > > > + int i, j, num_pm_dev, ret;
> > > > +
> > > > + cluster = dev_get_drvdata(dev->parent);
> > > > +
> > > > + /* get number of power-domains */
> > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > + "#power-domain-cells");
> > > > +
> > > > + if (num_pm_dev <= 0)
> > > > + return -EINVAL;
> > > > +
> > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > + sizeof(struct device *),
> > > > + GFP_KERNEL);
> > > > + if (!r5_core->pm_dev_core0)
> > > > + ret = -ENOMEM;
> > > > +
> > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > + sizeof(struct device_link *),
> > > > + GFP_KERNEL);
> > > > + if (!r5_core->pm_dev_core0_link) {
> > > > + kfree(r5_core->pm_dev_core0);
> > > > + r5_core->pm_dev_core0 = NULL;
> > > > + return -ENOMEM;
> > > > + }
> > > > +
> > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > +
> > > > + /*
> > > > + * start from 2nd entry in power-domains property list as
> > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > + * 1st entry is used to configure r5 operation mode.
> > >
> > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> >
> > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> >
> > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> >
>
> That is exactly what I am looking for. That way people don't have to go through
> the entire driver trying to figure out what is happening with pm_dev_core[0].
>
> I'm also not sure about the power-up order. Logically the TCMs should be
> powered up before the R5 in order to put code in them. The R5s are powered in
> zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> TCMs are powered - can you expand on that?
Sure. Following is call sequece
zynqmp_r5_rproc_prepare
zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
. . .
zynqmp_r5_rproc_start -> load firmware and Starts RPU
So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
>
> > revision. For new platforms pm_dev_core0[0] will be added in future.
>
> Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> future"?
ZynqMP platform has platform management firmware running on microblaze.
This firmware design does not expect R5 pm domains to be requested explicitly.
This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
However, this design was changed for new platforms i.e. "versal" and onwards.
Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
waking them up.
That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
Then, we should wake it up on r5_core.
To summarize:
For zynqmp only following call needed to start R5:
zynqmp_pm_request_wake
For "versal" and onwards we need two calls to start R5:
"device_link_add" and zynqmp_pm_request_wake
So, in future pm_core_dev[0] will be used.
> >
> > I hope this meets expectations.
> >
> >
> > >
> > > > + */
> > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > >
> > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > stick with it. I have no preference on which one.
> >
> > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> >
> >
> > >
> > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > + ret = -EINVAL;
> > > > + goto fail_add_pm_domains;
> > > > + }
> > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > + r5_core->pm_dev_core0[i],
> > > > + DL_FLAG_STATELESS |
> > > > + DL_FLAG_RPM_ACTIVE |
> > > > + DL_FLAG_PM_RUNTIME);
> > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > + ret = -EINVAL;
> > > > + goto fail_add_pm_domains;
> > > > + }
> > > > + }
> > > > +
> > > > + if (cluster->mode == SPLIT_MODE)
> > > > + return 0;
> > > > +
> > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > + sizeof(struct device *),
> > > > + GFP_KERNEL);
> > > > + if (!r5_core->pm_dev_core1) {
> > > > + ret = -ENOMEM;
> > > > + goto fail_add_pm_domains;
> > > > + }
> > > > +
> > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > + sizeof(struct device_link *),
> > > > + GFP_KERNEL);
> > > > + if (!r5_core->pm_dev_core1_link) {
> > > > + kfree(r5_core->pm_dev_core1);
> > > > + r5_core->pm_dev_core1 = NULL;
> > > > + ret = -ENOMEM;
> > > > + goto fail_add_pm_domains;
> > > > + }
> > > > +
> > > > + /* get second core's device to detach its power-domains */
> > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > +
> > > > + pdev = of_find_device_by_node(np);
> > > > + if (!pdev) {
> > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > + kfree(r5_core->pm_dev_core1);
> > > > + kfree(r5_core->pm_dev_core1_link);
> > > > + r5_core->pm_dev_core1 = NULL;
> > > > + r5_core->pm_dev_core1_link = NULL;
> > > > + ret = -EINVAL;
> > > > + goto fail_add_pm_domains;
> > > > + }
> > > > +
> > > > + dev2 = &pdev->dev;
> > > > +
> > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > + ret = -EINVAL;
> > > > + goto fail_add_pm_domains_lockstep;
> > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > + goto fail_add_pm_domains_lockstep;
> > > > + }
> > > > +
> > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > + r5_core->pm_dev_core1[j],
> > > > + DL_FLAG_STATELESS |
> > > > + DL_FLAG_RPM_ACTIVE |
> > > > + DL_FLAG_PM_RUNTIME);
> > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > + ret = -ENODEV;
> > > > + goto fail_add_pm_domains_lockstep;
> > > > + }
> > > > + }
> > > > +
> > > > +fail_add_pm_domains_lockstep:
> > > > + while (--j >= 0) {
> > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > + }
> > > > + kfree(r5_core->pm_dev_core1);
> > > > + r5_core->pm_dev_core1 = NULL;
> > > > + kfree(r5_core->pm_dev_core1_link);
> > > > + r5_core->pm_dev_core1_link = NULL;
> > > > +
> > > > +fail_add_pm_domains:
> > > > + while (--i >= 0) {
> > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > + }
> > > > + kfree(r5_core->pm_dev_core0);
> > > > + r5_core->pm_dev_core0 = NULL;
> > > > + kfree(r5_core->pm_dev_core0_link);
> > > > + r5_core->pm_dev_core0_link = NULL;
> > > > +
> > >
> > > The error path is much cleaner and readable now.
> > >
> > > I will continue tomorrow.
> > >
> > > Mathieu
> > >
> > > > + return ret;
> > > > +}
> > > > +
> > > > /**
> > > > * zynqmp_r5_rproc_prepare()
> > > > * adds carveouts for TCM bank and reserved memory regions
> > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > {
> > > > int ret;
> > > >
> > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > + if (ret) {
> > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > + return ret;
> > > > + }
> > > > +
> > > > ret = add_tcm_banks(rproc);
> > > > if (ret) {
> > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > - return ret;
> > > > + goto fail_prepare;
> > > > }
> > > >
> > > > ret = add_mem_regions_carveout(rproc);
> > > > if (ret) {
> > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > - return ret;
> > > > + goto fail_prepare;
> > > > }
> > > >
> > > > return 0;
> > > > +
> > > > +fail_prepare:
> > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > +
> > > > + return ret;
> > > > }
> > > >
> > > > /**
> > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > >
> > > > r5_core = rproc->priv;
> > > >
> > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > +
> > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > --
> > > > 2.25.1
> > > >
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-11-27 16:33 ` Tanmay Shah
0 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-27 16:33 UTC (permalink / raw)
To: Mathieu Poirier
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > Hi Mathieu,
> >
> > Please find my comments below.
> >
> > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > Hi,
> > >
> > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > Use TCM pm domains extracted from device-tree
> > > > to power on/off TCM using general pm domain framework.
> > > >
> > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > ---
> > > >
> > > > Changes in v7:
> > > > - %s/pm_dev1/pm_dev_core0/r
> > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > - %s/pm_dev2/pm_dev_core1/r
> > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > - remove pm_domain_id check to move next patch
> > > > - add comment about how 1st entry in pm domain list is used
> > > > - fix loop when jump to fail_add_pm_domains loop
> > > >
> > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > index 4395edea9a64..22bccc5075a0 100644
> > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > @@ -16,6 +16,7 @@
> > > > #include <linux/of_reserved_mem.h>
> > > > #include <linux/platform_device.h>
> > > > #include <linux/remoteproc.h>
> > > > +#include <linux/pm_domain.h>
> > > >
> > > > #include "remoteproc_internal.h"
> > > >
> > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > * @rproc: rproc handle
> > > > * @pm_domain_id: RPU CPU power domain id
> > > > * @ipi: pointer to mailbox information
> > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > + * registration
> > > > */
> > > > struct zynqmp_r5_core {
> > > > struct device *dev;
> > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > struct rproc *rproc;
> > > > u32 pm_domain_id;
> > > > struct mbox_info *ipi;
> > > > + int num_pm_dev;
> > > > + struct device **pm_dev_core0;
> > > > + struct device_link **pm_dev_core0_link;
> > > > + struct device **pm_dev_core1;
> > > > + struct device_link **pm_dev_core1_link;
> > > > };
> > > >
> > > > /**
> > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > if (ret < 0) {
> > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > + pm_domain_id);
> > >
> > > Spurious change, you should have caught that.
> >
> > Ack, need to observe changes more closely before sending them.
> >
> > >
> > > > goto release_tcm_lockstep;
> > > > }
> > > >
> > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > return ret;
> > > > }
> > > >
> > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > +{
> > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > + struct device *dev = r5_core->dev;
> > > > + struct zynqmp_r5_cluster *cluster;
> > > > + int i;
> > > > +
> > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > +
> > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > + }
> > > > +
> > > > + kfree(r5_core->pm_dev_core0);
> > > > + r5_core->pm_dev_core0 = NULL;
> > > > + kfree(r5_core->pm_dev_core0_link);
> > > > + r5_core->pm_dev_core0_link = NULL;
> > > > +
> > > > + if (cluster->mode == SPLIT_MODE) {
> > > > + r5_core->num_pm_dev = 0;
> > > > + return;
> > > > + }
> > > > +
> > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > + }
> > > > +
> > > > + kfree(r5_core->pm_dev_core1);
> > > > + r5_core->pm_dev_core1 = NULL;
> > > > + kfree(r5_core->pm_dev_core1_link);
> > > > + r5_core->pm_dev_core1_link = NULL;
> > > > + r5_core->num_pm_dev = 0;
> > > > +}
> > > > +
> > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > +{
> > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > + struct device *dev = r5_core->dev, *dev2;
> > > > + struct zynqmp_r5_cluster *cluster;
> > > > + struct platform_device *pdev;
> > > > + struct device_node *np;
> > > > + int i, j, num_pm_dev, ret;
> > > > +
> > > > + cluster = dev_get_drvdata(dev->parent);
> > > > +
> > > > + /* get number of power-domains */
> > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > + "#power-domain-cells");
> > > > +
> > > > + if (num_pm_dev <= 0)
> > > > + return -EINVAL;
> > > > +
> > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > + sizeof(struct device *),
> > > > + GFP_KERNEL);
> > > > + if (!r5_core->pm_dev_core0)
> > > > + ret = -ENOMEM;
> > > > +
> > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > + sizeof(struct device_link *),
> > > > + GFP_KERNEL);
> > > > + if (!r5_core->pm_dev_core0_link) {
> > > > + kfree(r5_core->pm_dev_core0);
> > > > + r5_core->pm_dev_core0 = NULL;
> > > > + return -ENOMEM;
> > > > + }
> > > > +
> > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > +
> > > > + /*
> > > > + * start from 2nd entry in power-domains property list as
> > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > + * 1st entry is used to configure r5 operation mode.
> > >
> > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> >
> > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> >
> > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> >
>
> That is exactly what I am looking for. That way people don't have to go through
> the entire driver trying to figure out what is happening with pm_dev_core[0].
>
> I'm also not sure about the power-up order. Logically the TCMs should be
> powered up before the R5 in order to put code in them. The R5s are powered in
> zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> TCMs are powered - can you expand on that?
Sure. Following is call sequece
zynqmp_r5_rproc_prepare
zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
. . .
zynqmp_r5_rproc_start -> load firmware and Starts RPU
So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
>
> > revision. For new platforms pm_dev_core0[0] will be added in future.
>
> Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> future"?
ZynqMP platform has platform management firmware running on microblaze.
This firmware design does not expect R5 pm domains to be requested explicitly.
This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
However, this design was changed for new platforms i.e. "versal" and onwards.
Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
waking them up.
That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
Then, we should wake it up on r5_core.
To summarize:
For zynqmp only following call needed to start R5:
zynqmp_pm_request_wake
For "versal" and onwards we need two calls to start R5:
"device_link_add" and zynqmp_pm_request_wake
So, in future pm_core_dev[0] will be used.
> >
> > I hope this meets expectations.
> >
> >
> > >
> > > > + */
> > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > >
> > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > stick with it. I have no preference on which one.
> >
> > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> >
> >
> > >
> > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > + ret = -EINVAL;
> > > > + goto fail_add_pm_domains;
> > > > + }
> > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > + r5_core->pm_dev_core0[i],
> > > > + DL_FLAG_STATELESS |
> > > > + DL_FLAG_RPM_ACTIVE |
> > > > + DL_FLAG_PM_RUNTIME);
> > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > + ret = -EINVAL;
> > > > + goto fail_add_pm_domains;
> > > > + }
> > > > + }
> > > > +
> > > > + if (cluster->mode == SPLIT_MODE)
> > > > + return 0;
> > > > +
> > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > + sizeof(struct device *),
> > > > + GFP_KERNEL);
> > > > + if (!r5_core->pm_dev_core1) {
> > > > + ret = -ENOMEM;
> > > > + goto fail_add_pm_domains;
> > > > + }
> > > > +
> > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > + sizeof(struct device_link *),
> > > > + GFP_KERNEL);
> > > > + if (!r5_core->pm_dev_core1_link) {
> > > > + kfree(r5_core->pm_dev_core1);
> > > > + r5_core->pm_dev_core1 = NULL;
> > > > + ret = -ENOMEM;
> > > > + goto fail_add_pm_domains;
> > > > + }
> > > > +
> > > > + /* get second core's device to detach its power-domains */
> > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > +
> > > > + pdev = of_find_device_by_node(np);
> > > > + if (!pdev) {
> > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > + kfree(r5_core->pm_dev_core1);
> > > > + kfree(r5_core->pm_dev_core1_link);
> > > > + r5_core->pm_dev_core1 = NULL;
> > > > + r5_core->pm_dev_core1_link = NULL;
> > > > + ret = -EINVAL;
> > > > + goto fail_add_pm_domains;
> > > > + }
> > > > +
> > > > + dev2 = &pdev->dev;
> > > > +
> > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > + ret = -EINVAL;
> > > > + goto fail_add_pm_domains_lockstep;
> > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > + goto fail_add_pm_domains_lockstep;
> > > > + }
> > > > +
> > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > + r5_core->pm_dev_core1[j],
> > > > + DL_FLAG_STATELESS |
> > > > + DL_FLAG_RPM_ACTIVE |
> > > > + DL_FLAG_PM_RUNTIME);
> > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > + ret = -ENODEV;
> > > > + goto fail_add_pm_domains_lockstep;
> > > > + }
> > > > + }
> > > > +
> > > > +fail_add_pm_domains_lockstep:
> > > > + while (--j >= 0) {
> > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > + }
> > > > + kfree(r5_core->pm_dev_core1);
> > > > + r5_core->pm_dev_core1 = NULL;
> > > > + kfree(r5_core->pm_dev_core1_link);
> > > > + r5_core->pm_dev_core1_link = NULL;
> > > > +
> > > > +fail_add_pm_domains:
> > > > + while (--i >= 0) {
> > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > + }
> > > > + kfree(r5_core->pm_dev_core0);
> > > > + r5_core->pm_dev_core0 = NULL;
> > > > + kfree(r5_core->pm_dev_core0_link);
> > > > + r5_core->pm_dev_core0_link = NULL;
> > > > +
> > >
> > > The error path is much cleaner and readable now.
> > >
> > > I will continue tomorrow.
> > >
> > > Mathieu
> > >
> > > > + return ret;
> > > > +}
> > > > +
> > > > /**
> > > > * zynqmp_r5_rproc_prepare()
> > > > * adds carveouts for TCM bank and reserved memory regions
> > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > {
> > > > int ret;
> > > >
> > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > + if (ret) {
> > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > + return ret;
> > > > + }
> > > > +
> > > > ret = add_tcm_banks(rproc);
> > > > if (ret) {
> > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > - return ret;
> > > > + goto fail_prepare;
> > > > }
> > > >
> > > > ret = add_mem_regions_carveout(rproc);
> > > > if (ret) {
> > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > - return ret;
> > > > + goto fail_prepare;
> > > > }
> > > >
> > > > return 0;
> > > > +
> > > > +fail_prepare:
> > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > +
> > > > + return ret;
> > > > }
> > > >
> > > > /**
> > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > >
> > > > r5_core = rproc->priv;
> > > >
> > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > +
> > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > --
> > > > 2.25.1
> > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-11-27 16:33 ` Tanmay Shah
@ 2023-11-29 17:10 ` Mathieu Poirier
-1 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-11-29 17:10 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
>
> On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > Hi Mathieu,
> > >
> > > Please find my comments below.
> > >
> > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > Hi,
> > > >
> > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > Use TCM pm domains extracted from device-tree
> > > > > to power on/off TCM using general pm domain framework.
> > > > >
> > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > ---
> > > > >
> > > > > Changes in v7:
> > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > - remove pm_domain_id check to move next patch
> > > > > - add comment about how 1st entry in pm domain list is used
> > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > >
> > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > >
> > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > @@ -16,6 +16,7 @@
> > > > > #include <linux/of_reserved_mem.h>
> > > > > #include <linux/platform_device.h>
> > > > > #include <linux/remoteproc.h>
> > > > > +#include <linux/pm_domain.h>
> > > > >
> > > > > #include "remoteproc_internal.h"
> > > > >
> > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > * @rproc: rproc handle
> > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > * @ipi: pointer to mailbox information
> > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > + * registration
> > > > > */
> > > > > struct zynqmp_r5_core {
> > > > > struct device *dev;
> > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > struct rproc *rproc;
> > > > > u32 pm_domain_id;
> > > > > struct mbox_info *ipi;
> > > > > + int num_pm_dev;
> > > > > + struct device **pm_dev_core0;
> > > > > + struct device_link **pm_dev_core0_link;
> > > > > + struct device **pm_dev_core1;
> > > > > + struct device_link **pm_dev_core1_link;
> > > > > };
> > > > >
> > > > > /**
> > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > if (ret < 0) {
> > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > + pm_domain_id);
> > > >
> > > > Spurious change, you should have caught that.
> > >
> > > Ack, need to observe changes more closely before sending them.
> > >
> > > >
> > > > > goto release_tcm_lockstep;
> > > > > }
> > > > >
> > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > return ret;
> > > > > }
> > > > >
> > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > +{
> > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > + struct device *dev = r5_core->dev;
> > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > + int i;
> > > > > +
> > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > +
> > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > + }
> > > > > +
> > > > > + kfree(r5_core->pm_dev_core0);
> > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > +
> > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > + r5_core->num_pm_dev = 0;
> > > > > + return;
> > > > > + }
> > > > > +
> > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > + }
> > > > > +
> > > > > + kfree(r5_core->pm_dev_core1);
> > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > + r5_core->num_pm_dev = 0;
> > > > > +}
> > > > > +
> > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > +{
> > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > + struct platform_device *pdev;
> > > > > + struct device_node *np;
> > > > > + int i, j, num_pm_dev, ret;
> > > > > +
> > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > +
> > > > > + /* get number of power-domains */
> > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > + "#power-domain-cells");
> > > > > +
> > > > > + if (num_pm_dev <= 0)
> > > > > + return -EINVAL;
> > > > > +
> > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > + sizeof(struct device *),
> > > > > + GFP_KERNEL);
> > > > > + if (!r5_core->pm_dev_core0)
> > > > > + ret = -ENOMEM;
> > > > > +
> > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > + sizeof(struct device_link *),
> > > > > + GFP_KERNEL);
> > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > + kfree(r5_core->pm_dev_core0);
> > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > + return -ENOMEM;
> > > > > + }
> > > > > +
> > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > +
> > > > > + /*
> > > > > + * start from 2nd entry in power-domains property list as
> > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > + * 1st entry is used to configure r5 operation mode.
> > > >
> > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > >
> > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > >
> > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > >
> >
> > That is exactly what I am looking for. That way people don't have to go through
> > the entire driver trying to figure out what is happening with pm_dev_core[0].
> >
> > I'm also not sure about the power-up order. Logically the TCMs should be
> > powered up before the R5 in order to put code in them. The R5s are powered in
> > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > TCMs are powered - can you expand on that?
>
>
> Sure. Following is call sequece
>
> zynqmp_r5_rproc_prepare
>
> zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
>
> . . .
>
> zynqmp_r5_rproc_start -> load firmware and Starts RPU
>
> So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
>
>
> >
> > > revision. For new platforms pm_dev_core0[0] will be added in future.
> >
> > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > future"?
>
>
> ZynqMP platform has platform management firmware running on microblaze.
>
> This firmware design does not expect R5 pm domains to be requested explicitly.
>
> This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
>
> firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
>
> However, this design was changed for new platforms i.e. "versal" and onwards.
>
> Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
>
> waking them up.
>
> That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
>
> Then, we should wake it up on r5_core.
>
> To summarize:
>
> For zynqmp only following call needed to start R5:
>
> zynqmp_pm_request_wake
>
> For "versal" and onwards we need two calls to start R5:
>
> "device_link_add" and zynqmp_pm_request_wake
>
> So, in future pm_core_dev[0] will be used.
>
Thanks for the clarification on both front. The problem here is that we are
keeping R5 power domain information in two different places, i.e
zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
Please see if you can retreive the power domain ID from
zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
zynqmp_r5_core::pm_domain_id.
>
> > >
> > > I hope this meets expectations.
> > >
> > >
> > > >
> > > > > + */
> > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > >
> > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > stick with it. I have no preference on which one.
> > >
> > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > >
> > >
> > > >
> > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > + ret = -EINVAL;
> > > > > + goto fail_add_pm_domains;
> > > > > + }
> > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > + r5_core->pm_dev_core0[i],
> > > > > + DL_FLAG_STATELESS |
> > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > + DL_FLAG_PM_RUNTIME);
> > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > + ret = -EINVAL;
> > > > > + goto fail_add_pm_domains;
> > > > > + }
> > > > > + }
> > > > > +
> > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > + return 0;
> > > > > +
> > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > + sizeof(struct device *),
> > > > > + GFP_KERNEL);
> > > > > + if (!r5_core->pm_dev_core1) {
> > > > > + ret = -ENOMEM;
> > > > > + goto fail_add_pm_domains;
> > > > > + }
> > > > > +
> > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > + sizeof(struct device_link *),
> > > > > + GFP_KERNEL);
> > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > + kfree(r5_core->pm_dev_core1);
> > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > + ret = -ENOMEM;
> > > > > + goto fail_add_pm_domains;
> > > > > + }
> > > > > +
> > > > > + /* get second core's device to detach its power-domains */
> > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > +
> > > > > + pdev = of_find_device_by_node(np);
> > > > > + if (!pdev) {
> > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > + kfree(r5_core->pm_dev_core1);
> > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > + ret = -EINVAL;
> > > > > + goto fail_add_pm_domains;
> > > > > + }
> > > > > +
> > > > > + dev2 = &pdev->dev;
> > > > > +
> > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > + ret = -EINVAL;
> > > > > + goto fail_add_pm_domains_lockstep;
> > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > + goto fail_add_pm_domains_lockstep;
> > > > > + }
> > > > > +
> > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > + r5_core->pm_dev_core1[j],
> > > > > + DL_FLAG_STATELESS |
> > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > + DL_FLAG_PM_RUNTIME);
> > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > + ret = -ENODEV;
> > > > > + goto fail_add_pm_domains_lockstep;
> > > > > + }
> > > > > + }
> > > > > +
> > > > > +fail_add_pm_domains_lockstep:
> > > > > + while (--j >= 0) {
> > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > + }
> > > > > + kfree(r5_core->pm_dev_core1);
> > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > +
> > > > > +fail_add_pm_domains:
> > > > > + while (--i >= 0) {
> > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > + }
> > > > > + kfree(r5_core->pm_dev_core0);
> > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > +
> > > >
> > > > The error path is much cleaner and readable now.
> > > >
> > > > I will continue tomorrow.
> > > >
> > > > Mathieu
> > > >
> > > > > + return ret;
> > > > > +}
> > > > > +
> > > > > /**
> > > > > * zynqmp_r5_rproc_prepare()
> > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > {
> > > > > int ret;
> > > > >
> > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > + if (ret) {
> > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > + return ret;
> > > > > + }
> > > > > +
> > > > > ret = add_tcm_banks(rproc);
> > > > > if (ret) {
> > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > - return ret;
> > > > > + goto fail_prepare;
> > > > > }
> > > > >
> > > > > ret = add_mem_regions_carveout(rproc);
> > > > > if (ret) {
> > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > - return ret;
> > > > > + goto fail_prepare;
> > > > > }
> > > > >
> > > > > return 0;
> > > > > +
> > > > > +fail_prepare:
> > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > +
> > > > > + return ret;
> > > > > }
> > > > >
> > > > > /**
> > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > >
> > > > > r5_core = rproc->priv;
> > > > >
> > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > +
> > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > --
> > > > > 2.25.1
> > > > >
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-11-29 17:10 ` Mathieu Poirier
0 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-11-29 17:10 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
>
> On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > Hi Mathieu,
> > >
> > > Please find my comments below.
> > >
> > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > Hi,
> > > >
> > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > Use TCM pm domains extracted from device-tree
> > > > > to power on/off TCM using general pm domain framework.
> > > > >
> > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > ---
> > > > >
> > > > > Changes in v7:
> > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > - remove pm_domain_id check to move next patch
> > > > > - add comment about how 1st entry in pm domain list is used
> > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > >
> > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > >
> > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > @@ -16,6 +16,7 @@
> > > > > #include <linux/of_reserved_mem.h>
> > > > > #include <linux/platform_device.h>
> > > > > #include <linux/remoteproc.h>
> > > > > +#include <linux/pm_domain.h>
> > > > >
> > > > > #include "remoteproc_internal.h"
> > > > >
> > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > * @rproc: rproc handle
> > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > * @ipi: pointer to mailbox information
> > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > + * registration
> > > > > */
> > > > > struct zynqmp_r5_core {
> > > > > struct device *dev;
> > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > struct rproc *rproc;
> > > > > u32 pm_domain_id;
> > > > > struct mbox_info *ipi;
> > > > > + int num_pm_dev;
> > > > > + struct device **pm_dev_core0;
> > > > > + struct device_link **pm_dev_core0_link;
> > > > > + struct device **pm_dev_core1;
> > > > > + struct device_link **pm_dev_core1_link;
> > > > > };
> > > > >
> > > > > /**
> > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > if (ret < 0) {
> > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > + pm_domain_id);
> > > >
> > > > Spurious change, you should have caught that.
> > >
> > > Ack, need to observe changes more closely before sending them.
> > >
> > > >
> > > > > goto release_tcm_lockstep;
> > > > > }
> > > > >
> > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > return ret;
> > > > > }
> > > > >
> > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > +{
> > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > + struct device *dev = r5_core->dev;
> > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > + int i;
> > > > > +
> > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > +
> > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > + }
> > > > > +
> > > > > + kfree(r5_core->pm_dev_core0);
> > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > +
> > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > + r5_core->num_pm_dev = 0;
> > > > > + return;
> > > > > + }
> > > > > +
> > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > + }
> > > > > +
> > > > > + kfree(r5_core->pm_dev_core1);
> > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > + r5_core->num_pm_dev = 0;
> > > > > +}
> > > > > +
> > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > +{
> > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > + struct platform_device *pdev;
> > > > > + struct device_node *np;
> > > > > + int i, j, num_pm_dev, ret;
> > > > > +
> > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > +
> > > > > + /* get number of power-domains */
> > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > + "#power-domain-cells");
> > > > > +
> > > > > + if (num_pm_dev <= 0)
> > > > > + return -EINVAL;
> > > > > +
> > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > + sizeof(struct device *),
> > > > > + GFP_KERNEL);
> > > > > + if (!r5_core->pm_dev_core0)
> > > > > + ret = -ENOMEM;
> > > > > +
> > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > + sizeof(struct device_link *),
> > > > > + GFP_KERNEL);
> > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > + kfree(r5_core->pm_dev_core0);
> > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > + return -ENOMEM;
> > > > > + }
> > > > > +
> > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > +
> > > > > + /*
> > > > > + * start from 2nd entry in power-domains property list as
> > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > + * 1st entry is used to configure r5 operation mode.
> > > >
> > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > >
> > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > >
> > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > >
> >
> > That is exactly what I am looking for. That way people don't have to go through
> > the entire driver trying to figure out what is happening with pm_dev_core[0].
> >
> > I'm also not sure about the power-up order. Logically the TCMs should be
> > powered up before the R5 in order to put code in them. The R5s are powered in
> > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > TCMs are powered - can you expand on that?
>
>
> Sure. Following is call sequece
>
> zynqmp_r5_rproc_prepare
>
> zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
>
> . . .
>
> zynqmp_r5_rproc_start -> load firmware and Starts RPU
>
> So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
>
>
> >
> > > revision. For new platforms pm_dev_core0[0] will be added in future.
> >
> > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > future"?
>
>
> ZynqMP platform has platform management firmware running on microblaze.
>
> This firmware design does not expect R5 pm domains to be requested explicitly.
>
> This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
>
> firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
>
> However, this design was changed for new platforms i.e. "versal" and onwards.
>
> Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
>
> waking them up.
>
> That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
>
> Then, we should wake it up on r5_core.
>
> To summarize:
>
> For zynqmp only following call needed to start R5:
>
> zynqmp_pm_request_wake
>
> For "versal" and onwards we need two calls to start R5:
>
> "device_link_add" and zynqmp_pm_request_wake
>
> So, in future pm_core_dev[0] will be used.
>
Thanks for the clarification on both front. The problem here is that we are
keeping R5 power domain information in two different places, i.e
zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
Please see if you can retreive the power domain ID from
zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
zynqmp_r5_core::pm_domain_id.
>
> > >
> > > I hope this meets expectations.
> > >
> > >
> > > >
> > > > > + */
> > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > >
> > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > stick with it. I have no preference on which one.
> > >
> > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > >
> > >
> > > >
> > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > + ret = -EINVAL;
> > > > > + goto fail_add_pm_domains;
> > > > > + }
> > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > + r5_core->pm_dev_core0[i],
> > > > > + DL_FLAG_STATELESS |
> > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > + DL_FLAG_PM_RUNTIME);
> > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > + ret = -EINVAL;
> > > > > + goto fail_add_pm_domains;
> > > > > + }
> > > > > + }
> > > > > +
> > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > + return 0;
> > > > > +
> > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > + sizeof(struct device *),
> > > > > + GFP_KERNEL);
> > > > > + if (!r5_core->pm_dev_core1) {
> > > > > + ret = -ENOMEM;
> > > > > + goto fail_add_pm_domains;
> > > > > + }
> > > > > +
> > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > + sizeof(struct device_link *),
> > > > > + GFP_KERNEL);
> > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > + kfree(r5_core->pm_dev_core1);
> > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > + ret = -ENOMEM;
> > > > > + goto fail_add_pm_domains;
> > > > > + }
> > > > > +
> > > > > + /* get second core's device to detach its power-domains */
> > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > +
> > > > > + pdev = of_find_device_by_node(np);
> > > > > + if (!pdev) {
> > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > + kfree(r5_core->pm_dev_core1);
> > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > + ret = -EINVAL;
> > > > > + goto fail_add_pm_domains;
> > > > > + }
> > > > > +
> > > > > + dev2 = &pdev->dev;
> > > > > +
> > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > + ret = -EINVAL;
> > > > > + goto fail_add_pm_domains_lockstep;
> > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > + goto fail_add_pm_domains_lockstep;
> > > > > + }
> > > > > +
> > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > + r5_core->pm_dev_core1[j],
> > > > > + DL_FLAG_STATELESS |
> > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > + DL_FLAG_PM_RUNTIME);
> > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > + ret = -ENODEV;
> > > > > + goto fail_add_pm_domains_lockstep;
> > > > > + }
> > > > > + }
> > > > > +
> > > > > +fail_add_pm_domains_lockstep:
> > > > > + while (--j >= 0) {
> > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > + }
> > > > > + kfree(r5_core->pm_dev_core1);
> > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > +
> > > > > +fail_add_pm_domains:
> > > > > + while (--i >= 0) {
> > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > + }
> > > > > + kfree(r5_core->pm_dev_core0);
> > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > +
> > > >
> > > > The error path is much cleaner and readable now.
> > > >
> > > > I will continue tomorrow.
> > > >
> > > > Mathieu
> > > >
> > > > > + return ret;
> > > > > +}
> > > > > +
> > > > > /**
> > > > > * zynqmp_r5_rproc_prepare()
> > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > {
> > > > > int ret;
> > > > >
> > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > + if (ret) {
> > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > + return ret;
> > > > > + }
> > > > > +
> > > > > ret = add_tcm_banks(rproc);
> > > > > if (ret) {
> > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > - return ret;
> > > > > + goto fail_prepare;
> > > > > }
> > > > >
> > > > > ret = add_mem_regions_carveout(rproc);
> > > > > if (ret) {
> > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > - return ret;
> > > > > + goto fail_prepare;
> > > > > }
> > > > >
> > > > > return 0;
> > > > > +
> > > > > +fail_prepare:
> > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > +
> > > > > + return ret;
> > > > > }
> > > > >
> > > > > /**
> > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > >
> > > > > r5_core = rproc->priv;
> > > > >
> > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > +
> > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > --
> > > > > 2.25.1
> > > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-11-29 17:10 ` Mathieu Poirier
@ 2023-11-29 17:42 ` Tanmay Shah
-1 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-29 17:42 UTC (permalink / raw)
To: Mathieu Poirier
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On 11/29/23 11:10 AM, Mathieu Poirier wrote:
> On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
> >
> > On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > > Hi Mathieu,
> > > >
> > > > Please find my comments below.
> > > >
> > > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > > Hi,
> > > > >
> > > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > > Use TCM pm domains extracted from device-tree
> > > > > > to power on/off TCM using general pm domain framework.
> > > > > >
> > > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > > ---
> > > > > >
> > > > > > Changes in v7:
> > > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > > - remove pm_domain_id check to move next patch
> > > > > > - add comment about how 1st entry in pm domain list is used
> > > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > > >
> > > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > > >
> > > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > @@ -16,6 +16,7 @@
> > > > > > #include <linux/of_reserved_mem.h>
> > > > > > #include <linux/platform_device.h>
> > > > > > #include <linux/remoteproc.h>
> > > > > > +#include <linux/pm_domain.h>
> > > > > >
> > > > > > #include "remoteproc_internal.h"
> > > > > >
> > > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > > * @rproc: rproc handle
> > > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > > * @ipi: pointer to mailbox information
> > > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > > + * registration
> > > > > > */
> > > > > > struct zynqmp_r5_core {
> > > > > > struct device *dev;
> > > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > > struct rproc *rproc;
> > > > > > u32 pm_domain_id;
> > > > > > struct mbox_info *ipi;
> > > > > > + int num_pm_dev;
> > > > > > + struct device **pm_dev_core0;
> > > > > > + struct device_link **pm_dev_core0_link;
> > > > > > + struct device **pm_dev_core1;
> > > > > > + struct device_link **pm_dev_core1_link;
> > > > > > };
> > > > > >
> > > > > > /**
> > > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > > if (ret < 0) {
> > > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > > + pm_domain_id);
> > > > >
> > > > > Spurious change, you should have caught that.
> > > >
> > > > Ack, need to observe changes more closely before sending them.
> > > >
> > > > >
> > > > > > goto release_tcm_lockstep;
> > > > > > }
> > > > > >
> > > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > > return ret;
> > > > > > }
> > > > > >
> > > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > > +{
> > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > + struct device *dev = r5_core->dev;
> > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > + int i;
> > > > > > +
> > > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > > +
> > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > > + }
> > > > > > +
> > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > +
> > > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > > + r5_core->num_pm_dev = 0;
> > > > > > + return;
> > > > > > + }
> > > > > > +
> > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > > + }
> > > > > > +
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > + r5_core->num_pm_dev = 0;
> > > > > > +}
> > > > > > +
> > > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > > +{
> > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > + struct platform_device *pdev;
> > > > > > + struct device_node *np;
> > > > > > + int i, j, num_pm_dev, ret;
> > > > > > +
> > > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > > +
> > > > > > + /* get number of power-domains */
> > > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > > + "#power-domain-cells");
> > > > > > +
> > > > > > + if (num_pm_dev <= 0)
> > > > > > + return -EINVAL;
> > > > > > +
> > > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core0)
> > > > > > + ret = -ENOMEM;
> > > > > > +
> > > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device_link *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > + return -ENOMEM;
> > > > > > + }
> > > > > > +
> > > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > > +
> > > > > > + /*
> > > > > > + * start from 2nd entry in power-domains property list as
> > > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > > + * 1st entry is used to configure r5 operation mode.
> > > > >
> > > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > > >
> > > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > > >
> > > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > > >
> > >
> > > That is exactly what I am looking for. That way people don't have to go through
> > > the entire driver trying to figure out what is happening with pm_dev_core[0].
> > >
> > > I'm also not sure about the power-up order. Logically the TCMs should be
> > > powered up before the R5 in order to put code in them. The R5s are powered in
> > > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > > TCMs are powered - can you expand on that?
> >
> >
> > Sure. Following is call sequece
> >
> > zynqmp_r5_rproc_prepare
> >
> > zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
> >
> > . . .
> >
> > zynqmp_r5_rproc_start -> load firmware and Starts RPU
> >
> > So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
> >
> >
> > >
> > > > revision. For new platforms pm_dev_core0[0] will be added in future.
> > >
> > > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > > future"?
> >
> >
> > ZynqMP platform has platform management firmware running on microblaze.
> >
> > This firmware design does not expect R5 pm domains to be requested explicitly.
> >
> > This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
> >
> > firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
> >
> > However, this design was changed for new platforms i.e. "versal" and onwards.
> >
> > Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
> >
> > waking them up.
> >
> > That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
> >
> > Then, we should wake it up on r5_core.
> >
> > To summarize:
> >
> > For zynqmp only following call needed to start R5:
> >
> > zynqmp_pm_request_wake
> >
> > For "versal" and onwards we need two calls to start R5:
> >
> > "device_link_add" and zynqmp_pm_request_wake
> >
> > So, in future pm_core_dev[0] will be used.
> >
>
> Thanks for the clarification on both front. The problem here is that we are
> keeping R5 power domain information in two different places, i.e
> zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
>
> Please see if you can retreive the power domain ID from
> zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
> calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
> zynqmp_r5_core::pm_domain_id.
Thanks for this suggestion. Let me find out if I can retrieve that or not.
I believe it should be possible.
> >
> > > >
> > > > I hope this meets expectations.
> > > >
> > > >
> > > > >
> > > > > > + */
> > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > > >
> > > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > > stick with it. I have no preference on which one.
> > > >
> > > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > > >
> > > >
> > > > >
> > > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > > + r5_core->pm_dev_core0[i],
> > > > > > + DL_FLAG_STATELESS |
> > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > + }
> > > > > > +
> > > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > > + return 0;
> > > > > > +
> > > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core1) {
> > > > > > + ret = -ENOMEM;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > +
> > > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device_link *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + ret = -ENOMEM;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > +
> > > > > > + /* get second core's device to detach its power-domains */
> > > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > > +
> > > > > > + pdev = of_find_device_by_node(np);
> > > > > > + if (!pdev) {
> > > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > +
> > > > > > + dev2 = &pdev->dev;
> > > > > > +
> > > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > + }
> > > > > > +
> > > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > > + r5_core->pm_dev_core1[j],
> > > > > > + DL_FLAG_STATELESS |
> > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > > + ret = -ENODEV;
> > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > + }
> > > > > > + }
> > > > > > +
> > > > > > +fail_add_pm_domains_lockstep:
> > > > > > + while (--j >= 0) {
> > > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > + }
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > +
> > > > > > +fail_add_pm_domains:
> > > > > > + while (--i >= 0) {
> > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > + }
> > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > +
> > > > >
> > > > > The error path is much cleaner and readable now.
> > > > >
> > > > > I will continue tomorrow.
> > > > >
> > > > > Mathieu
> > > > >
> > > > > > + return ret;
> > > > > > +}
> > > > > > +
> > > > > > /**
> > > > > > * zynqmp_r5_rproc_prepare()
> > > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > > {
> > > > > > int ret;
> > > > > >
> > > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > > + if (ret) {
> > > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > > + return ret;
> > > > > > + }
> > > > > > +
> > > > > > ret = add_tcm_banks(rproc);
> > > > > > if (ret) {
> > > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > > - return ret;
> > > > > > + goto fail_prepare;
> > > > > > }
> > > > > >
> > > > > > ret = add_mem_regions_carveout(rproc);
> > > > > > if (ret) {
> > > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > > - return ret;
> > > > > > + goto fail_prepare;
> > > > > > }
> > > > > >
> > > > > > return 0;
> > > > > > +
> > > > > > +fail_prepare:
> > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > +
> > > > > > + return ret;
> > > > > > }
> > > > > >
> > > > > > /**
> > > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > > >
> > > > > > r5_core = rproc->priv;
> > > > > >
> > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > +
> > > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > > --
> > > > > > 2.25.1
> > > > > >
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-11-29 17:42 ` Tanmay Shah
0 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-11-29 17:42 UTC (permalink / raw)
To: Mathieu Poirier
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On 11/29/23 11:10 AM, Mathieu Poirier wrote:
> On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
> >
> > On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > > Hi Mathieu,
> > > >
> > > > Please find my comments below.
> > > >
> > > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > > Hi,
> > > > >
> > > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > > Use TCM pm domains extracted from device-tree
> > > > > > to power on/off TCM using general pm domain framework.
> > > > > >
> > > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > > ---
> > > > > >
> > > > > > Changes in v7:
> > > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > > - remove pm_domain_id check to move next patch
> > > > > > - add comment about how 1st entry in pm domain list is used
> > > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > > >
> > > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > > >
> > > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > @@ -16,6 +16,7 @@
> > > > > > #include <linux/of_reserved_mem.h>
> > > > > > #include <linux/platform_device.h>
> > > > > > #include <linux/remoteproc.h>
> > > > > > +#include <linux/pm_domain.h>
> > > > > >
> > > > > > #include "remoteproc_internal.h"
> > > > > >
> > > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > > * @rproc: rproc handle
> > > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > > * @ipi: pointer to mailbox information
> > > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > > + * registration
> > > > > > */
> > > > > > struct zynqmp_r5_core {
> > > > > > struct device *dev;
> > > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > > struct rproc *rproc;
> > > > > > u32 pm_domain_id;
> > > > > > struct mbox_info *ipi;
> > > > > > + int num_pm_dev;
> > > > > > + struct device **pm_dev_core0;
> > > > > > + struct device_link **pm_dev_core0_link;
> > > > > > + struct device **pm_dev_core1;
> > > > > > + struct device_link **pm_dev_core1_link;
> > > > > > };
> > > > > >
> > > > > > /**
> > > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > > if (ret < 0) {
> > > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > > + pm_domain_id);
> > > > >
> > > > > Spurious change, you should have caught that.
> > > >
> > > > Ack, need to observe changes more closely before sending them.
> > > >
> > > > >
> > > > > > goto release_tcm_lockstep;
> > > > > > }
> > > > > >
> > > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > > return ret;
> > > > > > }
> > > > > >
> > > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > > +{
> > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > + struct device *dev = r5_core->dev;
> > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > + int i;
> > > > > > +
> > > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > > +
> > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > > + }
> > > > > > +
> > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > +
> > > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > > + r5_core->num_pm_dev = 0;
> > > > > > + return;
> > > > > > + }
> > > > > > +
> > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > > + }
> > > > > > +
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > + r5_core->num_pm_dev = 0;
> > > > > > +}
> > > > > > +
> > > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > > +{
> > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > + struct platform_device *pdev;
> > > > > > + struct device_node *np;
> > > > > > + int i, j, num_pm_dev, ret;
> > > > > > +
> > > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > > +
> > > > > > + /* get number of power-domains */
> > > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > > + "#power-domain-cells");
> > > > > > +
> > > > > > + if (num_pm_dev <= 0)
> > > > > > + return -EINVAL;
> > > > > > +
> > > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core0)
> > > > > > + ret = -ENOMEM;
> > > > > > +
> > > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device_link *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > + return -ENOMEM;
> > > > > > + }
> > > > > > +
> > > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > > +
> > > > > > + /*
> > > > > > + * start from 2nd entry in power-domains property list as
> > > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > > + * 1st entry is used to configure r5 operation mode.
> > > > >
> > > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > > >
> > > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > > >
> > > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > > >
> > >
> > > That is exactly what I am looking for. That way people don't have to go through
> > > the entire driver trying to figure out what is happening with pm_dev_core[0].
> > >
> > > I'm also not sure about the power-up order. Logically the TCMs should be
> > > powered up before the R5 in order to put code in them. The R5s are powered in
> > > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > > TCMs are powered - can you expand on that?
> >
> >
> > Sure. Following is call sequece
> >
> > zynqmp_r5_rproc_prepare
> >
> > zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
> >
> > . . .
> >
> > zynqmp_r5_rproc_start -> load firmware and Starts RPU
> >
> > So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
> >
> >
> > >
> > > > revision. For new platforms pm_dev_core0[0] will be added in future.
> > >
> > > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > > future"?
> >
> >
> > ZynqMP platform has platform management firmware running on microblaze.
> >
> > This firmware design does not expect R5 pm domains to be requested explicitly.
> >
> > This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
> >
> > firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
> >
> > However, this design was changed for new platforms i.e. "versal" and onwards.
> >
> > Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
> >
> > waking them up.
> >
> > That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
> >
> > Then, we should wake it up on r5_core.
> >
> > To summarize:
> >
> > For zynqmp only following call needed to start R5:
> >
> > zynqmp_pm_request_wake
> >
> > For "versal" and onwards we need two calls to start R5:
> >
> > "device_link_add" and zynqmp_pm_request_wake
> >
> > So, in future pm_core_dev[0] will be used.
> >
>
> Thanks for the clarification on both front. The problem here is that we are
> keeping R5 power domain information in two different places, i.e
> zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
>
> Please see if you can retreive the power domain ID from
> zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
> calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
> zynqmp_r5_core::pm_domain_id.
Thanks for this suggestion. Let me find out if I can retrieve that or not.
I believe it should be possible.
> >
> > > >
> > > > I hope this meets expectations.
> > > >
> > > >
> > > > >
> > > > > > + */
> > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > > >
> > > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > > stick with it. I have no preference on which one.
> > > >
> > > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > > >
> > > >
> > > > >
> > > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > > + r5_core->pm_dev_core0[i],
> > > > > > + DL_FLAG_STATELESS |
> > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > + }
> > > > > > +
> > > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > > + return 0;
> > > > > > +
> > > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core1) {
> > > > > > + ret = -ENOMEM;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > +
> > > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device_link *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + ret = -ENOMEM;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > +
> > > > > > + /* get second core's device to detach its power-domains */
> > > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > > +
> > > > > > + pdev = of_find_device_by_node(np);
> > > > > > + if (!pdev) {
> > > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > +
> > > > > > + dev2 = &pdev->dev;
> > > > > > +
> > > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > + }
> > > > > > +
> > > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > > + r5_core->pm_dev_core1[j],
> > > > > > + DL_FLAG_STATELESS |
> > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > > + ret = -ENODEV;
> > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > + }
> > > > > > + }
> > > > > > +
> > > > > > +fail_add_pm_domains_lockstep:
> > > > > > + while (--j >= 0) {
> > > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > + }
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > +
> > > > > > +fail_add_pm_domains:
> > > > > > + while (--i >= 0) {
> > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > + }
> > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > +
> > > > >
> > > > > The error path is much cleaner and readable now.
> > > > >
> > > > > I will continue tomorrow.
> > > > >
> > > > > Mathieu
> > > > >
> > > > > > + return ret;
> > > > > > +}
> > > > > > +
> > > > > > /**
> > > > > > * zynqmp_r5_rproc_prepare()
> > > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > > {
> > > > > > int ret;
> > > > > >
> > > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > > + if (ret) {
> > > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > > + return ret;
> > > > > > + }
> > > > > > +
> > > > > > ret = add_tcm_banks(rproc);
> > > > > > if (ret) {
> > > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > > - return ret;
> > > > > > + goto fail_prepare;
> > > > > > }
> > > > > >
> > > > > > ret = add_mem_regions_carveout(rproc);
> > > > > > if (ret) {
> > > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > > - return ret;
> > > > > > + goto fail_prepare;
> > > > > > }
> > > > > >
> > > > > > return 0;
> > > > > > +
> > > > > > +fail_prepare:
> > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > +
> > > > > > + return ret;
> > > > > > }
> > > > > >
> > > > > > /**
> > > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > > >
> > > > > > r5_core = rproc->priv;
> > > > > >
> > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > +
> > > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > > --
> > > > > > 2.25.1
> > > > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-11-29 17:10 ` Mathieu Poirier
@ 2023-12-01 18:10 ` Tanmay Shah
-1 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-12-01 18:10 UTC (permalink / raw)
To: Mathieu Poirier
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On 11/29/23 11:10 AM, Mathieu Poirier wrote:
> On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
> >
> > On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > > Hi Mathieu,
> > > >
> > > > Please find my comments below.
> > > >
> > > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > > Hi,
> > > > >
> > > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > > Use TCM pm domains extracted from device-tree
> > > > > > to power on/off TCM using general pm domain framework.
> > > > > >
> > > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > > ---
> > > > > >
> > > > > > Changes in v7:
> > > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > > - remove pm_domain_id check to move next patch
> > > > > > - add comment about how 1st entry in pm domain list is used
> > > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > > >
> > > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > > >
> > > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > @@ -16,6 +16,7 @@
> > > > > > #include <linux/of_reserved_mem.h>
> > > > > > #include <linux/platform_device.h>
> > > > > > #include <linux/remoteproc.h>
> > > > > > +#include <linux/pm_domain.h>
> > > > > >
> > > > > > #include "remoteproc_internal.h"
> > > > > >
> > > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > > * @rproc: rproc handle
> > > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > > * @ipi: pointer to mailbox information
> > > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > > + * registration
> > > > > > */
> > > > > > struct zynqmp_r5_core {
> > > > > > struct device *dev;
> > > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > > struct rproc *rproc;
> > > > > > u32 pm_domain_id;
> > > > > > struct mbox_info *ipi;
> > > > > > + int num_pm_dev;
> > > > > > + struct device **pm_dev_core0;
> > > > > > + struct device_link **pm_dev_core0_link;
> > > > > > + struct device **pm_dev_core1;
> > > > > > + struct device_link **pm_dev_core1_link;
> > > > > > };
> > > > > >
> > > > > > /**
> > > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > > if (ret < 0) {
> > > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > > + pm_domain_id);
> > > > >
> > > > > Spurious change, you should have caught that.
> > > >
> > > > Ack, need to observe changes more closely before sending them.
> > > >
> > > > >
> > > > > > goto release_tcm_lockstep;
> > > > > > }
> > > > > >
> > > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > > return ret;
> > > > > > }
> > > > > >
> > > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > > +{
> > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > + struct device *dev = r5_core->dev;
> > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > + int i;
> > > > > > +
> > > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > > +
> > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > > + }
> > > > > > +
> > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > +
> > > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > > + r5_core->num_pm_dev = 0;
> > > > > > + return;
> > > > > > + }
> > > > > > +
> > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > > + }
> > > > > > +
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > + r5_core->num_pm_dev = 0;
> > > > > > +}
> > > > > > +
> > > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > > +{
> > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > + struct platform_device *pdev;
> > > > > > + struct device_node *np;
> > > > > > + int i, j, num_pm_dev, ret;
> > > > > > +
> > > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > > +
> > > > > > + /* get number of power-domains */
> > > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > > + "#power-domain-cells");
> > > > > > +
> > > > > > + if (num_pm_dev <= 0)
> > > > > > + return -EINVAL;
> > > > > > +
> > > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core0)
> > > > > > + ret = -ENOMEM;
> > > > > > +
> > > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device_link *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > + return -ENOMEM;
> > > > > > + }
> > > > > > +
> > > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > > +
> > > > > > + /*
> > > > > > + * start from 2nd entry in power-domains property list as
> > > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > > + * 1st entry is used to configure r5 operation mode.
> > > > >
> > > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > > >
> > > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > > >
> > > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > > >
> > >
> > > That is exactly what I am looking for. That way people don't have to go through
> > > the entire driver trying to figure out what is happening with pm_dev_core[0].
> > >
> > > I'm also not sure about the power-up order. Logically the TCMs should be
> > > powered up before the R5 in order to put code in them. The R5s are powered in
> > > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > > TCMs are powered - can you expand on that?
> >
> >
> > Sure. Following is call sequece
> >
> > zynqmp_r5_rproc_prepare
> >
> > zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
> >
> > . . .
> >
> > zynqmp_r5_rproc_start -> load firmware and Starts RPU
> >
> > So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
> >
> >
> > >
> > > > revision. For new platforms pm_dev_core0[0] will be added in future.
> > >
> > > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > > future"?
> >
> >
> > ZynqMP platform has platform management firmware running on microblaze.
> >
> > This firmware design does not expect R5 pm domains to be requested explicitly.
> >
> > This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
> >
> > firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
> >
> > However, this design was changed for new platforms i.e. "versal" and onwards.
> >
> > Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
> >
> > waking them up.
> >
> > That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
> >
> > Then, we should wake it up on r5_core.
> >
> > To summarize:
> >
> > For zynqmp only following call needed to start R5:
> >
> > zynqmp_pm_request_wake
> >
> > For "versal" and onwards we need two calls to start R5:
> >
> > "device_link_add" and zynqmp_pm_request_wake
> >
> > So, in future pm_core_dev[0] will be used.
> >
>
> Thanks for the clarification on both front. The problem here is that we are
> keeping R5 power domain information in two different places, i.e
> zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
>
> Please see if you can retreive the power domain ID from
> zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
> calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
> zynqmp_r5_core::pm_domain_id.
Hi Mathieu,
I looked into this. Probably I can't retrieve pm_domain_id from pm_dev_core0[0],
However, I can retrieve it from device-tree when calling zynqmp_pm_request_wake
and zynqmp_pm_force_pwrdwn.
zynqmp_r5_core::pm_domain_id is caching that value during probe, and use it during
rest of the driver lifecycle.
I am okay either way, (keeping it as it is, or get it from device-tree). Let me know your
preference.
Thanks,
Tanmay
> >
> > > >
> > > > I hope this meets expectations.
> > > >
> > > >
> > > > >
> > > > > > + */
> > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > > >
> > > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > > stick with it. I have no preference on which one.
> > > >
> > > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > > >
> > > >
> > > > >
> > > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > > + r5_core->pm_dev_core0[i],
> > > > > > + DL_FLAG_STATELESS |
> > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > + }
> > > > > > +
> > > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > > + return 0;
> > > > > > +
> > > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core1) {
> > > > > > + ret = -ENOMEM;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > +
> > > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device_link *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + ret = -ENOMEM;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > +
> > > > > > + /* get second core's device to detach its power-domains */
> > > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > > +
> > > > > > + pdev = of_find_device_by_node(np);
> > > > > > + if (!pdev) {
> > > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > +
> > > > > > + dev2 = &pdev->dev;
> > > > > > +
> > > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > + }
> > > > > > +
> > > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > > + r5_core->pm_dev_core1[j],
> > > > > > + DL_FLAG_STATELESS |
> > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > > + ret = -ENODEV;
> > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > + }
> > > > > > + }
> > > > > > +
> > > > > > +fail_add_pm_domains_lockstep:
> > > > > > + while (--j >= 0) {
> > > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > + }
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > +
> > > > > > +fail_add_pm_domains:
> > > > > > + while (--i >= 0) {
> > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > + }
> > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > +
> > > > >
> > > > > The error path is much cleaner and readable now.
> > > > >
> > > > > I will continue tomorrow.
> > > > >
> > > > > Mathieu
> > > > >
> > > > > > + return ret;
> > > > > > +}
> > > > > > +
> > > > > > /**
> > > > > > * zynqmp_r5_rproc_prepare()
> > > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > > {
> > > > > > int ret;
> > > > > >
> > > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > > + if (ret) {
> > > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > > + return ret;
> > > > > > + }
> > > > > > +
> > > > > > ret = add_tcm_banks(rproc);
> > > > > > if (ret) {
> > > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > > - return ret;
> > > > > > + goto fail_prepare;
> > > > > > }
> > > > > >
> > > > > > ret = add_mem_regions_carveout(rproc);
> > > > > > if (ret) {
> > > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > > - return ret;
> > > > > > + goto fail_prepare;
> > > > > > }
> > > > > >
> > > > > > return 0;
> > > > > > +
> > > > > > +fail_prepare:
> > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > +
> > > > > > + return ret;
> > > > > > }
> > > > > >
> > > > > > /**
> > > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > > >
> > > > > > r5_core = rproc->priv;
> > > > > >
> > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > +
> > > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > > --
> > > > > > 2.25.1
> > > > > >
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-12-01 18:10 ` Tanmay Shah
0 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-12-01 18:10 UTC (permalink / raw)
To: Mathieu Poirier
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On 11/29/23 11:10 AM, Mathieu Poirier wrote:
> On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
> >
> > On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > > Hi Mathieu,
> > > >
> > > > Please find my comments below.
> > > >
> > > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > > Hi,
> > > > >
> > > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > > Use TCM pm domains extracted from device-tree
> > > > > > to power on/off TCM using general pm domain framework.
> > > > > >
> > > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > > ---
> > > > > >
> > > > > > Changes in v7:
> > > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > > - remove pm_domain_id check to move next patch
> > > > > > - add comment about how 1st entry in pm domain list is used
> > > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > > >
> > > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > > >
> > > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > @@ -16,6 +16,7 @@
> > > > > > #include <linux/of_reserved_mem.h>
> > > > > > #include <linux/platform_device.h>
> > > > > > #include <linux/remoteproc.h>
> > > > > > +#include <linux/pm_domain.h>
> > > > > >
> > > > > > #include "remoteproc_internal.h"
> > > > > >
> > > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > > * @rproc: rproc handle
> > > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > > * @ipi: pointer to mailbox information
> > > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > > + * registration
> > > > > > */
> > > > > > struct zynqmp_r5_core {
> > > > > > struct device *dev;
> > > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > > struct rproc *rproc;
> > > > > > u32 pm_domain_id;
> > > > > > struct mbox_info *ipi;
> > > > > > + int num_pm_dev;
> > > > > > + struct device **pm_dev_core0;
> > > > > > + struct device_link **pm_dev_core0_link;
> > > > > > + struct device **pm_dev_core1;
> > > > > > + struct device_link **pm_dev_core1_link;
> > > > > > };
> > > > > >
> > > > > > /**
> > > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > > if (ret < 0) {
> > > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > > + pm_domain_id);
> > > > >
> > > > > Spurious change, you should have caught that.
> > > >
> > > > Ack, need to observe changes more closely before sending them.
> > > >
> > > > >
> > > > > > goto release_tcm_lockstep;
> > > > > > }
> > > > > >
> > > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > > return ret;
> > > > > > }
> > > > > >
> > > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > > +{
> > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > + struct device *dev = r5_core->dev;
> > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > + int i;
> > > > > > +
> > > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > > +
> > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > > + }
> > > > > > +
> > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > +
> > > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > > + r5_core->num_pm_dev = 0;
> > > > > > + return;
> > > > > > + }
> > > > > > +
> > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > > + }
> > > > > > +
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > + r5_core->num_pm_dev = 0;
> > > > > > +}
> > > > > > +
> > > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > > +{
> > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > + struct platform_device *pdev;
> > > > > > + struct device_node *np;
> > > > > > + int i, j, num_pm_dev, ret;
> > > > > > +
> > > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > > +
> > > > > > + /* get number of power-domains */
> > > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > > + "#power-domain-cells");
> > > > > > +
> > > > > > + if (num_pm_dev <= 0)
> > > > > > + return -EINVAL;
> > > > > > +
> > > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core0)
> > > > > > + ret = -ENOMEM;
> > > > > > +
> > > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device_link *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > + return -ENOMEM;
> > > > > > + }
> > > > > > +
> > > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > > +
> > > > > > + /*
> > > > > > + * start from 2nd entry in power-domains property list as
> > > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > > + * 1st entry is used to configure r5 operation mode.
> > > > >
> > > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > > >
> > > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > > >
> > > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > > >
> > >
> > > That is exactly what I am looking for. That way people don't have to go through
> > > the entire driver trying to figure out what is happening with pm_dev_core[0].
> > >
> > > I'm also not sure about the power-up order. Logically the TCMs should be
> > > powered up before the R5 in order to put code in them. The R5s are powered in
> > > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > > TCMs are powered - can you expand on that?
> >
> >
> > Sure. Following is call sequece
> >
> > zynqmp_r5_rproc_prepare
> >
> > zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
> >
> > . . .
> >
> > zynqmp_r5_rproc_start -> load firmware and Starts RPU
> >
> > So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
> >
> >
> > >
> > > > revision. For new platforms pm_dev_core0[0] will be added in future.
> > >
> > > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > > future"?
> >
> >
> > ZynqMP platform has platform management firmware running on microblaze.
> >
> > This firmware design does not expect R5 pm domains to be requested explicitly.
> >
> > This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
> >
> > firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
> >
> > However, this design was changed for new platforms i.e. "versal" and onwards.
> >
> > Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
> >
> > waking them up.
> >
> > That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
> >
> > Then, we should wake it up on r5_core.
> >
> > To summarize:
> >
> > For zynqmp only following call needed to start R5:
> >
> > zynqmp_pm_request_wake
> >
> > For "versal" and onwards we need two calls to start R5:
> >
> > "device_link_add" and zynqmp_pm_request_wake
> >
> > So, in future pm_core_dev[0] will be used.
> >
>
> Thanks for the clarification on both front. The problem here is that we are
> keeping R5 power domain information in two different places, i.e
> zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
>
> Please see if you can retreive the power domain ID from
> zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
> calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
> zynqmp_r5_core::pm_domain_id.
Hi Mathieu,
I looked into this. Probably I can't retrieve pm_domain_id from pm_dev_core0[0],
However, I can retrieve it from device-tree when calling zynqmp_pm_request_wake
and zynqmp_pm_force_pwrdwn.
zynqmp_r5_core::pm_domain_id is caching that value during probe, and use it during
rest of the driver lifecycle.
I am okay either way, (keeping it as it is, or get it from device-tree). Let me know your
preference.
Thanks,
Tanmay
> >
> > > >
> > > > I hope this meets expectations.
> > > >
> > > >
> > > > >
> > > > > > + */
> > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > > >
> > > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > > stick with it. I have no preference on which one.
> > > >
> > > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > > >
> > > >
> > > > >
> > > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > > + r5_core->pm_dev_core0[i],
> > > > > > + DL_FLAG_STATELESS |
> > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > + }
> > > > > > +
> > > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > > + return 0;
> > > > > > +
> > > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core1) {
> > > > > > + ret = -ENOMEM;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > +
> > > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > > + sizeof(struct device_link *),
> > > > > > + GFP_KERNEL);
> > > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + ret = -ENOMEM;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > +
> > > > > > + /* get second core's device to detach its power-domains */
> > > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > > +
> > > > > > + pdev = of_find_device_by_node(np);
> > > > > > + if (!pdev) {
> > > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains;
> > > > > > + }
> > > > > > +
> > > > > > + dev2 = &pdev->dev;
> > > > > > +
> > > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > + ret = -EINVAL;
> > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > + }
> > > > > > +
> > > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > > + r5_core->pm_dev_core1[j],
> > > > > > + DL_FLAG_STATELESS |
> > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > > + ret = -ENODEV;
> > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > + }
> > > > > > + }
> > > > > > +
> > > > > > +fail_add_pm_domains_lockstep:
> > > > > > + while (--j >= 0) {
> > > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > + }
> > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > +
> > > > > > +fail_add_pm_domains:
> > > > > > + while (--i >= 0) {
> > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > + }
> > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > +
> > > > >
> > > > > The error path is much cleaner and readable now.
> > > > >
> > > > > I will continue tomorrow.
> > > > >
> > > > > Mathieu
> > > > >
> > > > > > + return ret;
> > > > > > +}
> > > > > > +
> > > > > > /**
> > > > > > * zynqmp_r5_rproc_prepare()
> > > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > > {
> > > > > > int ret;
> > > > > >
> > > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > > + if (ret) {
> > > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > > + return ret;
> > > > > > + }
> > > > > > +
> > > > > > ret = add_tcm_banks(rproc);
> > > > > > if (ret) {
> > > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > > - return ret;
> > > > > > + goto fail_prepare;
> > > > > > }
> > > > > >
> > > > > > ret = add_mem_regions_carveout(rproc);
> > > > > > if (ret) {
> > > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > > - return ret;
> > > > > > + goto fail_prepare;
> > > > > > }
> > > > > >
> > > > > > return 0;
> > > > > > +
> > > > > > +fail_prepare:
> > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > +
> > > > > > + return ret;
> > > > > > }
> > > > > >
> > > > > > /**
> > > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > > >
> > > > > > r5_core = rproc->priv;
> > > > > >
> > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > +
> > > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > > --
> > > > > > 2.25.1
> > > > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-12-01 18:10 ` Tanmay Shah
@ 2023-12-06 15:43 ` Mathieu Poirier
-1 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-12-06 15:43 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Fri, 1 Dec 2023 at 11:10, Tanmay Shah <tanmay.shah@amd.com> wrote:
>
>
> On 11/29/23 11:10 AM, Mathieu Poirier wrote:
> > On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
> > >
> > > On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > > > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > > > Hi Mathieu,
> > > > >
> > > > > Please find my comments below.
> > > > >
> > > > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > > > Hi,
> > > > > >
> > > > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > > > Use TCM pm domains extracted from device-tree
> > > > > > > to power on/off TCM using general pm domain framework.
> > > > > > >
> > > > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > > > ---
> > > > > > >
> > > > > > > Changes in v7:
> > > > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > > > - remove pm_domain_id check to move next patch
> > > > > > > - add comment about how 1st entry in pm domain list is used
> > > > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > > > >
> > > > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > > > >
> > > > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > @@ -16,6 +16,7 @@
> > > > > > > #include <linux/of_reserved_mem.h>
> > > > > > > #include <linux/platform_device.h>
> > > > > > > #include <linux/remoteproc.h>
> > > > > > > +#include <linux/pm_domain.h>
> > > > > > >
> > > > > > > #include "remoteproc_internal.h"
> > > > > > >
> > > > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > > > * @rproc: rproc handle
> > > > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > > > * @ipi: pointer to mailbox information
> > > > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > > > + * registration
> > > > > > > */
> > > > > > > struct zynqmp_r5_core {
> > > > > > > struct device *dev;
> > > > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > > > struct rproc *rproc;
> > > > > > > u32 pm_domain_id;
> > > > > > > struct mbox_info *ipi;
> > > > > > > + int num_pm_dev;
> > > > > > > + struct device **pm_dev_core0;
> > > > > > > + struct device_link **pm_dev_core0_link;
> > > > > > > + struct device **pm_dev_core1;
> > > > > > > + struct device_link **pm_dev_core1_link;
> > > > > > > };
> > > > > > >
> > > > > > > /**
> > > > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > > > if (ret < 0) {
> > > > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > > > + pm_domain_id);
> > > > > >
> > > > > > Spurious change, you should have caught that.
> > > > >
> > > > > Ack, need to observe changes more closely before sending them.
> > > > >
> > > > > >
> > > > > > > goto release_tcm_lockstep;
> > > > > > > }
> > > > > > >
> > > > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > > > return ret;
> > > > > > > }
> > > > > > >
> > > > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > > > +{
> > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > + struct device *dev = r5_core->dev;
> > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > + int i;
> > > > > > > +
> > > > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > > > +
> > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > > > + }
> > > > > > > +
> > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > +
> > > > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > + return;
> > > > > > > + }
> > > > > > > +
> > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > > > + }
> > > > > > > +
> > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > +}
> > > > > > > +
> > > > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > > > +{
> > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > + struct platform_device *pdev;
> > > > > > > + struct device_node *np;
> > > > > > > + int i, j, num_pm_dev, ret;
> > > > > > > +
> > > > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > > > +
> > > > > > > + /* get number of power-domains */
> > > > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > > > + "#power-domain-cells");
> > > > > > > +
> > > > > > > + if (num_pm_dev <= 0)
> > > > > > > + return -EINVAL;
> > > > > > > +
> > > > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > > > + sizeof(struct device *),
> > > > > > > + GFP_KERNEL);
> > > > > > > + if (!r5_core->pm_dev_core0)
> > > > > > > + ret = -ENOMEM;
> > > > > > > +
> > > > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > > > + sizeof(struct device_link *),
> > > > > > > + GFP_KERNEL);
> > > > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > + return -ENOMEM;
> > > > > > > + }
> > > > > > > +
> > > > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > > > +
> > > > > > > + /*
> > > > > > > + * start from 2nd entry in power-domains property list as
> > > > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > > > + * 1st entry is used to configure r5 operation mode.
> > > > > >
> > > > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > > > >
> > > > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > > > >
> > > > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > > > >
> > > >
> > > > That is exactly what I am looking for. That way people don't have to go through
> > > > the entire driver trying to figure out what is happening with pm_dev_core[0].
> > > >
> > > > I'm also not sure about the power-up order. Logically the TCMs should be
> > > > powered up before the R5 in order to put code in them. The R5s are powered in
> > > > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > > > TCMs are powered - can you expand on that?
> > >
> > >
> > > Sure. Following is call sequece
> > >
> > > zynqmp_r5_rproc_prepare
> > >
> > > zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
> > >
> > > . . .
> > >
> > > zynqmp_r5_rproc_start -> load firmware and Starts RPU
> > >
> > > So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
> > >
> > >
> > > >
> > > > > revision. For new platforms pm_dev_core0[0] will be added in future.
> > > >
> > > > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > > > future"?
> > >
> > >
> > > ZynqMP platform has platform management firmware running on microblaze.
> > >
> > > This firmware design does not expect R5 pm domains to be requested explicitly.
> > >
> > > This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
> > >
> > > firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
> > >
> > > However, this design was changed for new platforms i.e. "versal" and onwards.
> > >
> > > Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
> > >
> > > waking them up.
> > >
> > > That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
> > >
> > > Then, we should wake it up on r5_core.
> > >
> > > To summarize:
> > >
> > > For zynqmp only following call needed to start R5:
> > >
> > > zynqmp_pm_request_wake
> > >
> > > For "versal" and onwards we need two calls to start R5:
> > >
> > > "device_link_add" and zynqmp_pm_request_wake
> > >
> > > So, in future pm_core_dev[0] will be used.
> > >
> >
> > Thanks for the clarification on both front. The problem here is that we are
> > keeping R5 power domain information in two different places, i.e
> > zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
> >
> > Please see if you can retreive the power domain ID from
> > zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
> > calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
> > zynqmp_r5_core::pm_domain_id.
>
> Hi Mathieu,
>
> I looked into this. Probably I can't retrieve pm_domain_id from pm_dev_core0[0],
>
> However, I can retrieve it from device-tree when calling zynqmp_pm_request_wake
>
> and zynqmp_pm_force_pwrdwn.
>
> zynqmp_r5_core::pm_domain_id is caching that value during probe, and use it during
> rest of the driver lifecycle.
>
> I am okay either way, (keeping it as it is, or get it from device-tree). Let me know your
>
> preference.
>
Humm... Then I suggest to simply get rid of the device linking to
deal with the TCMs' power management. From where I stand it provides
more confusion than benefits, and that is without considering the
extra complexity.
> Thanks,
>
> Tanmay
>
> > >
> > > > >
> > > > > I hope this meets expectations.
> > > > >
> > > > >
> > > > > >
> > > > > > > + */
> > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > > > >
> > > > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > > > stick with it. I have no preference on which one.
> > > > >
> > > > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > > > >
> > > > >
> > > > > >
> > > > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > > > + ret = -EINVAL;
> > > > > > > + goto fail_add_pm_domains;
> > > > > > > + }
> > > > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > > > + r5_core->pm_dev_core0[i],
> > > > > > > + DL_FLAG_STATELESS |
> > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > > > + ret = -EINVAL;
> > > > > > > + goto fail_add_pm_domains;
> > > > > > > + }
> > > > > > > + }
> > > > > > > +
> > > > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > > > + return 0;
> > > > > > > +
> > > > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > > > + sizeof(struct device *),
> > > > > > > + GFP_KERNEL);
> > > > > > > + if (!r5_core->pm_dev_core1) {
> > > > > > > + ret = -ENOMEM;
> > > > > > > + goto fail_add_pm_domains;
> > > > > > > + }
> > > > > > > +
> > > > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > > > + sizeof(struct device_link *),
> > > > > > > + GFP_KERNEL);
> > > > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > + ret = -ENOMEM;
> > > > > > > + goto fail_add_pm_domains;
> > > > > > > + }
> > > > > > > +
> > > > > > > + /* get second core's device to detach its power-domains */
> > > > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > > > +
> > > > > > > + pdev = of_find_device_by_node(np);
> > > > > > > + if (!pdev) {
> > > > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > + ret = -EINVAL;
> > > > > > > + goto fail_add_pm_domains;
> > > > > > > + }
> > > > > > > +
> > > > > > > + dev2 = &pdev->dev;
> > > > > > > +
> > > > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > + ret = -EINVAL;
> > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > + }
> > > > > > > +
> > > > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > > > + r5_core->pm_dev_core1[j],
> > > > > > > + DL_FLAG_STATELESS |
> > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > > > + ret = -ENODEV;
> > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > + }
> > > > > > > + }
> > > > > > > +
> > > > > > > +fail_add_pm_domains_lockstep:
> > > > > > > + while (--j >= 0) {
> > > > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > + }
> > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > +
> > > > > > > +fail_add_pm_domains:
> > > > > > > + while (--i >= 0) {
> > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > + }
> > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > +
> > > > > >
> > > > > > The error path is much cleaner and readable now.
> > > > > >
> > > > > > I will continue tomorrow.
> > > > > >
> > > > > > Mathieu
> > > > > >
> > > > > > > + return ret;
> > > > > > > +}
> > > > > > > +
> > > > > > > /**
> > > > > > > * zynqmp_r5_rproc_prepare()
> > > > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > > > {
> > > > > > > int ret;
> > > > > > >
> > > > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > > > + if (ret) {
> > > > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > > > + return ret;
> > > > > > > + }
> > > > > > > +
> > > > > > > ret = add_tcm_banks(rproc);
> > > > > > > if (ret) {
> > > > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > > > - return ret;
> > > > > > > + goto fail_prepare;
> > > > > > > }
> > > > > > >
> > > > > > > ret = add_mem_regions_carveout(rproc);
> > > > > > > if (ret) {
> > > > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > > > - return ret;
> > > > > > > + goto fail_prepare;
> > > > > > > }
> > > > > > >
> > > > > > > return 0;
> > > > > > > +
> > > > > > > +fail_prepare:
> > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > +
> > > > > > > + return ret;
> > > > > > > }
> > > > > > >
> > > > > > > /**
> > > > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > > > >
> > > > > > > r5_core = rproc->priv;
> > > > > > >
> > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > +
> > > > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > > > --
> > > > > > > 2.25.1
> > > > > > >
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-12-06 15:43 ` Mathieu Poirier
0 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-12-06 15:43 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Fri, 1 Dec 2023 at 11:10, Tanmay Shah <tanmay.shah@amd.com> wrote:
>
>
> On 11/29/23 11:10 AM, Mathieu Poirier wrote:
> > On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
> > >
> > > On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > > > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > > > Hi Mathieu,
> > > > >
> > > > > Please find my comments below.
> > > > >
> > > > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > > > Hi,
> > > > > >
> > > > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > > > Use TCM pm domains extracted from device-tree
> > > > > > > to power on/off TCM using general pm domain framework.
> > > > > > >
> > > > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > > > ---
> > > > > > >
> > > > > > > Changes in v7:
> > > > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > > > - remove pm_domain_id check to move next patch
> > > > > > > - add comment about how 1st entry in pm domain list is used
> > > > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > > > >
> > > > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > > > >
> > > > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > @@ -16,6 +16,7 @@
> > > > > > > #include <linux/of_reserved_mem.h>
> > > > > > > #include <linux/platform_device.h>
> > > > > > > #include <linux/remoteproc.h>
> > > > > > > +#include <linux/pm_domain.h>
> > > > > > >
> > > > > > > #include "remoteproc_internal.h"
> > > > > > >
> > > > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > > > * @rproc: rproc handle
> > > > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > > > * @ipi: pointer to mailbox information
> > > > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > > > + * registration
> > > > > > > */
> > > > > > > struct zynqmp_r5_core {
> > > > > > > struct device *dev;
> > > > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > > > struct rproc *rproc;
> > > > > > > u32 pm_domain_id;
> > > > > > > struct mbox_info *ipi;
> > > > > > > + int num_pm_dev;
> > > > > > > + struct device **pm_dev_core0;
> > > > > > > + struct device_link **pm_dev_core0_link;
> > > > > > > + struct device **pm_dev_core1;
> > > > > > > + struct device_link **pm_dev_core1_link;
> > > > > > > };
> > > > > > >
> > > > > > > /**
> > > > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > > > if (ret < 0) {
> > > > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > > > + pm_domain_id);
> > > > > >
> > > > > > Spurious change, you should have caught that.
> > > > >
> > > > > Ack, need to observe changes more closely before sending them.
> > > > >
> > > > > >
> > > > > > > goto release_tcm_lockstep;
> > > > > > > }
> > > > > > >
> > > > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > > > return ret;
> > > > > > > }
> > > > > > >
> > > > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > > > +{
> > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > + struct device *dev = r5_core->dev;
> > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > + int i;
> > > > > > > +
> > > > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > > > +
> > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > > > + }
> > > > > > > +
> > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > +
> > > > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > + return;
> > > > > > > + }
> > > > > > > +
> > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > > > + }
> > > > > > > +
> > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > +}
> > > > > > > +
> > > > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > > > +{
> > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > + struct platform_device *pdev;
> > > > > > > + struct device_node *np;
> > > > > > > + int i, j, num_pm_dev, ret;
> > > > > > > +
> > > > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > > > +
> > > > > > > + /* get number of power-domains */
> > > > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > > > + "#power-domain-cells");
> > > > > > > +
> > > > > > > + if (num_pm_dev <= 0)
> > > > > > > + return -EINVAL;
> > > > > > > +
> > > > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > > > + sizeof(struct device *),
> > > > > > > + GFP_KERNEL);
> > > > > > > + if (!r5_core->pm_dev_core0)
> > > > > > > + ret = -ENOMEM;
> > > > > > > +
> > > > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > > > + sizeof(struct device_link *),
> > > > > > > + GFP_KERNEL);
> > > > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > + return -ENOMEM;
> > > > > > > + }
> > > > > > > +
> > > > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > > > +
> > > > > > > + /*
> > > > > > > + * start from 2nd entry in power-domains property list as
> > > > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > > > + * 1st entry is used to configure r5 operation mode.
> > > > > >
> > > > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > > > >
> > > > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > > > >
> > > > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > > > >
> > > >
> > > > That is exactly what I am looking for. That way people don't have to go through
> > > > the entire driver trying to figure out what is happening with pm_dev_core[0].
> > > >
> > > > I'm also not sure about the power-up order. Logically the TCMs should be
> > > > powered up before the R5 in order to put code in them. The R5s are powered in
> > > > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > > > TCMs are powered - can you expand on that?
> > >
> > >
> > > Sure. Following is call sequece
> > >
> > > zynqmp_r5_rproc_prepare
> > >
> > > zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
> > >
> > > . . .
> > >
> > > zynqmp_r5_rproc_start -> load firmware and Starts RPU
> > >
> > > So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
> > >
> > >
> > > >
> > > > > revision. For new platforms pm_dev_core0[0] will be added in future.
> > > >
> > > > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > > > future"?
> > >
> > >
> > > ZynqMP platform has platform management firmware running on microblaze.
> > >
> > > This firmware design does not expect R5 pm domains to be requested explicitly.
> > >
> > > This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
> > >
> > > firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
> > >
> > > However, this design was changed for new platforms i.e. "versal" and onwards.
> > >
> > > Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
> > >
> > > waking them up.
> > >
> > > That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
> > >
> > > Then, we should wake it up on r5_core.
> > >
> > > To summarize:
> > >
> > > For zynqmp only following call needed to start R5:
> > >
> > > zynqmp_pm_request_wake
> > >
> > > For "versal" and onwards we need two calls to start R5:
> > >
> > > "device_link_add" and zynqmp_pm_request_wake
> > >
> > > So, in future pm_core_dev[0] will be used.
> > >
> >
> > Thanks for the clarification on both front. The problem here is that we are
> > keeping R5 power domain information in two different places, i.e
> > zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
> >
> > Please see if you can retreive the power domain ID from
> > zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
> > calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
> > zynqmp_r5_core::pm_domain_id.
>
> Hi Mathieu,
>
> I looked into this. Probably I can't retrieve pm_domain_id from pm_dev_core0[0],
>
> However, I can retrieve it from device-tree when calling zynqmp_pm_request_wake
>
> and zynqmp_pm_force_pwrdwn.
>
> zynqmp_r5_core::pm_domain_id is caching that value during probe, and use it during
> rest of the driver lifecycle.
>
> I am okay either way, (keeping it as it is, or get it from device-tree). Let me know your
>
> preference.
>
Humm... Then I suggest to simply get rid of the device linking to
deal with the TCMs' power management. From where I stand it provides
more confusion than benefits, and that is without considering the
extra complexity.
> Thanks,
>
> Tanmay
>
> > >
> > > > >
> > > > > I hope this meets expectations.
> > > > >
> > > > >
> > > > > >
> > > > > > > + */
> > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > > > >
> > > > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > > > stick with it. I have no preference on which one.
> > > > >
> > > > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > > > >
> > > > >
> > > > > >
> > > > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > > > + ret = -EINVAL;
> > > > > > > + goto fail_add_pm_domains;
> > > > > > > + }
> > > > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > > > + r5_core->pm_dev_core0[i],
> > > > > > > + DL_FLAG_STATELESS |
> > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > > > + ret = -EINVAL;
> > > > > > > + goto fail_add_pm_domains;
> > > > > > > + }
> > > > > > > + }
> > > > > > > +
> > > > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > > > + return 0;
> > > > > > > +
> > > > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > > > + sizeof(struct device *),
> > > > > > > + GFP_KERNEL);
> > > > > > > + if (!r5_core->pm_dev_core1) {
> > > > > > > + ret = -ENOMEM;
> > > > > > > + goto fail_add_pm_domains;
> > > > > > > + }
> > > > > > > +
> > > > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > > > + sizeof(struct device_link *),
> > > > > > > + GFP_KERNEL);
> > > > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > + ret = -ENOMEM;
> > > > > > > + goto fail_add_pm_domains;
> > > > > > > + }
> > > > > > > +
> > > > > > > + /* get second core's device to detach its power-domains */
> > > > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > > > +
> > > > > > > + pdev = of_find_device_by_node(np);
> > > > > > > + if (!pdev) {
> > > > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > + ret = -EINVAL;
> > > > > > > + goto fail_add_pm_domains;
> > > > > > > + }
> > > > > > > +
> > > > > > > + dev2 = &pdev->dev;
> > > > > > > +
> > > > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > + ret = -EINVAL;
> > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > + }
> > > > > > > +
> > > > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > > > + r5_core->pm_dev_core1[j],
> > > > > > > + DL_FLAG_STATELESS |
> > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > > > + ret = -ENODEV;
> > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > + }
> > > > > > > + }
> > > > > > > +
> > > > > > > +fail_add_pm_domains_lockstep:
> > > > > > > + while (--j >= 0) {
> > > > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > + }
> > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > +
> > > > > > > +fail_add_pm_domains:
> > > > > > > + while (--i >= 0) {
> > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > + }
> > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > +
> > > > > >
> > > > > > The error path is much cleaner and readable now.
> > > > > >
> > > > > > I will continue tomorrow.
> > > > > >
> > > > > > Mathieu
> > > > > >
> > > > > > > + return ret;
> > > > > > > +}
> > > > > > > +
> > > > > > > /**
> > > > > > > * zynqmp_r5_rproc_prepare()
> > > > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > > > {
> > > > > > > int ret;
> > > > > > >
> > > > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > > > + if (ret) {
> > > > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > > > + return ret;
> > > > > > > + }
> > > > > > > +
> > > > > > > ret = add_tcm_banks(rproc);
> > > > > > > if (ret) {
> > > > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > > > - return ret;
> > > > > > > + goto fail_prepare;
> > > > > > > }
> > > > > > >
> > > > > > > ret = add_mem_regions_carveout(rproc);
> > > > > > > if (ret) {
> > > > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > > > - return ret;
> > > > > > > + goto fail_prepare;
> > > > > > > }
> > > > > > >
> > > > > > > return 0;
> > > > > > > +
> > > > > > > +fail_prepare:
> > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > +
> > > > > > > + return ret;
> > > > > > > }
> > > > > > >
> > > > > > > /**
> > > > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > > > >
> > > > > > > r5_core = rproc->priv;
> > > > > > >
> > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > +
> > > > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > > > --
> > > > > > > 2.25.1
> > > > > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-12-06 15:43 ` Mathieu Poirier
@ 2023-12-06 18:06 ` Tanmay Shah
-1 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-12-06 18:06 UTC (permalink / raw)
To: Mathieu Poirier
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On 12/6/23 9:43 AM, Mathieu Poirier wrote:
> On Fri, 1 Dec 2023 at 11:10, Tanmay Shah <tanmay.shah@amd.com> wrote:
> >
> >
> > On 11/29/23 11:10 AM, Mathieu Poirier wrote:
> > > On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
> > > >
> > > > On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > > > > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > > > > Hi Mathieu,
> > > > > >
> > > > > > Please find my comments below.
> > > > > >
> > > > > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > > > > Use TCM pm domains extracted from device-tree
> > > > > > > > to power on/off TCM using general pm domain framework.
> > > > > > > >
> > > > > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > > > > ---
> > > > > > > >
> > > > > > > > Changes in v7:
> > > > > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > > > > - remove pm_domain_id check to move next patch
> > > > > > > > - add comment about how 1st entry in pm domain list is used
> > > > > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > > > > >
> > > > > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > > > > >
> > > > > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > @@ -16,6 +16,7 @@
> > > > > > > > #include <linux/of_reserved_mem.h>
> > > > > > > > #include <linux/platform_device.h>
> > > > > > > > #include <linux/remoteproc.h>
> > > > > > > > +#include <linux/pm_domain.h>
> > > > > > > >
> > > > > > > > #include "remoteproc_internal.h"
> > > > > > > >
> > > > > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > > > > * @rproc: rproc handle
> > > > > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > > > > * @ipi: pointer to mailbox information
> > > > > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > > > > + * registration
> > > > > > > > */
> > > > > > > > struct zynqmp_r5_core {
> > > > > > > > struct device *dev;
> > > > > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > > > > struct rproc *rproc;
> > > > > > > > u32 pm_domain_id;
> > > > > > > > struct mbox_info *ipi;
> > > > > > > > + int num_pm_dev;
> > > > > > > > + struct device **pm_dev_core0;
> > > > > > > > + struct device_link **pm_dev_core0_link;
> > > > > > > > + struct device **pm_dev_core1;
> > > > > > > > + struct device_link **pm_dev_core1_link;
> > > > > > > > };
> > > > > > > >
> > > > > > > > /**
> > > > > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > > > > if (ret < 0) {
> > > > > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > > > > + pm_domain_id);
> > > > > > >
> > > > > > > Spurious change, you should have caught that.
> > > > > >
> > > > > > Ack, need to observe changes more closely before sending them.
> > > > > >
> > > > > > >
> > > > > > > > goto release_tcm_lockstep;
> > > > > > > > }
> > > > > > > >
> > > > > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > > > > return ret;
> > > > > > > > }
> > > > > > > >
> > > > > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > > > > +{
> > > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > > + struct device *dev = r5_core->dev;
> > > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > > + int i;
> > > > > > > > +
> > > > > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > > > > +
> > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > > +
> > > > > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > > + return;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > > > > +{
> > > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > > + struct platform_device *pdev;
> > > > > > > > + struct device_node *np;
> > > > > > > > + int i, j, num_pm_dev, ret;
> > > > > > > > +
> > > > > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > > > > +
> > > > > > > > + /* get number of power-domains */
> > > > > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > > > > + "#power-domain-cells");
> > > > > > > > +
> > > > > > > > + if (num_pm_dev <= 0)
> > > > > > > > + return -EINVAL;
> > > > > > > > +
> > > > > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > > > > + sizeof(struct device *),
> > > > > > > > + GFP_KERNEL);
> > > > > > > > + if (!r5_core->pm_dev_core0)
> > > > > > > > + ret = -ENOMEM;
> > > > > > > > +
> > > > > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > > > > + sizeof(struct device_link *),
> > > > > > > > + GFP_KERNEL);
> > > > > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > + return -ENOMEM;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > > > > +
> > > > > > > > + /*
> > > > > > > > + * start from 2nd entry in power-domains property list as
> > > > > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > > > > + * 1st entry is used to configure r5 operation mode.
> > > > > > >
> > > > > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > > > > >
> > > > > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > > > > >
> > > > > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > > > > >
> > > > >
> > > > > That is exactly what I am looking for. That way people don't have to go through
> > > > > the entire driver trying to figure out what is happening with pm_dev_core[0].
> > > > >
> > > > > I'm also not sure about the power-up order. Logically the TCMs should be
> > > > > powered up before the R5 in order to put code in them. The R5s are powered in
> > > > > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > > > > TCMs are powered - can you expand on that?
> > > >
> > > >
> > > > Sure. Following is call sequece
> > > >
> > > > zynqmp_r5_rproc_prepare
> > > >
> > > > zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
> > > >
> > > > . . .
> > > >
> > > > zynqmp_r5_rproc_start -> load firmware and Starts RPU
> > > >
> > > > So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
> > > >
> > > >
> > > > >
> > > > > > revision. For new platforms pm_dev_core0[0] will be added in future.
> > > > >
> > > > > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > > > > future"?
> > > >
> > > >
> > > > ZynqMP platform has platform management firmware running on microblaze.
> > > >
> > > > This firmware design does not expect R5 pm domains to be requested explicitly.
> > > >
> > > > This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
> > > >
> > > > firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
> > > >
> > > > However, this design was changed for new platforms i.e. "versal" and onwards.
> > > >
> > > > Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
> > > >
> > > > waking them up.
> > > >
> > > > That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
> > > >
> > > > Then, we should wake it up on r5_core.
> > > >
> > > > To summarize:
> > > >
> > > > For zynqmp only following call needed to start R5:
> > > >
> > > > zynqmp_pm_request_wake
> > > >
> > > > For "versal" and onwards we need two calls to start R5:
> > > >
> > > > "device_link_add" and zynqmp_pm_request_wake
> > > >
> > > > So, in future pm_core_dev[0] will be used.
> > > >
> > >
> > > Thanks for the clarification on both front. The problem here is that we are
> > > keeping R5 power domain information in two different places, i.e
> > > zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
> > >
> > > Please see if you can retreive the power domain ID from
> > > zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
> > > calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
> > > zynqmp_r5_core::pm_domain_id.
> >
> > Hi Mathieu,
> >
> > I looked into this. Probably I can't retrieve pm_domain_id from pm_dev_core0[0],
> >
> > However, I can retrieve it from device-tree when calling zynqmp_pm_request_wake
> >
> > and zynqmp_pm_force_pwrdwn.
> >
> > zynqmp_r5_core::pm_domain_id is caching that value during probe, and use it during
> > rest of the driver lifecycle.
> >
> > I am okay either way, (keeping it as it is, or get it from device-tree). Let me know your
> >
> > preference.
> >
>
> Humm... Then I suggest to simply get rid of the device linking to
> deal with the TCMs' power management. From where I stand it provides
> more confusion than benefits, and that is without considering the
> extra complexity.
Do you mean to get rid of pm_dev_core0[1], and pm_dev_core0[2] as well ?
If yes, its preferred to use pm_domain framework to power-on/off TCM.
If we want to get rid of zynqmp_r5_core::pm_domain_id, I will do what's done in
__genpd_dev_pm_attach API where, pm_domain_id is retrieved using of_node of pm_dev_core0[*] device.
ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
"#power-domain-cells", index, &pd_args);
However, Its preferred to use pm_domain framework when power-domains are available in device-tree.
Let me know.
Thanks,
Tanmay
> > Thanks,
> >
> > Tanmay
> >
> > > >
> > > > > >
> > > > > > I hope this meets expectations.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > > + */
> > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > > > > >
> > > > > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > > > > stick with it. I have no preference on which one.
> > > > > >
> > > > > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > > > > + ret = -EINVAL;
> > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > + }
> > > > > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > > > > + r5_core->pm_dev_core0[i],
> > > > > > > > + DL_FLAG_STATELESS |
> > > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > > > > + ret = -EINVAL;
> > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > + }
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > > > > + return 0;
> > > > > > > > +
> > > > > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > > > > + sizeof(struct device *),
> > > > > > > > + GFP_KERNEL);
> > > > > > > > + if (!r5_core->pm_dev_core1) {
> > > > > > > > + ret = -ENOMEM;
> > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > > > > + sizeof(struct device_link *),
> > > > > > > > + GFP_KERNEL);
> > > > > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > + ret = -ENOMEM;
> > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + /* get second core's device to detach its power-domains */
> > > > > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > > > > +
> > > > > > > > + pdev = of_find_device_by_node(np);
> > > > > > > > + if (!pdev) {
> > > > > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > + ret = -EINVAL;
> > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + dev2 = &pdev->dev;
> > > > > > > > +
> > > > > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > > + ret = -EINVAL;
> > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > > > > + r5_core->pm_dev_core1[j],
> > > > > > > > + DL_FLAG_STATELESS |
> > > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > > > > + ret = -ENODEV;
> > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > + }
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > +fail_add_pm_domains_lockstep:
> > > > > > > > + while (--j >= 0) {
> > > > > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > > + }
> > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > +
> > > > > > > > +fail_add_pm_domains:
> > > > > > > > + while (--i >= 0) {
> > > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > > + }
> > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > > +
> > > > > > >
> > > > > > > The error path is much cleaner and readable now.
> > > > > > >
> > > > > > > I will continue tomorrow.
> > > > > > >
> > > > > > > Mathieu
> > > > > > >
> > > > > > > > + return ret;
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > /**
> > > > > > > > * zynqmp_r5_rproc_prepare()
> > > > > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > > > > {
> > > > > > > > int ret;
> > > > > > > >
> > > > > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > > > > + if (ret) {
> > > > > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > > > > + return ret;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > ret = add_tcm_banks(rproc);
> > > > > > > > if (ret) {
> > > > > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > > > > - return ret;
> > > > > > > > + goto fail_prepare;
> > > > > > > > }
> > > > > > > >
> > > > > > > > ret = add_mem_regions_carveout(rproc);
> > > > > > > > if (ret) {
> > > > > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > > > > - return ret;
> > > > > > > > + goto fail_prepare;
> > > > > > > > }
> > > > > > > >
> > > > > > > > return 0;
> > > > > > > > +
> > > > > > > > +fail_prepare:
> > > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > > +
> > > > > > > > + return ret;
> > > > > > > > }
> > > > > > > >
> > > > > > > > /**
> > > > > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > > > > >
> > > > > > > > r5_core = rproc->priv;
> > > > > > > >
> > > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > > +
> > > > > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > > > > --
> > > > > > > > 2.25.1
> > > > > > > >
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-12-06 18:06 ` Tanmay Shah
0 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-12-06 18:06 UTC (permalink / raw)
To: Mathieu Poirier
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On 12/6/23 9:43 AM, Mathieu Poirier wrote:
> On Fri, 1 Dec 2023 at 11:10, Tanmay Shah <tanmay.shah@amd.com> wrote:
> >
> >
> > On 11/29/23 11:10 AM, Mathieu Poirier wrote:
> > > On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
> > > >
> > > > On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > > > > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > > > > Hi Mathieu,
> > > > > >
> > > > > > Please find my comments below.
> > > > > >
> > > > > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > > > > Use TCM pm domains extracted from device-tree
> > > > > > > > to power on/off TCM using general pm domain framework.
> > > > > > > >
> > > > > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > > > > ---
> > > > > > > >
> > > > > > > > Changes in v7:
> > > > > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > > > > - remove pm_domain_id check to move next patch
> > > > > > > > - add comment about how 1st entry in pm domain list is used
> > > > > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > > > > >
> > > > > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > > > > >
> > > > > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > @@ -16,6 +16,7 @@
> > > > > > > > #include <linux/of_reserved_mem.h>
> > > > > > > > #include <linux/platform_device.h>
> > > > > > > > #include <linux/remoteproc.h>
> > > > > > > > +#include <linux/pm_domain.h>
> > > > > > > >
> > > > > > > > #include "remoteproc_internal.h"
> > > > > > > >
> > > > > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > > > > * @rproc: rproc handle
> > > > > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > > > > * @ipi: pointer to mailbox information
> > > > > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > > > > + * registration
> > > > > > > > */
> > > > > > > > struct zynqmp_r5_core {
> > > > > > > > struct device *dev;
> > > > > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > > > > struct rproc *rproc;
> > > > > > > > u32 pm_domain_id;
> > > > > > > > struct mbox_info *ipi;
> > > > > > > > + int num_pm_dev;
> > > > > > > > + struct device **pm_dev_core0;
> > > > > > > > + struct device_link **pm_dev_core0_link;
> > > > > > > > + struct device **pm_dev_core1;
> > > > > > > > + struct device_link **pm_dev_core1_link;
> > > > > > > > };
> > > > > > > >
> > > > > > > > /**
> > > > > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > > > > if (ret < 0) {
> > > > > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > > > > + pm_domain_id);
> > > > > > >
> > > > > > > Spurious change, you should have caught that.
> > > > > >
> > > > > > Ack, need to observe changes more closely before sending them.
> > > > > >
> > > > > > >
> > > > > > > > goto release_tcm_lockstep;
> > > > > > > > }
> > > > > > > >
> > > > > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > > > > return ret;
> > > > > > > > }
> > > > > > > >
> > > > > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > > > > +{
> > > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > > + struct device *dev = r5_core->dev;
> > > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > > + int i;
> > > > > > > > +
> > > > > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > > > > +
> > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > > +
> > > > > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > > + return;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > > > > +{
> > > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > > + struct platform_device *pdev;
> > > > > > > > + struct device_node *np;
> > > > > > > > + int i, j, num_pm_dev, ret;
> > > > > > > > +
> > > > > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > > > > +
> > > > > > > > + /* get number of power-domains */
> > > > > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > > > > + "#power-domain-cells");
> > > > > > > > +
> > > > > > > > + if (num_pm_dev <= 0)
> > > > > > > > + return -EINVAL;
> > > > > > > > +
> > > > > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > > > > + sizeof(struct device *),
> > > > > > > > + GFP_KERNEL);
> > > > > > > > + if (!r5_core->pm_dev_core0)
> > > > > > > > + ret = -ENOMEM;
> > > > > > > > +
> > > > > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > > > > + sizeof(struct device_link *),
> > > > > > > > + GFP_KERNEL);
> > > > > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > + return -ENOMEM;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > > > > +
> > > > > > > > + /*
> > > > > > > > + * start from 2nd entry in power-domains property list as
> > > > > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > > > > + * 1st entry is used to configure r5 operation mode.
> > > > > > >
> > > > > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > > > > >
> > > > > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > > > > >
> > > > > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > > > > >
> > > > >
> > > > > That is exactly what I am looking for. That way people don't have to go through
> > > > > the entire driver trying to figure out what is happening with pm_dev_core[0].
> > > > >
> > > > > I'm also not sure about the power-up order. Logically the TCMs should be
> > > > > powered up before the R5 in order to put code in them. The R5s are powered in
> > > > > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > > > > TCMs are powered - can you expand on that?
> > > >
> > > >
> > > > Sure. Following is call sequece
> > > >
> > > > zynqmp_r5_rproc_prepare
> > > >
> > > > zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
> > > >
> > > > . . .
> > > >
> > > > zynqmp_r5_rproc_start -> load firmware and Starts RPU
> > > >
> > > > So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
> > > >
> > > >
> > > > >
> > > > > > revision. For new platforms pm_dev_core0[0] will be added in future.
> > > > >
> > > > > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > > > > future"?
> > > >
> > > >
> > > > ZynqMP platform has platform management firmware running on microblaze.
> > > >
> > > > This firmware design does not expect R5 pm domains to be requested explicitly.
> > > >
> > > > This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
> > > >
> > > > firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
> > > >
> > > > However, this design was changed for new platforms i.e. "versal" and onwards.
> > > >
> > > > Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
> > > >
> > > > waking them up.
> > > >
> > > > That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
> > > >
> > > > Then, we should wake it up on r5_core.
> > > >
> > > > To summarize:
> > > >
> > > > For zynqmp only following call needed to start R5:
> > > >
> > > > zynqmp_pm_request_wake
> > > >
> > > > For "versal" and onwards we need two calls to start R5:
> > > >
> > > > "device_link_add" and zynqmp_pm_request_wake
> > > >
> > > > So, in future pm_core_dev[0] will be used.
> > > >
> > >
> > > Thanks for the clarification on both front. The problem here is that we are
> > > keeping R5 power domain information in two different places, i.e
> > > zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
> > >
> > > Please see if you can retreive the power domain ID from
> > > zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
> > > calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
> > > zynqmp_r5_core::pm_domain_id.
> >
> > Hi Mathieu,
> >
> > I looked into this. Probably I can't retrieve pm_domain_id from pm_dev_core0[0],
> >
> > However, I can retrieve it from device-tree when calling zynqmp_pm_request_wake
> >
> > and zynqmp_pm_force_pwrdwn.
> >
> > zynqmp_r5_core::pm_domain_id is caching that value during probe, and use it during
> > rest of the driver lifecycle.
> >
> > I am okay either way, (keeping it as it is, or get it from device-tree). Let me know your
> >
> > preference.
> >
>
> Humm... Then I suggest to simply get rid of the device linking to
> deal with the TCMs' power management. From where I stand it provides
> more confusion than benefits, and that is without considering the
> extra complexity.
Do you mean to get rid of pm_dev_core0[1], and pm_dev_core0[2] as well ?
If yes, its preferred to use pm_domain framework to power-on/off TCM.
If we want to get rid of zynqmp_r5_core::pm_domain_id, I will do what's done in
__genpd_dev_pm_attach API where, pm_domain_id is retrieved using of_node of pm_dev_core0[*] device.
ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
"#power-domain-cells", index, &pd_args);
However, Its preferred to use pm_domain framework when power-domains are available in device-tree.
Let me know.
Thanks,
Tanmay
> > Thanks,
> >
> > Tanmay
> >
> > > >
> > > > > >
> > > > > > I hope this meets expectations.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > > + */
> > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > > > > >
> > > > > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > > > > stick with it. I have no preference on which one.
> > > > > >
> > > > > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > > > > + ret = -EINVAL;
> > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > + }
> > > > > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > > > > + r5_core->pm_dev_core0[i],
> > > > > > > > + DL_FLAG_STATELESS |
> > > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > > > > + ret = -EINVAL;
> > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > + }
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > > > > + return 0;
> > > > > > > > +
> > > > > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > > > > + sizeof(struct device *),
> > > > > > > > + GFP_KERNEL);
> > > > > > > > + if (!r5_core->pm_dev_core1) {
> > > > > > > > + ret = -ENOMEM;
> > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > > > > + sizeof(struct device_link *),
> > > > > > > > + GFP_KERNEL);
> > > > > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > + ret = -ENOMEM;
> > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + /* get second core's device to detach its power-domains */
> > > > > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > > > > +
> > > > > > > > + pdev = of_find_device_by_node(np);
> > > > > > > > + if (!pdev) {
> > > > > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > + ret = -EINVAL;
> > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + dev2 = &pdev->dev;
> > > > > > > > +
> > > > > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > > + ret = -EINVAL;
> > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > > > > + r5_core->pm_dev_core1[j],
> > > > > > > > + DL_FLAG_STATELESS |
> > > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > > > > + ret = -ENODEV;
> > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > + }
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > +fail_add_pm_domains_lockstep:
> > > > > > > > + while (--j >= 0) {
> > > > > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > > + }
> > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > +
> > > > > > > > +fail_add_pm_domains:
> > > > > > > > + while (--i >= 0) {
> > > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > > + }
> > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > > +
> > > > > > >
> > > > > > > The error path is much cleaner and readable now.
> > > > > > >
> > > > > > > I will continue tomorrow.
> > > > > > >
> > > > > > > Mathieu
> > > > > > >
> > > > > > > > + return ret;
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > /**
> > > > > > > > * zynqmp_r5_rproc_prepare()
> > > > > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > > > > {
> > > > > > > > int ret;
> > > > > > > >
> > > > > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > > > > + if (ret) {
> > > > > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > > > > + return ret;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > ret = add_tcm_banks(rproc);
> > > > > > > > if (ret) {
> > > > > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > > > > - return ret;
> > > > > > > > + goto fail_prepare;
> > > > > > > > }
> > > > > > > >
> > > > > > > > ret = add_mem_regions_carveout(rproc);
> > > > > > > > if (ret) {
> > > > > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > > > > - return ret;
> > > > > > > > + goto fail_prepare;
> > > > > > > > }
> > > > > > > >
> > > > > > > > return 0;
> > > > > > > > +
> > > > > > > > +fail_prepare:
> > > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > > +
> > > > > > > > + return ret;
> > > > > > > > }
> > > > > > > >
> > > > > > > > /**
> > > > > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > > > > >
> > > > > > > > r5_core = rproc->priv;
> > > > > > > >
> > > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > > +
> > > > > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > > > > --
> > > > > > > > 2.25.1
> > > > > > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-12-06 18:06 ` Tanmay Shah
@ 2023-12-08 15:51 ` Mathieu Poirier
-1 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-12-08 15:51 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Wed, Dec 06, 2023 at 12:06:45PM -0600, Tanmay Shah wrote:
>
> On 12/6/23 9:43 AM, Mathieu Poirier wrote:
> > On Fri, 1 Dec 2023 at 11:10, Tanmay Shah <tanmay.shah@amd.com> wrote:
> > >
> > >
> > > On 11/29/23 11:10 AM, Mathieu Poirier wrote:
> > > > On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
> > > > >
> > > > > On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > > > > > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > > > > > Hi Mathieu,
> > > > > > >
> > > > > > > Please find my comments below.
> > > > > > >
> > > > > > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > > > > > Use TCM pm domains extracted from device-tree
> > > > > > > > > to power on/off TCM using general pm domain framework.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > > > > > ---
> > > > > > > > >
> > > > > > > > > Changes in v7:
> > > > > > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > > > > > - remove pm_domain_id check to move next patch
> > > > > > > > > - add comment about how 1st entry in pm domain list is used
> > > > > > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > > > > > >
> > > > > > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > > > > > >
> > > > > > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > > @@ -16,6 +16,7 @@
> > > > > > > > > #include <linux/of_reserved_mem.h>
> > > > > > > > > #include <linux/platform_device.h>
> > > > > > > > > #include <linux/remoteproc.h>
> > > > > > > > > +#include <linux/pm_domain.h>
> > > > > > > > >
> > > > > > > > > #include "remoteproc_internal.h"
> > > > > > > > >
> > > > > > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > > > > > * @rproc: rproc handle
> > > > > > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > > > > > * @ipi: pointer to mailbox information
> > > > > > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > > > > > + * registration
> > > > > > > > > */
> > > > > > > > > struct zynqmp_r5_core {
> > > > > > > > > struct device *dev;
> > > > > > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > > > > > struct rproc *rproc;
> > > > > > > > > u32 pm_domain_id;
> > > > > > > > > struct mbox_info *ipi;
> > > > > > > > > + int num_pm_dev;
> > > > > > > > > + struct device **pm_dev_core0;
> > > > > > > > > + struct device_link **pm_dev_core0_link;
> > > > > > > > > + struct device **pm_dev_core1;
> > > > > > > > > + struct device_link **pm_dev_core1_link;
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > > /**
> > > > > > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > > > > > if (ret < 0) {
> > > > > > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > > > > > + pm_domain_id);
> > > > > > > >
> > > > > > > > Spurious change, you should have caught that.
> > > > > > >
> > > > > > > Ack, need to observe changes more closely before sending them.
> > > > > > >
> > > > > > > >
> > > > > > > > > goto release_tcm_lockstep;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > > > > > return ret;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > > > > > +{
> > > > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > > > + struct device *dev = r5_core->dev;
> > > > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > > > + int i;
> > > > > > > > > +
> > > > > > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > > > > > +
> > > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > > > +
> > > > > > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > > > + return;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > > > > > +{
> > > > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > > > + struct platform_device *pdev;
> > > > > > > > > + struct device_node *np;
> > > > > > > > > + int i, j, num_pm_dev, ret;
> > > > > > > > > +
> > > > > > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > > > > > +
> > > > > > > > > + /* get number of power-domains */
> > > > > > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > > > > > + "#power-domain-cells");
> > > > > > > > > +
> > > > > > > > > + if (num_pm_dev <= 0)
> > > > > > > > > + return -EINVAL;
> > > > > > > > > +
> > > > > > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > > > > > + sizeof(struct device *),
> > > > > > > > > + GFP_KERNEL);
> > > > > > > > > + if (!r5_core->pm_dev_core0)
> > > > > > > > > + ret = -ENOMEM;
> > > > > > > > > +
> > > > > > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > > > > > + sizeof(struct device_link *),
> > > > > > > > > + GFP_KERNEL);
> > > > > > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > > + return -ENOMEM;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > > > > > +
> > > > > > > > > + /*
> > > > > > > > > + * start from 2nd entry in power-domains property list as
> > > > > > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > > > > > + * 1st entry is used to configure r5 operation mode.
> > > > > > > >
> > > > > > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > > > > > >
> > > > > > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > > > > > >
> > > > > > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > > > > > >
> > > > > >
> > > > > > That is exactly what I am looking for. That way people don't have to go through
> > > > > > the entire driver trying to figure out what is happening with pm_dev_core[0].
> > > > > >
> > > > > > I'm also not sure about the power-up order. Logically the TCMs should be
> > > > > > powered up before the R5 in order to put code in them. The R5s are powered in
> > > > > > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > > > > > TCMs are powered - can you expand on that?
> > > > >
> > > > >
> > > > > Sure. Following is call sequece
> > > > >
> > > > > zynqmp_r5_rproc_prepare
> > > > >
> > > > > zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
> > > > >
> > > > > . . .
> > > > >
> > > > > zynqmp_r5_rproc_start -> load firmware and Starts RPU
> > > > >
> > > > > So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
> > > > >
> > > > >
> > > > > >
> > > > > > > revision. For new platforms pm_dev_core0[0] will be added in future.
> > > > > >
> > > > > > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > > > > > future"?
> > > > >
> > > > >
> > > > > ZynqMP platform has platform management firmware running on microblaze.
> > > > >
> > > > > This firmware design does not expect R5 pm domains to be requested explicitly.
> > > > >
> > > > > This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
> > > > >
> > > > > firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
> > > > >
> > > > > However, this design was changed for new platforms i.e. "versal" and onwards.
> > > > >
> > > > > Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
> > > > >
> > > > > waking them up.
> > > > >
> > > > > That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
> > > > >
> > > > > Then, we should wake it up on r5_core.
> > > > >
> > > > > To summarize:
> > > > >
> > > > > For zynqmp only following call needed to start R5:
> > > > >
> > > > > zynqmp_pm_request_wake
> > > > >
> > > > > For "versal" and onwards we need two calls to start R5:
> > > > >
> > > > > "device_link_add" and zynqmp_pm_request_wake
> > > > >
> > > > > So, in future pm_core_dev[0] will be used.
> > > > >
> > > >
> > > > Thanks for the clarification on both front. The problem here is that we are
> > > > keeping R5 power domain information in two different places, i.e
> > > > zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
> > > >
> > > > Please see if you can retreive the power domain ID from
> > > > zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
> > > > calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
> > > > zynqmp_r5_core::pm_domain_id.
> > >
> > > Hi Mathieu,
> > >
> > > I looked into this. Probably I can't retrieve pm_domain_id from pm_dev_core0[0],
> > >
> > > However, I can retrieve it from device-tree when calling zynqmp_pm_request_wake
> > >
> > > and zynqmp_pm_force_pwrdwn.
> > >
> > > zynqmp_r5_core::pm_domain_id is caching that value during probe, and use it during
> > > rest of the driver lifecycle.
> > >
> > > I am okay either way, (keeping it as it is, or get it from device-tree). Let me know your
> > >
> > > preference.
> > >
> >
> > Humm... Then I suggest to simply get rid of the device linking to
> > deal with the TCMs' power management. From where I stand it provides
> > more confusion than benefits, and that is without considering the
> > extra complexity.
>
>
> Do you mean to get rid of pm_dev_core0[1], and pm_dev_core0[2] as well ?
>
Yes
> If yes, its preferred to use pm_domain framework to power-on/off TCM.
>
That is when the pm runtime subsystem is used extensively but it isn't
the case here. In this scenario using device linking is adding a lot of
complexity for something that could be done in a single line of code.
> If we want to get rid of zynqmp_r5_core::pm_domain_id, I will do what's done in
>
> __genpd_dev_pm_attach API where, pm_domain_id is retrieved using of_node of pm_dev_core0[*] device.
>
> ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
> "#power-domain-cells", index, &pd_args);
That is what I asked for in my previous email, i.e use ->pm_dev_core0[0] to
retrieve the domain ID.
>
> However, Its preferred to use pm_domain framework when power-domains are available in device-tree.
>
> Let me know.
>
> Thanks,
>
> Tanmay
>
>
> > > Thanks,
> > >
> > > Tanmay
> > >
> > > > >
> > > > > > >
> > > > > > > I hope this meets expectations.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > + */
> > > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > > > > > >
> > > > > > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > > > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > > > > > stick with it. I have no preference on which one.
> > > > > > >
> > > > > > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > > > > > + ret = -EINVAL;
> > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > + }
> > > > > > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > > > > > + r5_core->pm_dev_core0[i],
> > > > > > > > > + DL_FLAG_STATELESS |
> > > > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > > > > > + ret = -EINVAL;
> > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > + }
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > > > > > + return 0;
> > > > > > > > > +
> > > > > > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > > > > > + sizeof(struct device *),
> > > > > > > > > + GFP_KERNEL);
> > > > > > > > > + if (!r5_core->pm_dev_core1) {
> > > > > > > > > + ret = -ENOMEM;
> > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > > > > > + sizeof(struct device_link *),
> > > > > > > > > + GFP_KERNEL);
> > > > > > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > + ret = -ENOMEM;
> > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + /* get second core's device to detach its power-domains */
> > > > > > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > > > > > +
> > > > > > > > > + pdev = of_find_device_by_node(np);
> > > > > > > > > + if (!pdev) {
> > > > > > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > > + ret = -EINVAL;
> > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + dev2 = &pdev->dev;
> > > > > > > > > +
> > > > > > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > > > + ret = -EINVAL;
> > > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > > > > > + r5_core->pm_dev_core1[j],
> > > > > > > > > + DL_FLAG_STATELESS |
> > > > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > > > > > + ret = -ENODEV;
> > > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > > + }
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > +fail_add_pm_domains_lockstep:
> > > > > > > > > + while (--j >= 0) {
> > > > > > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > > > + }
> > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > > +
> > > > > > > > > +fail_add_pm_domains:
> > > > > > > > > + while (--i >= 0) {
> > > > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > > > + }
> > > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > > > +
> > > > > > > >
> > > > > > > > The error path is much cleaner and readable now.
> > > > > > > >
> > > > > > > > I will continue tomorrow.
> > > > > > > >
> > > > > > > > Mathieu
> > > > > > > >
> > > > > > > > > + return ret;
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > /**
> > > > > > > > > * zynqmp_r5_rproc_prepare()
> > > > > > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > > > > > {
> > > > > > > > > int ret;
> > > > > > > > >
> > > > > > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > > > > > + if (ret) {
> > > > > > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > > > > > + return ret;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > ret = add_tcm_banks(rproc);
> > > > > > > > > if (ret) {
> > > > > > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > > > > > - return ret;
> > > > > > > > > + goto fail_prepare;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > ret = add_mem_regions_carveout(rproc);
> > > > > > > > > if (ret) {
> > > > > > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > > > > > - return ret;
> > > > > > > > > + goto fail_prepare;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > return 0;
> > > > > > > > > +
> > > > > > > > > +fail_prepare:
> > > > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > > > +
> > > > > > > > > + return ret;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > /**
> > > > > > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > > > > > >
> > > > > > > > > r5_core = rproc->priv;
> > > > > > > > >
> > > > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > > > +
> > > > > > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > > > > > --
> > > > > > > > > 2.25.1
> > > > > > > > >
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-12-08 15:51 ` Mathieu Poirier
0 siblings, 0 replies; 38+ messages in thread
From: Mathieu Poirier @ 2023-12-08 15:51 UTC (permalink / raw)
To: Tanmay Shah
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On Wed, Dec 06, 2023 at 12:06:45PM -0600, Tanmay Shah wrote:
>
> On 12/6/23 9:43 AM, Mathieu Poirier wrote:
> > On Fri, 1 Dec 2023 at 11:10, Tanmay Shah <tanmay.shah@amd.com> wrote:
> > >
> > >
> > > On 11/29/23 11:10 AM, Mathieu Poirier wrote:
> > > > On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
> > > > >
> > > > > On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > > > > > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > > > > > Hi Mathieu,
> > > > > > >
> > > > > > > Please find my comments below.
> > > > > > >
> > > > > > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > > > > > Use TCM pm domains extracted from device-tree
> > > > > > > > > to power on/off TCM using general pm domain framework.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > > > > > ---
> > > > > > > > >
> > > > > > > > > Changes in v7:
> > > > > > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > > > > > - remove pm_domain_id check to move next patch
> > > > > > > > > - add comment about how 1st entry in pm domain list is used
> > > > > > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > > > > > >
> > > > > > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > > > > > >
> > > > > > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > > @@ -16,6 +16,7 @@
> > > > > > > > > #include <linux/of_reserved_mem.h>
> > > > > > > > > #include <linux/platform_device.h>
> > > > > > > > > #include <linux/remoteproc.h>
> > > > > > > > > +#include <linux/pm_domain.h>
> > > > > > > > >
> > > > > > > > > #include "remoteproc_internal.h"
> > > > > > > > >
> > > > > > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > > > > > * @rproc: rproc handle
> > > > > > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > > > > > * @ipi: pointer to mailbox information
> > > > > > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > > > > > + * registration
> > > > > > > > > */
> > > > > > > > > struct zynqmp_r5_core {
> > > > > > > > > struct device *dev;
> > > > > > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > > > > > struct rproc *rproc;
> > > > > > > > > u32 pm_domain_id;
> > > > > > > > > struct mbox_info *ipi;
> > > > > > > > > + int num_pm_dev;
> > > > > > > > > + struct device **pm_dev_core0;
> > > > > > > > > + struct device_link **pm_dev_core0_link;
> > > > > > > > > + struct device **pm_dev_core1;
> > > > > > > > > + struct device_link **pm_dev_core1_link;
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > > /**
> > > > > > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > > > > > if (ret < 0) {
> > > > > > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > > > > > + pm_domain_id);
> > > > > > > >
> > > > > > > > Spurious change, you should have caught that.
> > > > > > >
> > > > > > > Ack, need to observe changes more closely before sending them.
> > > > > > >
> > > > > > > >
> > > > > > > > > goto release_tcm_lockstep;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > > > > > return ret;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > > > > > +{
> > > > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > > > + struct device *dev = r5_core->dev;
> > > > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > > > + int i;
> > > > > > > > > +
> > > > > > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > > > > > +
> > > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > > > +
> > > > > > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > > > + return;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > > > > > +{
> > > > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > > > + struct platform_device *pdev;
> > > > > > > > > + struct device_node *np;
> > > > > > > > > + int i, j, num_pm_dev, ret;
> > > > > > > > > +
> > > > > > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > > > > > +
> > > > > > > > > + /* get number of power-domains */
> > > > > > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > > > > > + "#power-domain-cells");
> > > > > > > > > +
> > > > > > > > > + if (num_pm_dev <= 0)
> > > > > > > > > + return -EINVAL;
> > > > > > > > > +
> > > > > > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > > > > > + sizeof(struct device *),
> > > > > > > > > + GFP_KERNEL);
> > > > > > > > > + if (!r5_core->pm_dev_core0)
> > > > > > > > > + ret = -ENOMEM;
> > > > > > > > > +
> > > > > > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > > > > > + sizeof(struct device_link *),
> > > > > > > > > + GFP_KERNEL);
> > > > > > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > > + return -ENOMEM;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > > > > > +
> > > > > > > > > + /*
> > > > > > > > > + * start from 2nd entry in power-domains property list as
> > > > > > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > > > > > + * 1st entry is used to configure r5 operation mode.
> > > > > > > >
> > > > > > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > > > > > >
> > > > > > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > > > > > >
> > > > > > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > > > > > >
> > > > > >
> > > > > > That is exactly what I am looking for. That way people don't have to go through
> > > > > > the entire driver trying to figure out what is happening with pm_dev_core[0].
> > > > > >
> > > > > > I'm also not sure about the power-up order. Logically the TCMs should be
> > > > > > powered up before the R5 in order to put code in them. The R5s are powered in
> > > > > > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > > > > > TCMs are powered - can you expand on that?
> > > > >
> > > > >
> > > > > Sure. Following is call sequece
> > > > >
> > > > > zynqmp_r5_rproc_prepare
> > > > >
> > > > > zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
> > > > >
> > > > > . . .
> > > > >
> > > > > zynqmp_r5_rproc_start -> load firmware and Starts RPU
> > > > >
> > > > > So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
> > > > >
> > > > >
> > > > > >
> > > > > > > revision. For new platforms pm_dev_core0[0] will be added in future.
> > > > > >
> > > > > > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > > > > > future"?
> > > > >
> > > > >
> > > > > ZynqMP platform has platform management firmware running on microblaze.
> > > > >
> > > > > This firmware design does not expect R5 pm domains to be requested explicitly.
> > > > >
> > > > > This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
> > > > >
> > > > > firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
> > > > >
> > > > > However, this design was changed for new platforms i.e. "versal" and onwards.
> > > > >
> > > > > Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
> > > > >
> > > > > waking them up.
> > > > >
> > > > > That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
> > > > >
> > > > > Then, we should wake it up on r5_core.
> > > > >
> > > > > To summarize:
> > > > >
> > > > > For zynqmp only following call needed to start R5:
> > > > >
> > > > > zynqmp_pm_request_wake
> > > > >
> > > > > For "versal" and onwards we need two calls to start R5:
> > > > >
> > > > > "device_link_add" and zynqmp_pm_request_wake
> > > > >
> > > > > So, in future pm_core_dev[0] will be used.
> > > > >
> > > >
> > > > Thanks for the clarification on both front. The problem here is that we are
> > > > keeping R5 power domain information in two different places, i.e
> > > > zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
> > > >
> > > > Please see if you can retreive the power domain ID from
> > > > zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
> > > > calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
> > > > zynqmp_r5_core::pm_domain_id.
> > >
> > > Hi Mathieu,
> > >
> > > I looked into this. Probably I can't retrieve pm_domain_id from pm_dev_core0[0],
> > >
> > > However, I can retrieve it from device-tree when calling zynqmp_pm_request_wake
> > >
> > > and zynqmp_pm_force_pwrdwn.
> > >
> > > zynqmp_r5_core::pm_domain_id is caching that value during probe, and use it during
> > > rest of the driver lifecycle.
> > >
> > > I am okay either way, (keeping it as it is, or get it from device-tree). Let me know your
> > >
> > > preference.
> > >
> >
> > Humm... Then I suggest to simply get rid of the device linking to
> > deal with the TCMs' power management. From where I stand it provides
> > more confusion than benefits, and that is without considering the
> > extra complexity.
>
>
> Do you mean to get rid of pm_dev_core0[1], and pm_dev_core0[2] as well ?
>
Yes
> If yes, its preferred to use pm_domain framework to power-on/off TCM.
>
That is when the pm runtime subsystem is used extensively but it isn't
the case here. In this scenario using device linking is adding a lot of
complexity for something that could be done in a single line of code.
> If we want to get rid of zynqmp_r5_core::pm_domain_id, I will do what's done in
>
> __genpd_dev_pm_attach API where, pm_domain_id is retrieved using of_node of pm_dev_core0[*] device.
>
> ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
> "#power-domain-cells", index, &pd_args);
That is what I asked for in my previous email, i.e use ->pm_dev_core0[0] to
retrieve the domain ID.
>
> However, Its preferred to use pm_domain framework when power-domains are available in device-tree.
>
> Let me know.
>
> Thanks,
>
> Tanmay
>
>
> > > Thanks,
> > >
> > > Tanmay
> > >
> > > > >
> > > > > > >
> > > > > > > I hope this meets expectations.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > + */
> > > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > > > > > >
> > > > > > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > > > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > > > > > stick with it. I have no preference on which one.
> > > > > > >
> > > > > > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > > > > > + ret = -EINVAL;
> > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > + }
> > > > > > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > > > > > + r5_core->pm_dev_core0[i],
> > > > > > > > > + DL_FLAG_STATELESS |
> > > > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > > > > > + ret = -EINVAL;
> > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > + }
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > > > > > + return 0;
> > > > > > > > > +
> > > > > > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > > > > > + sizeof(struct device *),
> > > > > > > > > + GFP_KERNEL);
> > > > > > > > > + if (!r5_core->pm_dev_core1) {
> > > > > > > > > + ret = -ENOMEM;
> > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > > > > > + sizeof(struct device_link *),
> > > > > > > > > + GFP_KERNEL);
> > > > > > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > + ret = -ENOMEM;
> > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + /* get second core's device to detach its power-domains */
> > > > > > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > > > > > +
> > > > > > > > > + pdev = of_find_device_by_node(np);
> > > > > > > > > + if (!pdev) {
> > > > > > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > > + ret = -EINVAL;
> > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + dev2 = &pdev->dev;
> > > > > > > > > +
> > > > > > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > > > + ret = -EINVAL;
> > > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > > > > > + r5_core->pm_dev_core1[j],
> > > > > > > > > + DL_FLAG_STATELESS |
> > > > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > > > > > + ret = -ENODEV;
> > > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > > + }
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > +fail_add_pm_domains_lockstep:
> > > > > > > > > + while (--j >= 0) {
> > > > > > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > > > + }
> > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > > +
> > > > > > > > > +fail_add_pm_domains:
> > > > > > > > > + while (--i >= 0) {
> > > > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > > > + }
> > > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > > > +
> > > > > > > >
> > > > > > > > The error path is much cleaner and readable now.
> > > > > > > >
> > > > > > > > I will continue tomorrow.
> > > > > > > >
> > > > > > > > Mathieu
> > > > > > > >
> > > > > > > > > + return ret;
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > /**
> > > > > > > > > * zynqmp_r5_rproc_prepare()
> > > > > > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > > > > > {
> > > > > > > > > int ret;
> > > > > > > > >
> > > > > > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > > > > > + if (ret) {
> > > > > > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > > > > > + return ret;
> > > > > > > > > + }
> > > > > > > > > +
> > > > > > > > > ret = add_tcm_banks(rproc);
> > > > > > > > > if (ret) {
> > > > > > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > > > > > - return ret;
> > > > > > > > > + goto fail_prepare;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > ret = add_mem_regions_carveout(rproc);
> > > > > > > > > if (ret) {
> > > > > > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > > > > > - return ret;
> > > > > > > > > + goto fail_prepare;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > return 0;
> > > > > > > > > +
> > > > > > > > > +fail_prepare:
> > > > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > > > +
> > > > > > > > > + return ret;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > /**
> > > > > > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > > > > > >
> > > > > > > > > r5_core = rproc->priv;
> > > > > > > > >
> > > > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > > > +
> > > > > > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > > > > > --
> > > > > > > > > 2.25.1
> > > > > > > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
2023-12-08 15:51 ` Mathieu Poirier
@ 2023-12-08 16:05 ` Tanmay Shah
-1 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-12-08 16:05 UTC (permalink / raw)
To: Mathieu Poirier
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On 12/8/23 9:51 AM, Mathieu Poirier wrote:
> On Wed, Dec 06, 2023 at 12:06:45PM -0600, Tanmay Shah wrote:
> >
> > On 12/6/23 9:43 AM, Mathieu Poirier wrote:
> > > On Fri, 1 Dec 2023 at 11:10, Tanmay Shah <tanmay.shah@amd.com> wrote:
> > > >
> > > >
> > > > On 11/29/23 11:10 AM, Mathieu Poirier wrote:
> > > > > On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
> > > > > >
> > > > > > On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > > > > > > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > > > > > > Hi Mathieu,
> > > > > > > >
> > > > > > > > Please find my comments below.
> > > > > > > >
> > > > > > > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > > > > > > Use TCM pm domains extracted from device-tree
> > > > > > > > > > to power on/off TCM using general pm domain framework.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > > > > > > ---
> > > > > > > > > >
> > > > > > > > > > Changes in v7:
> > > > > > > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > > > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > > > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > > > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > > > > > > - remove pm_domain_id check to move next patch
> > > > > > > > > > - add comment about how 1st entry in pm domain list is used
> > > > > > > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > > > > > > >
> > > > > > > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > > > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > > > > > > >
> > > > > > > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > > > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > > > @@ -16,6 +16,7 @@
> > > > > > > > > > #include <linux/of_reserved_mem.h>
> > > > > > > > > > #include <linux/platform_device.h>
> > > > > > > > > > #include <linux/remoteproc.h>
> > > > > > > > > > +#include <linux/pm_domain.h>
> > > > > > > > > >
> > > > > > > > > > #include "remoteproc_internal.h"
> > > > > > > > > >
> > > > > > > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > > > > > > * @rproc: rproc handle
> > > > > > > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > > > > > > * @ipi: pointer to mailbox information
> > > > > > > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > > > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > > > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > > > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > > > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > > > > > > + * registration
> > > > > > > > > > */
> > > > > > > > > > struct zynqmp_r5_core {
> > > > > > > > > > struct device *dev;
> > > > > > > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > > > > > > struct rproc *rproc;
> > > > > > > > > > u32 pm_domain_id;
> > > > > > > > > > struct mbox_info *ipi;
> > > > > > > > > > + int num_pm_dev;
> > > > > > > > > > + struct device **pm_dev_core0;
> > > > > > > > > > + struct device_link **pm_dev_core0_link;
> > > > > > > > > > + struct device **pm_dev_core1;
> > > > > > > > > > + struct device_link **pm_dev_core1_link;
> > > > > > > > > > };
> > > > > > > > > >
> > > > > > > > > > /**
> > > > > > > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > > > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > > > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > > > > > > if (ret < 0) {
> > > > > > > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > > > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > > > > > > + pm_domain_id);
> > > > > > > > >
> > > > > > > > > Spurious change, you should have caught that.
> > > > > > > >
> > > > > > > > Ack, need to observe changes more closely before sending them.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > goto release_tcm_lockstep;
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > > > > > > return ret;
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > > > > > > +{
> > > > > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > > > > + struct device *dev = r5_core->dev;
> > > > > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > > > > + int i;
> > > > > > > > > > +
> > > > > > > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > > > > > > +
> > > > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > > > > +
> > > > > > > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > > > > + return;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > > > > +}
> > > > > > > > > > +
> > > > > > > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > > > > > > +{
> > > > > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > > > > + struct platform_device *pdev;
> > > > > > > > > > + struct device_node *np;
> > > > > > > > > > + int i, j, num_pm_dev, ret;
> > > > > > > > > > +
> > > > > > > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > > > > > > +
> > > > > > > > > > + /* get number of power-domains */
> > > > > > > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > > > > > > + "#power-domain-cells");
> > > > > > > > > > +
> > > > > > > > > > + if (num_pm_dev <= 0)
> > > > > > > > > > + return -EINVAL;
> > > > > > > > > > +
> > > > > > > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > > > > > > + sizeof(struct device *),
> > > > > > > > > > + GFP_KERNEL);
> > > > > > > > > > + if (!r5_core->pm_dev_core0)
> > > > > > > > > > + ret = -ENOMEM;
> > > > > > > > > > +
> > > > > > > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > > > > > > + sizeof(struct device_link *),
> > > > > > > > > > + GFP_KERNEL);
> > > > > > > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > > > + return -ENOMEM;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > > > > > > +
> > > > > > > > > > + /*
> > > > > > > > > > + * start from 2nd entry in power-domains property list as
> > > > > > > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > > > > > > + * 1st entry is used to configure r5 operation mode.
> > > > > > > > >
> > > > > > > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > > > > > > >
> > > > > > > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > > > > > > >
> > > > > > > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > > > > > > >
> > > > > > >
> > > > > > > That is exactly what I am looking for. That way people don't have to go through
> > > > > > > the entire driver trying to figure out what is happening with pm_dev_core[0].
> > > > > > >
> > > > > > > I'm also not sure about the power-up order. Logically the TCMs should be
> > > > > > > powered up before the R5 in order to put code in them. The R5s are powered in
> > > > > > > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > > > > > > TCMs are powered - can you expand on that?
> > > > > >
> > > > > >
> > > > > > Sure. Following is call sequece
> > > > > >
> > > > > > zynqmp_r5_rproc_prepare
> > > > > >
> > > > > > zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
> > > > > >
> > > > > > . . .
> > > > > >
> > > > > > zynqmp_r5_rproc_start -> load firmware and Starts RPU
> > > > > >
> > > > > > So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > > revision. For new platforms pm_dev_core0[0] will be added in future.
> > > > > > >
> > > > > > > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > > > > > > future"?
> > > > > >
> > > > > >
> > > > > > ZynqMP platform has platform management firmware running on microblaze.
> > > > > >
> > > > > > This firmware design does not expect R5 pm domains to be requested explicitly.
> > > > > >
> > > > > > This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
> > > > > >
> > > > > > firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
> > > > > >
> > > > > > However, this design was changed for new platforms i.e. "versal" and onwards.
> > > > > >
> > > > > > Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
> > > > > >
> > > > > > waking them up.
> > > > > >
> > > > > > That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
> > > > > >
> > > > > > Then, we should wake it up on r5_core.
> > > > > >
> > > > > > To summarize:
> > > > > >
> > > > > > For zynqmp only following call needed to start R5:
> > > > > >
> > > > > > zynqmp_pm_request_wake
> > > > > >
> > > > > > For "versal" and onwards we need two calls to start R5:
> > > > > >
> > > > > > "device_link_add" and zynqmp_pm_request_wake
> > > > > >
> > > > > > So, in future pm_core_dev[0] will be used.
> > > > > >
> > > > >
> > > > > Thanks for the clarification on both front. The problem here is that we are
> > > > > keeping R5 power domain information in two different places, i.e
> > > > > zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
> > > > >
> > > > > Please see if you can retreive the power domain ID from
> > > > > zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
> > > > > calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
> > > > > zynqmp_r5_core::pm_domain_id.
> > > >
> > > > Hi Mathieu,
> > > >
> > > > I looked into this. Probably I can't retrieve pm_domain_id from pm_dev_core0[0],
> > > >
> > > > However, I can retrieve it from device-tree when calling zynqmp_pm_request_wake
> > > >
> > > > and zynqmp_pm_force_pwrdwn.
> > > >
> > > > zynqmp_r5_core::pm_domain_id is caching that value during probe, and use it during
> > > > rest of the driver lifecycle.
> > > >
> > > > I am okay either way, (keeping it as it is, or get it from device-tree). Let me know your
> > > >
> > > > preference.
> > > >
> > >
> > > Humm... Then I suggest to simply get rid of the device linking to
> > > deal with the TCMs' power management. From where I stand it provides
> > > more confusion than benefits, and that is without considering the
> > > extra complexity.
> >
> >
> > Do you mean to get rid of pm_dev_core0[1], and pm_dev_core0[2] as well ?
> >
>
> Yes
>
> > If yes, its preferred to use pm_domain framework to power-on/off TCM.
> >
>
> That is when the pm runtime subsystem is used extensively but it isn't
> the case here. In this scenario using device linking is adding a lot of
> complexity for something that could be done in a single line of code.
Ok fair point, please give me sometime to explore more.
I don't mind getting rid of pm domain framework usage. For some reason, if we must use it,
we can always introduce it later.
I just want to make sure if we don't use pm domains framework, then remoteproc works for
future platforms as well. I will get back to you as need to do more testing after removing pm domains framework.
Thanks,
Tanmay
> > If we want to get rid of zynqmp_r5_core::pm_domain_id, I will do what's done in
> >
> > __genpd_dev_pm_attach API where, pm_domain_id is retrieved using of_node of pm_dev_core0[*] device.
> >
> > ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
> > "#power-domain-cells", index, &pd_args);
>
> That is what I asked for in my previous email, i.e use ->pm_dev_core0[0] to
> retrieve the domain ID.
>
> >
> > However, Its preferred to use pm_domain framework when power-domains are available in device-tree.
> >
> > Let me know.
> >
> > Thanks,
> >
> > Tanmay
> >
> >
> > > > Thanks,
> > > >
> > > > Tanmay
> > > >
> > > > > >
> > > > > > > >
> > > > > > > > I hope this meets expectations.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > + */
> > > > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > > > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > > > > > > >
> > > > > > > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > > > > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > > > > > > stick with it. I have no preference on which one.
> > > > > > > >
> > > > > > > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > > > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > > > > > > + ret = -EINVAL;
> > > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > > + }
> > > > > > > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > > > > > > + r5_core->pm_dev_core0[i],
> > > > > > > > > > + DL_FLAG_STATELESS |
> > > > > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > > > > > > + ret = -EINVAL;
> > > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > > + }
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > > > > > > + return 0;
> > > > > > > > > > +
> > > > > > > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > > > > > > + sizeof(struct device *),
> > > > > > > > > > + GFP_KERNEL);
> > > > > > > > > > + if (!r5_core->pm_dev_core1) {
> > > > > > > > > > + ret = -ENOMEM;
> > > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > > > > > > + sizeof(struct device_link *),
> > > > > > > > > > + GFP_KERNEL);
> > > > > > > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > > + ret = -ENOMEM;
> > > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + /* get second core's device to detach its power-domains */
> > > > > > > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > > > > > > +
> > > > > > > > > > + pdev = of_find_device_by_node(np);
> > > > > > > > > > + if (!pdev) {
> > > > > > > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > > > + ret = -EINVAL;
> > > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + dev2 = &pdev->dev;
> > > > > > > > > > +
> > > > > > > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > > > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > > > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > > > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > > > > + ret = -EINVAL;
> > > > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > > > > > > + r5_core->pm_dev_core1[j],
> > > > > > > > > > + DL_FLAG_STATELESS |
> > > > > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > > > > > > + ret = -ENODEV;
> > > > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > > > + }
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > +fail_add_pm_domains_lockstep:
> > > > > > > > > > + while (--j >= 0) {
> > > > > > > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > > > > + }
> > > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > > > +
> > > > > > > > > > +fail_add_pm_domains:
> > > > > > > > > > + while (--i >= 0) {
> > > > > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > > > > + }
> > > > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > > > > +
> > > > > > > > >
> > > > > > > > > The error path is much cleaner and readable now.
> > > > > > > > >
> > > > > > > > > I will continue tomorrow.
> > > > > > > > >
> > > > > > > > > Mathieu
> > > > > > > > >
> > > > > > > > > > + return ret;
> > > > > > > > > > +}
> > > > > > > > > > +
> > > > > > > > > > /**
> > > > > > > > > > * zynqmp_r5_rproc_prepare()
> > > > > > > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > > > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > > > > > > {
> > > > > > > > > > int ret;
> > > > > > > > > >
> > > > > > > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > > > > > > + if (ret) {
> > > > > > > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > > > > > > + return ret;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > ret = add_tcm_banks(rproc);
> > > > > > > > > > if (ret) {
> > > > > > > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > > > > > > - return ret;
> > > > > > > > > > + goto fail_prepare;
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > ret = add_mem_regions_carveout(rproc);
> > > > > > > > > > if (ret) {
> > > > > > > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > > > > > > - return ret;
> > > > > > > > > > + goto fail_prepare;
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > return 0;
> > > > > > > > > > +
> > > > > > > > > > +fail_prepare:
> > > > > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > > > > +
> > > > > > > > > > + return ret;
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > /**
> > > > > > > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > > > > > > >
> > > > > > > > > > r5_core = rproc->priv;
> > > > > > > > > >
> > > > > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > > > > +
> > > > > > > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > > > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > > > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > > > > > > --
> > > > > > > > > > 2.25.1
> > > > > > > > > >
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support
@ 2023-12-08 16:05 ` Tanmay Shah
0 siblings, 0 replies; 38+ messages in thread
From: Tanmay Shah @ 2023-12-08 16:05 UTC (permalink / raw)
To: Mathieu Poirier
Cc: andersson, robh+dt, krzysztof.kozlowski+dt, conor+dt,
michal.simek, ben.levinsky, linux-remoteproc, devicetree,
linux-arm-kernel, linux-kernel
On 12/8/23 9:51 AM, Mathieu Poirier wrote:
> On Wed, Dec 06, 2023 at 12:06:45PM -0600, Tanmay Shah wrote:
> >
> > On 12/6/23 9:43 AM, Mathieu Poirier wrote:
> > > On Fri, 1 Dec 2023 at 11:10, Tanmay Shah <tanmay.shah@amd.com> wrote:
> > > >
> > > >
> > > > On 11/29/23 11:10 AM, Mathieu Poirier wrote:
> > > > > On Mon, Nov 27, 2023 at 10:33:05AM -0600, Tanmay Shah wrote:
> > > > > >
> > > > > > On 11/23/23 12:11 PM, Mathieu Poirier wrote:
> > > > > > > On Wed, Nov 22, 2023 at 03:00:36PM -0600, Tanmay Shah wrote:
> > > > > > > > Hi Mathieu,
> > > > > > > >
> > > > > > > > Please find my comments below.
> > > > > > > >
> > > > > > > > On 11/21/23 4:59 PM, Mathieu Poirier wrote:
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > On Fri, Nov 17, 2023 at 09:42:37AM -0800, Tanmay Shah wrote:
> > > > > > > > > > Use TCM pm domains extracted from device-tree
> > > > > > > > > > to power on/off TCM using general pm domain framework.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > > > > > > > > ---
> > > > > > > > > >
> > > > > > > > > > Changes in v7:
> > > > > > > > > > - %s/pm_dev1/pm_dev_core0/r
> > > > > > > > > > - %s/pm_dev_link1/pm_dev_core0_link/r
> > > > > > > > > > - %s/pm_dev2/pm_dev_core1/r
> > > > > > > > > > - %s/pm_dev_link2/pm_dev_core1_link/r
> > > > > > > > > > - remove pm_domain_id check to move next patch
> > > > > > > > > > - add comment about how 1st entry in pm domain list is used
> > > > > > > > > > - fix loop when jump to fail_add_pm_domains loop
> > > > > > > > > >
> > > > > > > > > > drivers/remoteproc/xlnx_r5_remoteproc.c | 215 +++++++++++++++++++++++-
> > > > > > > > > > 1 file changed, 212 insertions(+), 3 deletions(-)
> > > > > > > > > >
> > > > > > > > > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > > > index 4395edea9a64..22bccc5075a0 100644
> > > > > > > > > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > > > > > > > > @@ -16,6 +16,7 @@
> > > > > > > > > > #include <linux/of_reserved_mem.h>
> > > > > > > > > > #include <linux/platform_device.h>
> > > > > > > > > > #include <linux/remoteproc.h>
> > > > > > > > > > +#include <linux/pm_domain.h>
> > > > > > > > > >
> > > > > > > > > > #include "remoteproc_internal.h"
> > > > > > > > > >
> > > > > > > > > > @@ -102,6 +103,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > > > > > > > > > * @rproc: rproc handle
> > > > > > > > > > * @pm_domain_id: RPU CPU power domain id
> > > > > > > > > > * @ipi: pointer to mailbox information
> > > > > > > > > > + * @num_pm_dev: number of tcm pm domain devices for this core
> > > > > > > > > > + * @pm_dev_core0: pm domain virtual devices for power domain framework
> > > > > > > > > > + * @pm_dev_core0_link: pm domain device links after registration
> > > > > > > > > > + * @pm_dev_core1: used only in lockstep mode. second core's pm domain virtual devices
> > > > > > > > > > + * @pm_dev_core1_link: used only in lockstep mode. second core's pm device links after
> > > > > > > > > > + * registration
> > > > > > > > > > */
> > > > > > > > > > struct zynqmp_r5_core {
> > > > > > > > > > struct device *dev;
> > > > > > > > > > @@ -111,6 +118,11 @@ struct zynqmp_r5_core {
> > > > > > > > > > struct rproc *rproc;
> > > > > > > > > > u32 pm_domain_id;
> > > > > > > > > > struct mbox_info *ipi;
> > > > > > > > > > + int num_pm_dev;
> > > > > > > > > > + struct device **pm_dev_core0;
> > > > > > > > > > + struct device_link **pm_dev_core0_link;
> > > > > > > > > > + struct device **pm_dev_core1;
> > > > > > > > > > + struct device_link **pm_dev_core1_link;
> > > > > > > > > > };
> > > > > > > > > >
> > > > > > > > > > /**
> > > > > > > > > > @@ -651,7 +663,8 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > > > > > > > > > ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > > > > > > > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > > > > > > > if (ret < 0) {
> > > > > > > > > > - dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > > > > > > > > > + dev_err(dev, "failed to turn on TCM 0x%x",
> > > > > > > > > > + pm_domain_id);
> > > > > > > > >
> > > > > > > > > Spurious change, you should have caught that.
> > > > > > > >
> > > > > > > > Ack, need to observe changes more closely before sending them.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > goto release_tcm_lockstep;
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > @@ -758,6 +771,189 @@ static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> > > > > > > > > > return ret;
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > +static void zynqmp_r5_remove_pm_domains(struct rproc *rproc)
> > > > > > > > > > +{
> > > > > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > > > > + struct device *dev = r5_core->dev;
> > > > > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > > > > + int i;
> > > > > > > > > > +
> > > > > > > > > > + cluster = platform_get_drvdata(to_platform_device(dev->parent));
> > > > > > > > > > +
> > > > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], false);
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > > > > +
> > > > > > > > > > + if (cluster->mode == SPLIT_MODE) {
> > > > > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > > > > + return;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > > > + device_link_del(r5_core->pm_dev_core1_link[i]);
> > > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[i], false);
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > > > + r5_core->num_pm_dev = 0;
> > > > > > > > > > +}
> > > > > > > > > > +
> > > > > > > > > > +static int zynqmp_r5_add_pm_domains(struct rproc *rproc)
> > > > > > > > > > +{
> > > > > > > > > > + struct zynqmp_r5_core *r5_core = rproc->priv;
> > > > > > > > > > + struct device *dev = r5_core->dev, *dev2;
> > > > > > > > > > + struct zynqmp_r5_cluster *cluster;
> > > > > > > > > > + struct platform_device *pdev;
> > > > > > > > > > + struct device_node *np;
> > > > > > > > > > + int i, j, num_pm_dev, ret;
> > > > > > > > > > +
> > > > > > > > > > + cluster = dev_get_drvdata(dev->parent);
> > > > > > > > > > +
> > > > > > > > > > + /* get number of power-domains */
> > > > > > > > > > + num_pm_dev = of_count_phandle_with_args(r5_core->np, "power-domains",
> > > > > > > > > > + "#power-domain-cells");
> > > > > > > > > > +
> > > > > > > > > > + if (num_pm_dev <= 0)
> > > > > > > > > > + return -EINVAL;
> > > > > > > > > > +
> > > > > > > > > > + r5_core->pm_dev_core0 = kcalloc(num_pm_dev,
> > > > > > > > > > + sizeof(struct device *),
> > > > > > > > > > + GFP_KERNEL);
> > > > > > > > > > + if (!r5_core->pm_dev_core0)
> > > > > > > > > > + ret = -ENOMEM;
> > > > > > > > > > +
> > > > > > > > > > + r5_core->pm_dev_core0_link = kcalloc(num_pm_dev,
> > > > > > > > > > + sizeof(struct device_link *),
> > > > > > > > > > + GFP_KERNEL);
> > > > > > > > > > + if (!r5_core->pm_dev_core0_link) {
> > > > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > > > + return -ENOMEM;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + r5_core->num_pm_dev = num_pm_dev;
> > > > > > > > > > +
> > > > > > > > > > + /*
> > > > > > > > > > + * start from 2nd entry in power-domains property list as
> > > > > > > > > > + * for zynqmp we only add TCM power domains and not core's power domain.
> > > > > > > > > > + * 1st entry is used to configure r5 operation mode.
> > > > > > > > >
> > > > > > > > > You are still not saying _where_ ->pm_dev_core0[0] gets added.
> > > > > > > >
> > > > > > > > So, pm_dev_core0[0] isn't really need to be added for zynqmp platform, as firmware starts it with call,
> > > > > > > >
> > > > > > > > zynqmp_pm_request_wake during rproc_start callback. I will document this in next
> > > > > > > >
> > > > > > >
> > > > > > > That is exactly what I am looking for. That way people don't have to go through
> > > > > > > the entire driver trying to figure out what is happening with pm_dev_core[0].
> > > > > > >
> > > > > > > I'm also not sure about the power-up order. Logically the TCMs should be
> > > > > > > powered up before the R5 in order to put code in them. The R5s are powered in
> > > > > > > zynqmp_r5_rproc_start() but I am unclear as to where in the boot sequence the
> > > > > > > TCMs are powered - can you expand on that?
> > > > > >
> > > > > >
> > > > > > Sure. Following is call sequece
> > > > > >
> > > > > > zynqmp_r5_rproc_prepare
> > > > > >
> > > > > > zynqmp_r5_add_pm_domains -> Here TCM is powered on when device_link_add is called via zynqmp-pm-domains.c driver.
> > > > > >
> > > > > > . . .
> > > > > >
> > > > > > zynqmp_r5_rproc_start -> load firmware and Starts RPU
> > > > > >
> > > > > > So what you mentioned is correct, TCM is being powerd-on before we load firmware and start RPU.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > > revision. For new platforms pm_dev_core0[0] will be added in future.
> > > > > > >
> > > > > > > Now I'm really confused - what do you mean by "pm_dev_core0[0] will be added in
> > > > > > > future"?
> > > > > >
> > > > > >
> > > > > > ZynqMP platform has platform management firmware running on microblaze.
> > > > > >
> > > > > > This firmware design does not expect R5 pm domains to be requested explicitly.
> > > > > >
> > > > > > This means, during zynqmp_r5_rproc_start when "zynqmp_pm_request_wake" is called,
> > > > > >
> > > > > > firmware powers on R5. So, pm_dev_core[0] is not really used for ZynqMP.
> > > > > >
> > > > > > However, this design was changed for new platforms i.e. "versal" and onwards.
> > > > > >
> > > > > > Firmware of new platform expects pm domains to be requested explicitly for R5 cores before
> > > > > >
> > > > > > waking them up.
> > > > > >
> > > > > > That means, pm_dev_core[0] for R5 cores on new platform (Versal) needs to be used same as TCM.
> > > > > >
> > > > > > Then, we should wake it up on r5_core.
> > > > > >
> > > > > > To summarize:
> > > > > >
> > > > > > For zynqmp only following call needed to start R5:
> > > > > >
> > > > > > zynqmp_pm_request_wake
> > > > > >
> > > > > > For "versal" and onwards we need two calls to start R5:
> > > > > >
> > > > > > "device_link_add" and zynqmp_pm_request_wake
> > > > > >
> > > > > > So, in future pm_core_dev[0] will be used.
> > > > > >
> > > > >
> > > > > Thanks for the clarification on both front. The problem here is that we are
> > > > > keeping R5 power domain information in two different places, i.e
> > > > > zynqmp_r5_core::pm_domain_id and zynqmp_r5_core::pm_dev_core0[0].
> > > > >
> > > > > Please see if you can retreive the power domain ID from
> > > > > zynqmp_r5_core::pm_dev_core0[0]. That way you can get the power domain ID when
> > > > > calling zynqmp_pm_request_wake() and zynqmp_pm_force_pwrdwn() and get rid of
> > > > > zynqmp_r5_core::pm_domain_id.
> > > >
> > > > Hi Mathieu,
> > > >
> > > > I looked into this. Probably I can't retrieve pm_domain_id from pm_dev_core0[0],
> > > >
> > > > However, I can retrieve it from device-tree when calling zynqmp_pm_request_wake
> > > >
> > > > and zynqmp_pm_force_pwrdwn.
> > > >
> > > > zynqmp_r5_core::pm_domain_id is caching that value during probe, and use it during
> > > > rest of the driver lifecycle.
> > > >
> > > > I am okay either way, (keeping it as it is, or get it from device-tree). Let me know your
> > > >
> > > > preference.
> > > >
> > >
> > > Humm... Then I suggest to simply get rid of the device linking to
> > > deal with the TCMs' power management. From where I stand it provides
> > > more confusion than benefits, and that is without considering the
> > > extra complexity.
> >
> >
> > Do you mean to get rid of pm_dev_core0[1], and pm_dev_core0[2] as well ?
> >
>
> Yes
>
> > If yes, its preferred to use pm_domain framework to power-on/off TCM.
> >
>
> That is when the pm runtime subsystem is used extensively but it isn't
> the case here. In this scenario using device linking is adding a lot of
> complexity for something that could be done in a single line of code.
Ok fair point, please give me sometime to explore more.
I don't mind getting rid of pm domain framework usage. For some reason, if we must use it,
we can always introduce it later.
I just want to make sure if we don't use pm domains framework, then remoteproc works for
future platforms as well. I will get back to you as need to do more testing after removing pm domains framework.
Thanks,
Tanmay
> > If we want to get rid of zynqmp_r5_core::pm_domain_id, I will do what's done in
> >
> > __genpd_dev_pm_attach API where, pm_domain_id is retrieved using of_node of pm_dev_core0[*] device.
> >
> > ret = of_parse_phandle_with_args(dev->of_node, "power-domains",
> > "#power-domain-cells", index, &pd_args);
>
> That is what I asked for in my previous email, i.e use ->pm_dev_core0[0] to
> retrieve the domain ID.
>
> >
> > However, Its preferred to use pm_domain framework when power-domains are available in device-tree.
> >
> > Let me know.
> >
> > Thanks,
> >
> > Tanmay
> >
> >
> > > > Thanks,
> > > >
> > > > Tanmay
> > > >
> > > > > >
> > > > > > > >
> > > > > > > > I hope this meets expectations.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > + */
> > > > > > > > > > + for (i = 1; i < r5_core->num_pm_dev; i++) {
> > > > > > > > > > + r5_core->pm_dev_core0[i] = dev_pm_domain_attach_by_id(dev, i);
> > > > > > > > > > + if (IS_ERR_OR_NULL(r5_core->pm_dev_core0[i])) {
> > > > > > > > >
> > > > > > > > > Here IS_ERR_OR_NULL() is used while two if conditions for NULL and an error
> > > > > > > > > code are used in the loop for the lockstep mode. Please pick one heuristic and
> > > > > > > > > stick with it. I have no preference on which one.
> > > > > > > >
> > > > > > > > Ok, I think IS_ERR_OR_NULL is more cleaner, I will address it in next revision.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > + dev_dbg(dev, "failed to attach pm domain %d, ret=%ld\n", i,
> > > > > > > > > > + PTR_ERR(r5_core->pm_dev_core0[i]));
> > > > > > > > > > + ret = -EINVAL;
> > > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > > + }
> > > > > > > > > > + r5_core->pm_dev_core0_link[i] = device_link_add(dev,
> > > > > > > > > > + r5_core->pm_dev_core0[i],
> > > > > > > > > > + DL_FLAG_STATELESS |
> > > > > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > > > > + if (!r5_core->pm_dev_core0_link[i]) {
> > > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > > > > + r5_core->pm_dev_core0[i] = NULL;
> > > > > > > > > > + ret = -EINVAL;
> > > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > > + }
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + if (cluster->mode == SPLIT_MODE)
> > > > > > > > > > + return 0;
> > > > > > > > > > +
> > > > > > > > > > + r5_core->pm_dev_core1 = kcalloc(num_pm_dev,
> > > > > > > > > > + sizeof(struct device *),
> > > > > > > > > > + GFP_KERNEL);
> > > > > > > > > > + if (!r5_core->pm_dev_core1) {
> > > > > > > > > > + ret = -ENOMEM;
> > > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + r5_core->pm_dev_core1_link = kcalloc(num_pm_dev,
> > > > > > > > > > + sizeof(struct device_link *),
> > > > > > > > > > + GFP_KERNEL);
> > > > > > > > > > + if (!r5_core->pm_dev_core1_link) {
> > > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > > + ret = -ENOMEM;
> > > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + /* get second core's device to detach its power-domains */
> > > > > > > > > > + np = of_get_next_child(cluster->dev->of_node, of_node_get(dev->of_node));
> > > > > > > > > > +
> > > > > > > > > > + pdev = of_find_device_by_node(np);
> > > > > > > > > > + if (!pdev) {
> > > > > > > > > > + dev_err(cluster->dev, "core1 platform device not available\n");
> > > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > > > + ret = -EINVAL;
> > > > > > > > > > + goto fail_add_pm_domains;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + dev2 = &pdev->dev;
> > > > > > > > > > +
> > > > > > > > > > + /* for zynqmp we only add TCM power domains and not core's power domain */
> > > > > > > > > > + for (j = 1; j < r5_core->num_pm_dev; j++) {
> > > > > > > > > > + r5_core->pm_dev_core1[j] = dev_pm_domain_attach_by_id(dev2, j);
> > > > > > > > > > + if (!r5_core->pm_dev_core1[j]) {
> > > > > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > > > > + ret = -EINVAL;
> > > > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > > > + } else if (IS_ERR(r5_core->pm_dev_core1[j])) {
> > > > > > > > > > + dev_dbg(dev, "can't attach to pm domain %d\n", j);
> > > > > > > > > > + ret = PTR_ERR(r5_core->pm_dev_core1[j]);
> > > > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > + r5_core->pm_dev_core1_link[j] = device_link_add(dev,
> > > > > > > > > > + r5_core->pm_dev_core1[j],
> > > > > > > > > > + DL_FLAG_STATELESS |
> > > > > > > > > > + DL_FLAG_RPM_ACTIVE |
> > > > > > > > > > + DL_FLAG_PM_RUNTIME);
> > > > > > > > > > + if (!r5_core->pm_dev_core1_link[j]) {
> > > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > > > > + r5_core->pm_dev_core1[j] = NULL;
> > > > > > > > > > + ret = -ENODEV;
> > > > > > > > > > + goto fail_add_pm_domains_lockstep;
> > > > > > > > > > + }
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > +fail_add_pm_domains_lockstep:
> > > > > > > > > > + while (--j >= 0) {
> > > > > > > > > > + device_link_del(r5_core->pm_dev_core1_link[j]);
> > > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core1[j], true);
> > > > > > > > > > + }
> > > > > > > > > > + kfree(r5_core->pm_dev_core1);
> > > > > > > > > > + r5_core->pm_dev_core1 = NULL;
> > > > > > > > > > + kfree(r5_core->pm_dev_core1_link);
> > > > > > > > > > + r5_core->pm_dev_core1_link = NULL;
> > > > > > > > > > +
> > > > > > > > > > +fail_add_pm_domains:
> > > > > > > > > > + while (--i >= 0) {
> > > > > > > > > > + device_link_del(r5_core->pm_dev_core0_link[i]);
> > > > > > > > > > + dev_pm_domain_detach(r5_core->pm_dev_core0[i], true);
> > > > > > > > > > + }
> > > > > > > > > > + kfree(r5_core->pm_dev_core0);
> > > > > > > > > > + r5_core->pm_dev_core0 = NULL;
> > > > > > > > > > + kfree(r5_core->pm_dev_core0_link);
> > > > > > > > > > + r5_core->pm_dev_core0_link = NULL;
> > > > > > > > > > +
> > > > > > > > >
> > > > > > > > > The error path is much cleaner and readable now.
> > > > > > > > >
> > > > > > > > > I will continue tomorrow.
> > > > > > > > >
> > > > > > > > > Mathieu
> > > > > > > > >
> > > > > > > > > > + return ret;
> > > > > > > > > > +}
> > > > > > > > > > +
> > > > > > > > > > /**
> > > > > > > > > > * zynqmp_r5_rproc_prepare()
> > > > > > > > > > * adds carveouts for TCM bank and reserved memory regions
> > > > > > > > > > @@ -770,19 +966,30 @@ static int zynqmp_r5_rproc_prepare(struct rproc *rproc)
> > > > > > > > > > {
> > > > > > > > > > int ret;
> > > > > > > > > >
> > > > > > > > > > + ret = zynqmp_r5_add_pm_domains(rproc);
> > > > > > > > > > + if (ret) {
> > > > > > > > > > + dev_err(&rproc->dev, "failed to add pm domains\n");
> > > > > > > > > > + return ret;
> > > > > > > > > > + }
> > > > > > > > > > +
> > > > > > > > > > ret = add_tcm_banks(rproc);
> > > > > > > > > > if (ret) {
> > > > > > > > > > dev_err(&rproc->dev, "failed to get TCM banks, err %d\n", ret);
> > > > > > > > > > - return ret;
> > > > > > > > > > + goto fail_prepare;
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > ret = add_mem_regions_carveout(rproc);
> > > > > > > > > > if (ret) {
> > > > > > > > > > dev_err(&rproc->dev, "failed to get reserve mem regions %d\n", ret);
> > > > > > > > > > - return ret;
> > > > > > > > > > + goto fail_prepare;
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > return 0;
> > > > > > > > > > +
> > > > > > > > > > +fail_prepare:
> > > > > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > > > > +
> > > > > > > > > > + return ret;
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > /**
> > > > > > > > > > @@ -801,6 +1008,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > > > > > > > > >
> > > > > > > > > > r5_core = rproc->priv;
> > > > > > > > > >
> > > > > > > > > > + zynqmp_r5_remove_pm_domains(rproc);
> > > > > > > > > > +
> > > > > > > > > > for (i = 0; i < r5_core->tcm_bank_count; i++) {
> > > > > > > > > > pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > > > > > > > > > if (zynqmp_pm_release_node(pm_domain_id))
> > > > > > > > > > --
> > > > > > > > > > 2.25.1
> > > > > > > > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 38+ messages in thread
end of thread, other threads:[~2023-12-08 16:06 UTC | newest]
Thread overview: 38+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-11-17 17:42 [PATCH v7 0/4] add zynqmp TCM bindings Tanmay Shah
2023-11-17 17:42 ` Tanmay Shah
2023-11-17 17:42 ` [PATCH v7 1/4] dt-bindings: remoteproc: add Tightly Coupled Memory (TCM) bindings Tanmay Shah
2023-11-17 17:42 ` Tanmay Shah
2023-11-17 17:42 ` [PATCH v7 2/4] dts: zynqmp: add properties for TCM in remoteproc Tanmay Shah
2023-11-17 17:42 ` Tanmay Shah
2023-11-17 17:42 ` [PATCH v7 3/4] remoteproc: zynqmp: add pm domains support Tanmay Shah
2023-11-17 17:42 ` Tanmay Shah
2023-11-21 22:59 ` Mathieu Poirier
2023-11-21 22:59 ` Mathieu Poirier
2023-11-22 21:00 ` Tanmay Shah
2023-11-22 21:00 ` Tanmay Shah
2023-11-23 18:11 ` Mathieu Poirier
2023-11-23 18:11 ` Mathieu Poirier
2023-11-27 16:33 ` Tanmay Shah
2023-11-27 16:33 ` Tanmay Shah
2023-11-29 17:10 ` Mathieu Poirier
2023-11-29 17:10 ` Mathieu Poirier
2023-11-29 17:42 ` Tanmay Shah
2023-11-29 17:42 ` Tanmay Shah
2023-12-01 18:10 ` Tanmay Shah
2023-12-01 18:10 ` Tanmay Shah
2023-12-06 15:43 ` Mathieu Poirier
2023-12-06 15:43 ` Mathieu Poirier
2023-12-06 18:06 ` Tanmay Shah
2023-12-06 18:06 ` Tanmay Shah
2023-12-08 15:51 ` Mathieu Poirier
2023-12-08 15:51 ` Mathieu Poirier
2023-12-08 16:05 ` Tanmay Shah
2023-12-08 16:05 ` Tanmay Shah
2023-11-22 17:12 ` Mathieu Poirier
2023-11-22 17:12 ` Mathieu Poirier
2023-11-17 17:42 ` [PATCH v7 4/4] remoteproc: zynqmp: parse TCM from device tree Tanmay Shah
2023-11-17 17:42 ` Tanmay Shah
2023-11-22 17:51 ` Mathieu Poirier
2023-11-22 17:51 ` Mathieu Poirier
2023-11-22 20:27 ` Mathieu Poirier
2023-11-22 20:27 ` Mathieu Poirier
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.