* [PATCH v2 0/6] Add Xilinx RPU subsystem support
@ 2021-11-23 6:20 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
This patch series adds bindings document for RPU subsystem found on Xilinx
ZynqMP platforms. It also adds device nodes and driver to enable RPU subsytem
in split mode and lockstep mode.
Xilinx ZynqMP platform contains Remote Processing Unit(RPU). RPU subsystem
contains two arm cortex r5f cores. RPU subsystem can be configured in
split mode, locsktep mode and single-cpu mode.
RPU subsystem also contains 4 Tightly Coupled Memory(TCM) banks. In lockstep
mode, all 4 banks are combined and total of 256KB memory is made available to
r5 core0. In split mode, both cores can access two TCM banks i.e. 128 KB.
RPU can also fetch data and execute instructions from DDR memory along with
TCM memory.
Changes in v2:
- Remove proprietary copyright footer from cover letter
Ben Levinsky (3):
firmware: xilinx: Add ZynqMP firmware ioctl enums for RPU
configuration.
firmware: xilinx: Add shutdown/wakeup APIs
firmware: xilinx: Add RPU configuration APIs
Tanmay Shah (3):
dt-bindings: remoteproc: Add Xilinx RPU subsystem bindings
arm64: dts: xilinx: zynqmp: Add RPU subsystem device node
drivers: remoteproc: Add Xilinx r5 remoteproc driver
.../bindings/remoteproc/xlnx,r5f-rproc.yaml | 139 +++
arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 17 +
drivers/firmware/xilinx/zynqmp.c | 96 ++
drivers/remoteproc/Kconfig | 12 +
drivers/remoteproc/Makefile | 1 +
drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++
include/dt-bindings/power/xlnx-zynqmp-power.h | 6 +
include/linux/firmware/xlnx-zynqmp.h | 60 ++
8 files changed, 1290 insertions(+)
create mode 100644 Documentation/devicetree/bindings/remoteproc/xlnx,r5f-rproc.yaml
create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
base-commit: c1fe10d238c0256a77dbc4bf6493b9782b2a218d
--
2.25.1
^ permalink raw reply [flat|nested] 56+ messages in thread
* [PATCH v2 0/6] Add Xilinx RPU subsystem support
@ 2021-11-23 6:20 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
This patch series adds bindings document for RPU subsystem found on Xilinx
ZynqMP platforms. It also adds device nodes and driver to enable RPU subsytem
in split mode and lockstep mode.
Xilinx ZynqMP platform contains Remote Processing Unit(RPU). RPU subsystem
contains two arm cortex r5f cores. RPU subsystem can be configured in
split mode, locsktep mode and single-cpu mode.
RPU subsystem also contains 4 Tightly Coupled Memory(TCM) banks. In lockstep
mode, all 4 banks are combined and total of 256KB memory is made available to
r5 core0. In split mode, both cores can access two TCM banks i.e. 128 KB.
RPU can also fetch data and execute instructions from DDR memory along with
TCM memory.
Changes in v2:
- Remove proprietary copyright footer from cover letter
Ben Levinsky (3):
firmware: xilinx: Add ZynqMP firmware ioctl enums for RPU
configuration.
firmware: xilinx: Add shutdown/wakeup APIs
firmware: xilinx: Add RPU configuration APIs
Tanmay Shah (3):
dt-bindings: remoteproc: Add Xilinx RPU subsystem bindings
arm64: dts: xilinx: zynqmp: Add RPU subsystem device node
drivers: remoteproc: Add Xilinx r5 remoteproc driver
.../bindings/remoteproc/xlnx,r5f-rproc.yaml | 139 +++
arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 17 +
drivers/firmware/xilinx/zynqmp.c | 96 ++
drivers/remoteproc/Kconfig | 12 +
drivers/remoteproc/Makefile | 1 +
drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++
include/dt-bindings/power/xlnx-zynqmp-power.h | 6 +
include/linux/firmware/xlnx-zynqmp.h | 60 ++
8 files changed, 1290 insertions(+)
create mode 100644 Documentation/devicetree/bindings/remoteproc/xlnx,r5f-rproc.yaml
create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
base-commit: c1fe10d238c0256a77dbc4bf6493b9782b2a218d
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* [PATCH v2 1/6] dt-bindings: remoteproc: Add Xilinx RPU subsystem bindings
2021-11-23 6:20 ` Tanmay Shah
@ 2021-11-23 6:20 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
Xilinx ZynqMP platform has dual-core ARM Cortex R5 Realtime Processing
Unit(RPU) subsystem. This patch adds dt-bindings for RPU subsystem (cluster).
Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
---
.../bindings/remoteproc/xlnx,r5f-rproc.yaml | 139 ++++++++++++++++++
include/dt-bindings/power/xlnx-zynqmp-power.h | 6 +
2 files changed, 145 insertions(+)
create mode 100644 Documentation/devicetree/bindings/remoteproc/xlnx,r5f-rproc.yaml
diff --git a/Documentation/devicetree/bindings/remoteproc/xlnx,r5f-rproc.yaml b/Documentation/devicetree/bindings/remoteproc/xlnx,r5f-rproc.yaml
new file mode 100644
index 000000000000..d43f0b16ad7f
--- /dev/null
+++ b/Documentation/devicetree/bindings/remoteproc/xlnx,r5f-rproc.yaml
@@ -0,0 +1,139 @@
+# SPDX-License-Identifier: (GPL-2.0-only or BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/remoteproc/xlnx,r5f-rproc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Xilinx R5F processor subsystem
+
+maintainers:
+ - Ben Levinsky <ben.levinsky@xilinx.com>
+ - Tanmay Shah <tanmay.shah@xilinx.com>
+
+description: |
+ The Xilinx platforms include a pair of Cortex-R5F processors (RPU) for
+ real-time processing based on the Cortex-R5F processor core from ARM.
+ The Cortex-R5F processor implements the Arm v7-R architecture and includes a
+ floating-point unit that implements the Arm VFPv3 instruction set.
+
+properties:
+ compatible:
+ const: xlnx,zynqmp-r5fss
+
+ xlnx,cluster-mode:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ description: |
+ The RPU MPCore can operate in split mode(Dual-processor performance), Safety
+ lock-step mode(Both RPU cores execute the same code in lock-step,
+ clock-for-clock) or Single CPU mode (RPU core 0 can be held in reset while
+ core 1 runs normally). The processor does not support dynamic configuration.
+ Switching between modes is only permitted immediately after a processor reset.
+ If set to 1 then lockstep mode and if 0 then split mode.
+ If set to 2 then single CPU mode. When not defined, default will be lockstep mode.
+
+ "#address-cells":
+ const: 1
+
+ "#size-cells":
+ const: 1
+
+ reg:
+ items:
+ - description: RPU subsystem status and control registers
+
+patternProperties:
+ "^r5f-[a-f0-9]+$":
+ type: object
+ description: |
+ The RPU is located in the Low Power Domain of the Processor Subsystem.
+ Each processor includes separate L1 instruction and data caches and
+ tightly coupled memories (TCM). System memory is cacheable, but the TCM
+ memory space is non-cacheable.
+
+ Each RPU contains one 64KB memory and two 32KB memories that
+ are accessed via the TCM A and B port interfaces, for a total of 128KB
+ per processor. In lock-step mode, the processor has access to 256KB of
+ TCM memory.
+
+ properties:
+ compatible:
+ const: xlnx,zynqmp-r5f
+
+ power-domains:
+ description: |
+ phandle to a PM domain provider node and an args specifier containing
+ the r5f0 and r5f1 node id value.
+
+ reg:
+ items:
+ - description: RPU0 and RPU1 control and status registers
+
+ mboxes:
+ items:
+ - description: |
+ Bi-directional channel to send data to RPU and receive ack from RPU.
+ Request and response message buffers are available and each buffer is 32 bytes.
+ - description: |
+ Bi-directional channel to receive data from RPU and send ack from RPU.
+ Request and response message buffers are available and each buffer is 32 bytes.
+ minItems: 1
+
+ mbox-names:
+ items:
+ - const: tx
+ - const: rx
+ minItems: 1
+
+ sram:
+ $ref: /schemas/types.yaml#/definitions/phandle-array
+ minItems: 1
+ description: |
+ phandles to one or more reserved on-chip SRAM regions. Other than TCM,
+ the RPU can execute instructions and access data from, the OCM memory,
+ the main DDR memory, and other system memories.
+
+ The regions should be defined as child nodes of the respective SRAM
+ node, and should be defined as per the generic bindings in,
+ Documentation/devicetree/bindings/sram/sram.yaml
+
+ memory-region:
+ $ref: /schemas/types.yaml#/definitions/phandle-array
+ description: |
+ List of phandles to the reserved memory regions associated with the
+ remoteproc device. This is variable and describes the memories shared with
+ the remote processor (e.g. remoteproc firmware and carveouts, rpmsg
+ vrings, ...). This reserved memory region will be allocated on DDR memory.
+ See Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
+
+ required:
+ - compatible
+ - power-domains
+
+ unevaluatedProperties: false
+
+required:
+ - compatible
+
+additionalProperties: false
+
+examples:
+ - |
+ r5fss: r5fss@ff9a0000 {
+ compatible = "xlnx,zynqmp-r5fss";
+ xlnx,cluster-mode = <1>;
+
+ #address-cells = <1>;
+ #size-cells = <1>;
+ reg = <0xff9a0000 0x228>;
+
+ r5f-0 {
+ compatible = "xlnx,zynqmp-r5f";
+ power-domains = <&zynqmp_firmware 0x7>;
+ };
+
+ r5f-1 {
+ compatible = "xlnx,zynqmp-r5f";
+ power-domains = <&zynqmp_firmware 0x8>;
+ };
+ };
+...
diff --git a/include/dt-bindings/power/xlnx-zynqmp-power.h b/include/dt-bindings/power/xlnx-zynqmp-power.h
index 0d9a412fd5e0..618024cbb20d 100644
--- a/include/dt-bindings/power/xlnx-zynqmp-power.h
+++ b/include/dt-bindings/power/xlnx-zynqmp-power.h
@@ -6,6 +6,12 @@
#ifndef _DT_BINDINGS_ZYNQMP_POWER_H
#define _DT_BINDINGS_ZYNQMP_POWER_H
+#define PD_RPU_0 7
+#define PD_RPU_1 8
+#define PD_R5_0_ATCM 15
+#define PD_R5_0_BTCM 16
+#define PD_R5_1_ATCM 17
+#define PD_R5_1_BTCM 18
#define PD_USB_0 22
#define PD_USB_1 23
#define PD_TTC_0 24
--
2.25.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 1/6] dt-bindings: remoteproc: Add Xilinx RPU subsystem bindings
@ 2021-11-23 6:20 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
Xilinx ZynqMP platform has dual-core ARM Cortex R5 Realtime Processing
Unit(RPU) subsystem. This patch adds dt-bindings for RPU subsystem (cluster).
Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
---
.../bindings/remoteproc/xlnx,r5f-rproc.yaml | 139 ++++++++++++++++++
include/dt-bindings/power/xlnx-zynqmp-power.h | 6 +
2 files changed, 145 insertions(+)
create mode 100644 Documentation/devicetree/bindings/remoteproc/xlnx,r5f-rproc.yaml
diff --git a/Documentation/devicetree/bindings/remoteproc/xlnx,r5f-rproc.yaml b/Documentation/devicetree/bindings/remoteproc/xlnx,r5f-rproc.yaml
new file mode 100644
index 000000000000..d43f0b16ad7f
--- /dev/null
+++ b/Documentation/devicetree/bindings/remoteproc/xlnx,r5f-rproc.yaml
@@ -0,0 +1,139 @@
+# SPDX-License-Identifier: (GPL-2.0-only or BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/remoteproc/xlnx,r5f-rproc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Xilinx R5F processor subsystem
+
+maintainers:
+ - Ben Levinsky <ben.levinsky@xilinx.com>
+ - Tanmay Shah <tanmay.shah@xilinx.com>
+
+description: |
+ The Xilinx platforms include a pair of Cortex-R5F processors (RPU) for
+ real-time processing based on the Cortex-R5F processor core from ARM.
+ The Cortex-R5F processor implements the Arm v7-R architecture and includes a
+ floating-point unit that implements the Arm VFPv3 instruction set.
+
+properties:
+ compatible:
+ const: xlnx,zynqmp-r5fss
+
+ xlnx,cluster-mode:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ description: |
+ The RPU MPCore can operate in split mode(Dual-processor performance), Safety
+ lock-step mode(Both RPU cores execute the same code in lock-step,
+ clock-for-clock) or Single CPU mode (RPU core 0 can be held in reset while
+ core 1 runs normally). The processor does not support dynamic configuration.
+ Switching between modes is only permitted immediately after a processor reset.
+ If set to 1 then lockstep mode and if 0 then split mode.
+ If set to 2 then single CPU mode. When not defined, default will be lockstep mode.
+
+ "#address-cells":
+ const: 1
+
+ "#size-cells":
+ const: 1
+
+ reg:
+ items:
+ - description: RPU subsystem status and control registers
+
+patternProperties:
+ "^r5f-[a-f0-9]+$":
+ type: object
+ description: |
+ The RPU is located in the Low Power Domain of the Processor Subsystem.
+ Each processor includes separate L1 instruction and data caches and
+ tightly coupled memories (TCM). System memory is cacheable, but the TCM
+ memory space is non-cacheable.
+
+ Each RPU contains one 64KB memory and two 32KB memories that
+ are accessed via the TCM A and B port interfaces, for a total of 128KB
+ per processor. In lock-step mode, the processor has access to 256KB of
+ TCM memory.
+
+ properties:
+ compatible:
+ const: xlnx,zynqmp-r5f
+
+ power-domains:
+ description: |
+ phandle to a PM domain provider node and an args specifier containing
+ the r5f0 and r5f1 node id value.
+
+ reg:
+ items:
+ - description: RPU0 and RPU1 control and status registers
+
+ mboxes:
+ items:
+ - description: |
+ Bi-directional channel to send data to RPU and receive ack from RPU.
+ Request and response message buffers are available and each buffer is 32 bytes.
+ - description: |
+ Bi-directional channel to receive data from RPU and send ack from RPU.
+ Request and response message buffers are available and each buffer is 32 bytes.
+ minItems: 1
+
+ mbox-names:
+ items:
+ - const: tx
+ - const: rx
+ minItems: 1
+
+ sram:
+ $ref: /schemas/types.yaml#/definitions/phandle-array
+ minItems: 1
+ description: |
+ phandles to one or more reserved on-chip SRAM regions. Other than TCM,
+ the RPU can execute instructions and access data from, the OCM memory,
+ the main DDR memory, and other system memories.
+
+ The regions should be defined as child nodes of the respective SRAM
+ node, and should be defined as per the generic bindings in,
+ Documentation/devicetree/bindings/sram/sram.yaml
+
+ memory-region:
+ $ref: /schemas/types.yaml#/definitions/phandle-array
+ description: |
+ List of phandles to the reserved memory regions associated with the
+ remoteproc device. This is variable and describes the memories shared with
+ the remote processor (e.g. remoteproc firmware and carveouts, rpmsg
+ vrings, ...). This reserved memory region will be allocated on DDR memory.
+ See Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
+
+ required:
+ - compatible
+ - power-domains
+
+ unevaluatedProperties: false
+
+required:
+ - compatible
+
+additionalProperties: false
+
+examples:
+ - |
+ r5fss: r5fss@ff9a0000 {
+ compatible = "xlnx,zynqmp-r5fss";
+ xlnx,cluster-mode = <1>;
+
+ #address-cells = <1>;
+ #size-cells = <1>;
+ reg = <0xff9a0000 0x228>;
+
+ r5f-0 {
+ compatible = "xlnx,zynqmp-r5f";
+ power-domains = <&zynqmp_firmware 0x7>;
+ };
+
+ r5f-1 {
+ compatible = "xlnx,zynqmp-r5f";
+ power-domains = <&zynqmp_firmware 0x8>;
+ };
+ };
+...
diff --git a/include/dt-bindings/power/xlnx-zynqmp-power.h b/include/dt-bindings/power/xlnx-zynqmp-power.h
index 0d9a412fd5e0..618024cbb20d 100644
--- a/include/dt-bindings/power/xlnx-zynqmp-power.h
+++ b/include/dt-bindings/power/xlnx-zynqmp-power.h
@@ -6,6 +6,12 @@
#ifndef _DT_BINDINGS_ZYNQMP_POWER_H
#define _DT_BINDINGS_ZYNQMP_POWER_H
+#define PD_RPU_0 7
+#define PD_RPU_1 8
+#define PD_R5_0_ATCM 15
+#define PD_R5_0_BTCM 16
+#define PD_R5_1_ATCM 17
+#define PD_R5_1_BTCM 18
#define PD_USB_0 22
#define PD_USB_1 23
#define PD_TTC_0 24
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 2/6] arm64: dts: xilinx: zynqmp: Add RPU subsystem device node
2021-11-23 6:20 ` Tanmay Shah
@ 2021-11-23 6:20 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
RPU subsystem can be configured in cluster-mode or split mode.
Also each r5 core has separate power domains.
Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
---
arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
index 28dccb891a53..f4fb98ccb1b5 100644
--- a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
+++ b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
@@ -654,6 +654,23 @@ qspi: spi@ff0f0000 {
power-domains = <&zynqmp_firmware PD_QSPI>;
};
+ r5fss: r5fss@ff9a0000 {
+ compatible = "xlnx,zynqmp-r5fss";
+ xlnx,cluster-mode = <1>;
+
+ reg = <0x0 0Xff9a0000 0x0 0x228>;
+
+ r5f_core0: r5f-0 {
+ compatible = "xlnx,zynqmp-r5f";
+ power-domains = <&zynqmp_firmware PD_RPU_0>;
+ };
+
+ r5f_core1: r5f-1 {
+ compatible = "xlnx,zynqmp-r5f";
+ power-domains = <&zynqmp_firmware PD_RPU_1>;
+ };
+ };
+
psgtr: phy@fd400000 {
compatible = "xlnx,zynqmp-psgtr-v1.1";
status = "disabled";
--
2.25.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 2/6] arm64: dts: xilinx: zynqmp: Add RPU subsystem device node
@ 2021-11-23 6:20 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
RPU subsystem can be configured in cluster-mode or split mode.
Also each r5 core has separate power domains.
Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
---
arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
index 28dccb891a53..f4fb98ccb1b5 100644
--- a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
+++ b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
@@ -654,6 +654,23 @@ qspi: spi@ff0f0000 {
power-domains = <&zynqmp_firmware PD_QSPI>;
};
+ r5fss: r5fss@ff9a0000 {
+ compatible = "xlnx,zynqmp-r5fss";
+ xlnx,cluster-mode = <1>;
+
+ reg = <0x0 0Xff9a0000 0x0 0x228>;
+
+ r5f_core0: r5f-0 {
+ compatible = "xlnx,zynqmp-r5f";
+ power-domains = <&zynqmp_firmware PD_RPU_0>;
+ };
+
+ r5f_core1: r5f-1 {
+ compatible = "xlnx,zynqmp-r5f";
+ power-domains = <&zynqmp_firmware PD_RPU_1>;
+ };
+ };
+
psgtr: phy@fd400000 {
compatible = "xlnx,zynqmp-psgtr-v1.1";
status = "disabled";
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 3/6] firmware: xilinx: Add ZynqMP firmware ioctl enums for RPU configuration.
2021-11-23 6:20 ` Tanmay Shah
@ 2021-11-23 6:20 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
From: Ben Levinsky <ben.levinsky@xilinx.com>
Add ZynqMP firmware ioctl enums for RPU configuration and TCM Nodes for
later use via request_node and release_node
Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
---
include/linux/firmware/xlnx-zynqmp.h | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index 56b426fe020c..92bf3ae20524 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -111,6 +111,10 @@ enum pm_ret_status {
};
enum pm_ioctl_id {
+ IOCTL_GET_RPU_OPER_MODE = 0,
+ IOCTL_SET_RPU_OPER_MODE = 1,
+ IOCTL_RPU_BOOT_ADDR_CONFIG = 2,
+ IOCTL_TCM_COMB_CONFIG = 3,
IOCTL_SD_DLL_RESET = 6,
IOCTL_SET_SD_TAPDELAY = 7,
IOCTL_SET_PLL_FRAC_MODE = 8,
@@ -142,6 +146,21 @@ enum pm_query_id {
PM_QID_CLOCK_GET_MAX_DIVISOR = 13,
};
+enum rpu_oper_mode {
+ PM_RPU_MODE_LOCKSTEP = 0,
+ PM_RPU_MODE_SPLIT = 1,
+};
+
+enum rpu_boot_mem {
+ PM_RPU_BOOTMEM_LOVEC = 0,
+ PM_RPU_BOOTMEM_HIVEC = 1,
+};
+
+enum rpu_tcm_comb {
+ PM_RPU_TCM_SPLIT = 0,
+ PM_RPU_TCM_COMB = 1,
+};
+
enum zynqmp_pm_reset_action {
PM_RESET_ACTION_RELEASE = 0,
PM_RESET_ACTION_ASSERT = 1,
--
2.25.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 3/6] firmware: xilinx: Add ZynqMP firmware ioctl enums for RPU configuration.
@ 2021-11-23 6:20 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
From: Ben Levinsky <ben.levinsky@xilinx.com>
Add ZynqMP firmware ioctl enums for RPU configuration and TCM Nodes for
later use via request_node and release_node
Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
---
include/linux/firmware/xlnx-zynqmp.h | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index 56b426fe020c..92bf3ae20524 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -111,6 +111,10 @@ enum pm_ret_status {
};
enum pm_ioctl_id {
+ IOCTL_GET_RPU_OPER_MODE = 0,
+ IOCTL_SET_RPU_OPER_MODE = 1,
+ IOCTL_RPU_BOOT_ADDR_CONFIG = 2,
+ IOCTL_TCM_COMB_CONFIG = 3,
IOCTL_SD_DLL_RESET = 6,
IOCTL_SET_SD_TAPDELAY = 7,
IOCTL_SET_PLL_FRAC_MODE = 8,
@@ -142,6 +146,21 @@ enum pm_query_id {
PM_QID_CLOCK_GET_MAX_DIVISOR = 13,
};
+enum rpu_oper_mode {
+ PM_RPU_MODE_LOCKSTEP = 0,
+ PM_RPU_MODE_SPLIT = 1,
+};
+
+enum rpu_boot_mem {
+ PM_RPU_BOOTMEM_LOVEC = 0,
+ PM_RPU_BOOTMEM_HIVEC = 1,
+};
+
+enum rpu_tcm_comb {
+ PM_RPU_TCM_SPLIT = 0,
+ PM_RPU_TCM_COMB = 1,
+};
+
enum zynqmp_pm_reset_action {
PM_RESET_ACTION_RELEASE = 0,
PM_RESET_ACTION_ASSERT = 1,
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 4/6] firmware: xilinx: Add shutdown/wakeup APIs
2021-11-23 6:20 ` Tanmay Shah
@ 2021-11-23 6:20 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
From: Ben Levinsky <ben.levinsky@xilinx.com>
Add shutdown/wakeup a resource eemi operations to shutdown
or bringup a resource.
Note alignment of args matches convention of other fn's in this file.
The reason being that the long fn name results in aligned args that
otherwise go over 80 chars so shift right to avoid this
Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
---
drivers/firmware/xilinx/zynqmp.c | 35 ++++++++++++++++++++++++++++
include/linux/firmware/xlnx-zynqmp.h | 23 ++++++++++++++++++
2 files changed, 58 insertions(+)
diff --git a/drivers/firmware/xilinx/zynqmp.c b/drivers/firmware/xilinx/zynqmp.c
index a3cadbaf3cba..e772404235c1 100644
--- a/drivers/firmware/xilinx/zynqmp.c
+++ b/drivers/firmware/xilinx/zynqmp.c
@@ -990,6 +990,41 @@ int zynqmp_pm_release_node(const u32 node)
}
EXPORT_SYMBOL_GPL(zynqmp_pm_release_node);
+/**
+ * zynqmp_pm_force_pwrdwn - PM call to request for another PU or subsystem to
+ * be powered down forcefully
+ * @node: Node ID of the targeted PU or subsystem
+ * @ack: Flag to specify whether acknowledge is requested
+ *
+ * Return: status, either success or error+reason
+ */
+int zynqmp_pm_force_pwrdwn(const u32 node,
+ const enum zynqmp_pm_request_ack ack)
+{
+ return zynqmp_pm_invoke_fn(PM_FORCE_POWERDOWN, node, ack, 0, 0, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_force_pwrdwn);
+
+/**
+ * zynqmp_pm_request_wake - PM call to wake up selected master or subsystem
+ * @node: Node ID of the master or subsystem
+ * @set_addr: Specifies whether the address argument is relevant
+ * @address: Address from which to resume when woken up
+ * @ack: Flag to specify whether acknowledge requested
+ *
+ * Return: status, either success or error+reason
+ */
+int zynqmp_pm_request_wake(const u32 node,
+ const bool set_addr,
+ const u64 address,
+ const enum zynqmp_pm_request_ack ack)
+{
+ /* set_addr flag is encoded into 1st bit of address */
+ return zynqmp_pm_invoke_fn(PM_REQUEST_WAKEUP, node, address | set_addr,
+ address >> 32, ack, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_request_wake);
+
/**
* zynqmp_pm_set_requirement() - PM call to set requirement for PM slaves
* @node: Node ID of the slave
diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index 92bf3ae20524..5da3d490e80e 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -12,6 +12,7 @@
#ifndef __FIRMWARE_ZYNQMP_H__
#define __FIRMWARE_ZYNQMP_H__
+#include <linux/types.h>
#include <linux/err.h>
@@ -66,6 +67,8 @@
enum pm_api_id {
PM_GET_API_VERSION = 1,
+ PM_FORCE_POWERDOWN = 8,
+ PM_REQUEST_WAKEUP = 10,
PM_SYSTEM_SHUTDOWN = 12,
PM_REQUEST_NODE = 13,
PM_RELEASE_NODE = 14,
@@ -435,6 +438,12 @@ int zynqmp_pm_pinctrl_get_config(const u32 pin, const u32 param,
int zynqmp_pm_pinctrl_set_config(const u32 pin, const u32 param,
u32 value);
int zynqmp_pm_load_pdi(const u32 src, const u64 address);
+int zynqmp_pm_force_pwrdwn(const u32 target,
+ const enum zynqmp_pm_request_ack ack);
+int zynqmp_pm_request_wake(const u32 node,
+ const bool set_addr,
+ const u64 address,
+ const enum zynqmp_pm_request_ack ack);
#else
static inline int zynqmp_pm_get_api_version(u32 *version)
{
@@ -620,6 +629,12 @@ static inline int zynqmp_pm_pinctrl_request(const u32 pin)
return -ENODEV;
}
+static inline int zynqmp_pm_force_pwrdwn(const u32 target,
+ const enum zynqmp_pm_request_ack ack)
+{
+ return -ENODEV;
+}
+
static inline int zynqmp_pm_pinctrl_release(const u32 pin)
{
return -ENODEV;
@@ -651,6 +666,14 @@ static inline int zynqmp_pm_load_pdi(const u32 src, const u64 address)
{
return -ENODEV;
}
+
+static inline int zynqmp_pm_request_wake(const u32 node,
+ const bool set_addr,
+ const u64 address,
+ const enum zynqmp_pm_request_ack ack)
+{
+ return -ENODEV;
+}
#endif
#endif /* __FIRMWARE_ZYNQMP_H__ */
--
2.25.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 4/6] firmware: xilinx: Add shutdown/wakeup APIs
@ 2021-11-23 6:20 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
From: Ben Levinsky <ben.levinsky@xilinx.com>
Add shutdown/wakeup a resource eemi operations to shutdown
or bringup a resource.
Note alignment of args matches convention of other fn's in this file.
The reason being that the long fn name results in aligned args that
otherwise go over 80 chars so shift right to avoid this
Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
---
drivers/firmware/xilinx/zynqmp.c | 35 ++++++++++++++++++++++++++++
include/linux/firmware/xlnx-zynqmp.h | 23 ++++++++++++++++++
2 files changed, 58 insertions(+)
diff --git a/drivers/firmware/xilinx/zynqmp.c b/drivers/firmware/xilinx/zynqmp.c
index a3cadbaf3cba..e772404235c1 100644
--- a/drivers/firmware/xilinx/zynqmp.c
+++ b/drivers/firmware/xilinx/zynqmp.c
@@ -990,6 +990,41 @@ int zynqmp_pm_release_node(const u32 node)
}
EXPORT_SYMBOL_GPL(zynqmp_pm_release_node);
+/**
+ * zynqmp_pm_force_pwrdwn - PM call to request for another PU or subsystem to
+ * be powered down forcefully
+ * @node: Node ID of the targeted PU or subsystem
+ * @ack: Flag to specify whether acknowledge is requested
+ *
+ * Return: status, either success or error+reason
+ */
+int zynqmp_pm_force_pwrdwn(const u32 node,
+ const enum zynqmp_pm_request_ack ack)
+{
+ return zynqmp_pm_invoke_fn(PM_FORCE_POWERDOWN, node, ack, 0, 0, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_force_pwrdwn);
+
+/**
+ * zynqmp_pm_request_wake - PM call to wake up selected master or subsystem
+ * @node: Node ID of the master or subsystem
+ * @set_addr: Specifies whether the address argument is relevant
+ * @address: Address from which to resume when woken up
+ * @ack: Flag to specify whether acknowledge requested
+ *
+ * Return: status, either success or error+reason
+ */
+int zynqmp_pm_request_wake(const u32 node,
+ const bool set_addr,
+ const u64 address,
+ const enum zynqmp_pm_request_ack ack)
+{
+ /* set_addr flag is encoded into 1st bit of address */
+ return zynqmp_pm_invoke_fn(PM_REQUEST_WAKEUP, node, address | set_addr,
+ address >> 32, ack, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_request_wake);
+
/**
* zynqmp_pm_set_requirement() - PM call to set requirement for PM slaves
* @node: Node ID of the slave
diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index 92bf3ae20524..5da3d490e80e 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -12,6 +12,7 @@
#ifndef __FIRMWARE_ZYNQMP_H__
#define __FIRMWARE_ZYNQMP_H__
+#include <linux/types.h>
#include <linux/err.h>
@@ -66,6 +67,8 @@
enum pm_api_id {
PM_GET_API_VERSION = 1,
+ PM_FORCE_POWERDOWN = 8,
+ PM_REQUEST_WAKEUP = 10,
PM_SYSTEM_SHUTDOWN = 12,
PM_REQUEST_NODE = 13,
PM_RELEASE_NODE = 14,
@@ -435,6 +438,12 @@ int zynqmp_pm_pinctrl_get_config(const u32 pin, const u32 param,
int zynqmp_pm_pinctrl_set_config(const u32 pin, const u32 param,
u32 value);
int zynqmp_pm_load_pdi(const u32 src, const u64 address);
+int zynqmp_pm_force_pwrdwn(const u32 target,
+ const enum zynqmp_pm_request_ack ack);
+int zynqmp_pm_request_wake(const u32 node,
+ const bool set_addr,
+ const u64 address,
+ const enum zynqmp_pm_request_ack ack);
#else
static inline int zynqmp_pm_get_api_version(u32 *version)
{
@@ -620,6 +629,12 @@ static inline int zynqmp_pm_pinctrl_request(const u32 pin)
return -ENODEV;
}
+static inline int zynqmp_pm_force_pwrdwn(const u32 target,
+ const enum zynqmp_pm_request_ack ack)
+{
+ return -ENODEV;
+}
+
static inline int zynqmp_pm_pinctrl_release(const u32 pin)
{
return -ENODEV;
@@ -651,6 +666,14 @@ static inline int zynqmp_pm_load_pdi(const u32 src, const u64 address)
{
return -ENODEV;
}
+
+static inline int zynqmp_pm_request_wake(const u32 node,
+ const bool set_addr,
+ const u64 address,
+ const enum zynqmp_pm_request_ack ack)
+{
+ return -ENODEV;
+}
#endif
#endif /* __FIRMWARE_ZYNQMP_H__ */
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 5/6] firmware: xilinx: Add RPU configuration APIs
2021-11-23 6:20 ` Tanmay Shah
@ 2021-11-23 6:20 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
From: Ben Levinsky <ben.levinsky@xilinx.com>
This patch adds APIs to access to configure RPU and its
processor-specific memory.
That is query the run-time mode of RPU as either split or lockstep as well
as API to set this mode. In addition add APIs to access configuration of
the RPUs' tightly coupled memory (TCM).
Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
---
drivers/firmware/xilinx/zynqmp.c | 61 ++++++++++++++++++++++++++++
include/linux/firmware/xlnx-zynqmp.h | 18 ++++++++
2 files changed, 79 insertions(+)
diff --git a/drivers/firmware/xilinx/zynqmp.c b/drivers/firmware/xilinx/zynqmp.c
index e772404235c1..908d5b0d6947 100644
--- a/drivers/firmware/xilinx/zynqmp.c
+++ b/drivers/firmware/xilinx/zynqmp.c
@@ -990,6 +990,67 @@ int zynqmp_pm_release_node(const u32 node)
}
EXPORT_SYMBOL_GPL(zynqmp_pm_release_node);
+/**
+ * zynqmp_pm_get_rpu_mode() - Get RPU mode
+ * @node_id: Node ID of the device
+ * @rpu_mode: return by reference value
+ * either split or lockstep
+ *
+ * Return: return 0 on success or error+reason.
+ * if success, then rpu_mode will be set
+ * to current rpu mode.
+ */
+int zynqmp_pm_get_rpu_mode(u32 node_id, enum rpu_oper_mode *rpu_mode)
+{
+ u32 ret_payload[PAYLOAD_ARG_CNT];
+ int ret;
+
+ ret = zynqmp_pm_invoke_fn(PM_IOCTL, node_id,
+ IOCTL_GET_RPU_OPER_MODE, 0, 0, ret_payload);
+
+ /* only set rpu_mode if no error */
+ if (ret == XST_PM_SUCCESS)
+ *rpu_mode = ret_payload[0];
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_get_rpu_mode);
+
+/**
+ * zynqmp_pm_set_rpu_mode() - Set RPU mode
+ * @node_id: Node ID of the device
+ * @rpu_mode: Argument 1 to requested IOCTL call. either split or lockstep
+ *
+ * This function is used to set RPU mode to split or
+ * lockstep
+ *
+ * Return: Returns status, either success or error+reason
+ */
+int zynqmp_pm_set_rpu_mode(u32 node_id, enum rpu_oper_mode rpu_mode)
+{
+ return zynqmp_pm_invoke_fn(PM_IOCTL, node_id,
+ IOCTL_SET_RPU_OPER_MODE, (u32)rpu_mode,
+ 0, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_set_rpu_mode);
+
+/**
+ * zynqmp_pm_set_tcm_config - configure TCM
+ * @tcm_mode: Argument 1 to requested IOCTL call
+ * either PM_RPU_TCM_COMB or PM_RPU_TCM_SPLIT
+ *
+ * This function is used to set RPU mode to split or combined
+ *
+ * Return: status: 0 for success, else failure
+ */
+int zynqmp_pm_set_tcm_config(u32 node_id, enum rpu_tcm_comb tcm_mode)
+{
+ return zynqmp_pm_invoke_fn(PM_IOCTL, node_id,
+ IOCTL_TCM_COMB_CONFIG, (u32)tcm_mode, 0,
+ NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_set_tcm_config);
+
/**
* zynqmp_pm_force_pwrdwn - PM call to request for another PU or subsystem to
* be powered down forcefully
diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index 5da3d490e80e..eea9bb5990ea 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -444,6 +444,9 @@ int zynqmp_pm_request_wake(const u32 node,
const bool set_addr,
const u64 address,
const enum zynqmp_pm_request_ack ack);
+int zynqmp_pm_get_rpu_mode(u32 node_id, enum rpu_oper_mode *rpu_mode);
+int zynqmp_pm_set_rpu_mode(u32 node_id, u32 arg1);
+int zynqmp_pm_set_tcm_config(u32 node_id, u32 arg1);
#else
static inline int zynqmp_pm_get_api_version(u32 *version)
{
@@ -674,6 +677,21 @@ static inline int zynqmp_pm_request_wake(const u32 node,
{
return -ENODEV;
}
+
+static inline int zynqmp_pm_get_rpu_mode(u32 node_id, enum rpu_oper_mode *rpu_mode)
+{
+ return -ENODEV;
+}
+
+static inline int zynqmp_pm_set_rpu_mode(u32 node_id, u32 arg1)
+{
+ return -ENODEV;
+}
+
+static inline int zynqmp_pm_set_tcm_config(u32 node_id, u32 arg1)
+{
+ return -ENODEV;
+}
#endif
#endif /* __FIRMWARE_ZYNQMP_H__ */
--
2.25.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 5/6] firmware: xilinx: Add RPU configuration APIs
@ 2021-11-23 6:20 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
From: Ben Levinsky <ben.levinsky@xilinx.com>
This patch adds APIs to access to configure RPU and its
processor-specific memory.
That is query the run-time mode of RPU as either split or lockstep as well
as API to set this mode. In addition add APIs to access configuration of
the RPUs' tightly coupled memory (TCM).
Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
---
drivers/firmware/xilinx/zynqmp.c | 61 ++++++++++++++++++++++++++++
include/linux/firmware/xlnx-zynqmp.h | 18 ++++++++
2 files changed, 79 insertions(+)
diff --git a/drivers/firmware/xilinx/zynqmp.c b/drivers/firmware/xilinx/zynqmp.c
index e772404235c1..908d5b0d6947 100644
--- a/drivers/firmware/xilinx/zynqmp.c
+++ b/drivers/firmware/xilinx/zynqmp.c
@@ -990,6 +990,67 @@ int zynqmp_pm_release_node(const u32 node)
}
EXPORT_SYMBOL_GPL(zynqmp_pm_release_node);
+/**
+ * zynqmp_pm_get_rpu_mode() - Get RPU mode
+ * @node_id: Node ID of the device
+ * @rpu_mode: return by reference value
+ * either split or lockstep
+ *
+ * Return: return 0 on success or error+reason.
+ * if success, then rpu_mode will be set
+ * to current rpu mode.
+ */
+int zynqmp_pm_get_rpu_mode(u32 node_id, enum rpu_oper_mode *rpu_mode)
+{
+ u32 ret_payload[PAYLOAD_ARG_CNT];
+ int ret;
+
+ ret = zynqmp_pm_invoke_fn(PM_IOCTL, node_id,
+ IOCTL_GET_RPU_OPER_MODE, 0, 0, ret_payload);
+
+ /* only set rpu_mode if no error */
+ if (ret == XST_PM_SUCCESS)
+ *rpu_mode = ret_payload[0];
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_get_rpu_mode);
+
+/**
+ * zynqmp_pm_set_rpu_mode() - Set RPU mode
+ * @node_id: Node ID of the device
+ * @rpu_mode: Argument 1 to requested IOCTL call. either split or lockstep
+ *
+ * This function is used to set RPU mode to split or
+ * lockstep
+ *
+ * Return: Returns status, either success or error+reason
+ */
+int zynqmp_pm_set_rpu_mode(u32 node_id, enum rpu_oper_mode rpu_mode)
+{
+ return zynqmp_pm_invoke_fn(PM_IOCTL, node_id,
+ IOCTL_SET_RPU_OPER_MODE, (u32)rpu_mode,
+ 0, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_set_rpu_mode);
+
+/**
+ * zynqmp_pm_set_tcm_config - configure TCM
+ * @tcm_mode: Argument 1 to requested IOCTL call
+ * either PM_RPU_TCM_COMB or PM_RPU_TCM_SPLIT
+ *
+ * This function is used to set RPU mode to split or combined
+ *
+ * Return: status: 0 for success, else failure
+ */
+int zynqmp_pm_set_tcm_config(u32 node_id, enum rpu_tcm_comb tcm_mode)
+{
+ return zynqmp_pm_invoke_fn(PM_IOCTL, node_id,
+ IOCTL_TCM_COMB_CONFIG, (u32)tcm_mode, 0,
+ NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_set_tcm_config);
+
/**
* zynqmp_pm_force_pwrdwn - PM call to request for another PU or subsystem to
* be powered down forcefully
diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index 5da3d490e80e..eea9bb5990ea 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -444,6 +444,9 @@ int zynqmp_pm_request_wake(const u32 node,
const bool set_addr,
const u64 address,
const enum zynqmp_pm_request_ack ack);
+int zynqmp_pm_get_rpu_mode(u32 node_id, enum rpu_oper_mode *rpu_mode);
+int zynqmp_pm_set_rpu_mode(u32 node_id, u32 arg1);
+int zynqmp_pm_set_tcm_config(u32 node_id, u32 arg1);
#else
static inline int zynqmp_pm_get_api_version(u32 *version)
{
@@ -674,6 +677,21 @@ static inline int zynqmp_pm_request_wake(const u32 node,
{
return -ENODEV;
}
+
+static inline int zynqmp_pm_get_rpu_mode(u32 node_id, enum rpu_oper_mode *rpu_mode)
+{
+ return -ENODEV;
+}
+
+static inline int zynqmp_pm_set_rpu_mode(u32 node_id, u32 arg1)
+{
+ return -ENODEV;
+}
+
+static inline int zynqmp_pm_set_tcm_config(u32 node_id, u32 arg1)
+{
+ return -ENODEV;
+}
#endif
#endif /* __FIRMWARE_ZYNQMP_H__ */
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-11-23 6:20 ` Tanmay Shah
@ 2021-11-23 6:20 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
This driver enables r5f dual core Real time Processing Unit subsystem
available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
(cluster) can be configured in different modes e.g. split mode in which
two r5f cores work independent of each other and lock-step mode in which
both r5f cores execute same code clock-for-clock and notify if the
result is different.
The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
Platform Management Unit that handles the R5 configuration, memory access
and R5 lifecycle management. The interface to this manager is done in this
driver via zynqmp_pm_* function calls.
Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
---
drivers/remoteproc/Kconfig | 12 +
drivers/remoteproc/Makefile | 1 +
drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
3 files changed, 972 insertions(+)
create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
index f30d00a3aabe..27f66910d8d3 100644
--- a/drivers/remoteproc/Kconfig
+++ b/drivers/remoteproc/Kconfig
@@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
It's safe to say N here if you're not interested in utilizing
a slave processor.
+config XLNX_R5_REMOTEPROC
+ tristate "Xilinx R5 remoteproc support"
+ depends on PM && ARCH_ZYNQMP
+ depends on ZYNQMP_FIRMWARE
+ select RPMSG_VIRTIO
+ select ZYNQMP_IPI_MBOX
+ help
+ Say y or m here to support Xilinx R5 remote processors via the remote
+ processor framework.
+
+ It's safe to say N if not interested in using RPU r5f cores.
+
endif # REMOTEPROC
endmenu
diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
index bb26c9e4ef9c..334a8bed4c14 100644
--- a/drivers/remoteproc/Makefile
+++ b/drivers/remoteproc/Makefile
@@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
+obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
new file mode 100644
index 000000000000..c2167fd3869d
--- /dev/null
+++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
@@ -0,0 +1,959 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ZynqMP R5 Remote Processor driver
+ *
+ */
+
+#include <dt-bindings/power/xlnx-zynqmp-power.h>
+#include <linux/firmware/xlnx-zynqmp.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/mailbox_client.h>
+#include <linux/mailbox/zynqmp-ipi-message.h>
+#include <linux/module.h>
+#include <linux/of_address.h>
+#include <linux/of_platform.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/remoteproc.h>
+#include <linux/skbuff.h>
+#include <linux/sysfs.h>
+
+#include "remoteproc_internal.h"
+
+/* settings for RPU cluster mode */
+enum zynqmp_r5_cluster_mode {
+ SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
+ LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
+ SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
+};
+
+/**
+ * struct mem_bank_data - Memory Bank description
+ *
+ * @addr: Start address of memory bank
+ * @size: Size of Memory bank
+ * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
+ * @bank_name: name of the bank for remoteproc framework
+ */
+struct mem_bank_data {
+ phys_addr_t addr;
+ size_t size;
+ enum pm_node_id pm_domain_id;
+ char *bank_name;
+};
+
+static const struct mem_bank_data zynqmp_tcm_banks[] = {
+ {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
+ {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
+ {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
+ {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
+};
+
+/**
+ * struct zynqmp_r5_core - ZynqMP R5 core structure
+ *
+ * @dev: device of RPU instance
+ * @np: device node of RPU instance
+ * @tcm_bank_count: number TCM banks accessible to this RPU
+ * @tcm_banks: array of each TCM bank data
+ * @res_mem_count: number of Reserved Memory regions per core
+ * @res_mem: array of reserved memory regions
+ * @rproc: rproc handle
+ * @pm_domain_id: RPU CPU power domain id
+ */
+struct zynqmp_r5_core {
+ struct device *dev;
+ struct device_node *np;
+ int tcm_bank_count;
+ struct mem_bank_data *tcm_banks;
+ int res_mem_count;
+ struct reserved_mem *res_mem;
+ struct rproc *rproc;
+ enum pm_node_id pm_domain_id;
+};
+
+/**
+ * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
+ *
+ * @dev: r5f subsystem cluster device node
+ * @mode: cluster mode of type zynqmp_r5_cluster_mode
+ * @core_count: number of r5 cores used for this cluster mode
+ * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
+ */
+struct zynqmp_r5_cluster {
+ struct device *dev;
+ enum zynqmp_r5_cluster_mode mode;
+ int core_count;
+ struct zynqmp_r5_core *r5_cores;
+};
+
+/*
+ * zynqmp_r5_set_mode - set RPU operation mode
+ *
+ * set RPU operation mode
+ *
+ * Return: 0 for success, negative value for failure
+ */
+static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
+ enum zynqmp_r5_cluster_mode rpu_mode)
+{
+ enum rpu_tcm_comb tcm_mode;
+ int ret, reg_val;
+
+ reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
+
+ ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
+ if (ret < 0) {
+ pr_err("failed to set RPU mode\n");
+ return ret;
+ }
+
+ tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
+ PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
+ ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
+ if (ret < 0)
+ pr_err("failed to configure TCM\n");
+
+ return ret;
+}
+
+/*
+ * zynqmp_r5_rproc_start
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Start R5 Core from designated boot address.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_start(struct rproc *rproc)
+{
+ struct zynqmp_r5_core *r5_core = rproc->priv;
+ enum rpu_boot_mem bootmem;
+ int ret;
+
+ if (!r5_core) {
+ pr_err("can't get r5 core\n");
+ return -EINVAL;
+ }
+
+ bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
+ PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
+
+ dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
+ bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
+
+ ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
+ bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
+ if (ret)
+ pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
+ return ret;
+}
+
+/*
+ * zynqmp_r5_rproc_stop
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Power down R5 Core.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_stop(struct rproc *rproc)
+{
+ struct zynqmp_r5_core *r5_core = rproc->priv;
+ int ret;
+
+ ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
+ ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+ if (ret)
+ pr_err("failed to stop remoteproc RPU %d\n", ret);
+
+ return ret;
+}
+
+/*
+ * zynqmp_r5_rproc_mem_map
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to map
+ *
+ * Callback to map va for memory-region's carveout.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
+ struct rproc_mem_entry *mem)
+{
+ void __iomem *va;
+
+ va = ioremap_wc(mem->dma, mem->len);
+ if (IS_ERR_OR_NULL(va))
+ return -ENOMEM;
+
+ mem->va = (void *)va;
+
+ return 0;
+}
+
+/*
+ * zynqmp_r5_rproc_mem_unmap
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to unmap
+ *
+ * Unmap memory-region carveout
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
+ struct rproc_mem_entry *mem)
+{
+ iounmap((void __iomem *)mem->va);
+ return 0;
+}
+
+/*
+ * add_mem_regions
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Construct rproc mem carveouts from carveout provided in
+ * memory-region property
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int add_mem_regions(struct rproc *rproc)
+{
+ struct device *dev;
+ struct rproc_mem_entry *mem;
+ struct reserved_mem *rmem;
+ struct zynqmp_r5_core *r5_core;
+ int i;
+
+ r5_core = rproc->priv;
+ dev = r5_core->dev;
+
+ /* Register associated reserved memory regions */
+ for (i = 0; i < r5_core->res_mem_count; i++) {
+ rmem = &r5_core->res_mem[i];
+ mem = rproc_mem_entry_init(dev, NULL,
+ (dma_addr_t)rmem->base,
+ rmem->size, rmem->base,
+ zynqmp_r5_rproc_mem_map,
+ zynqmp_r5_rproc_mem_unmap,
+ rmem->name);
+ if (IS_ERR_OR_NULL(mem))
+ return -ENOMEM;
+
+ rproc_add_carveout(rproc, mem);
+ }
+
+ return 0;
+}
+
+/*
+ * zynqmp_r5_rproc_mem_unmap
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to unmap
+ *
+ * Unmap TCM banks when powering down R5 core.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
+{
+ struct zynqmp_r5_core *r5_core;
+ int i;
+ enum pm_node_id pm_domain_id;
+
+ r5_core = rproc->priv;
+ if (!r5_core) {
+ pr_err("r5 core is not available\n");
+ return -EINVAL;
+ }
+
+ iounmap((void __iomem *)mem->va);
+
+ for (i = 0; i < r5_core->tcm_bank_count; i++) {
+ pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
+ if (zynqmp_pm_release_node(pm_domain_id))
+ pr_warn("can't turn off TCM bank %d", pm_domain_id);
+ }
+
+ return 0;
+}
+
+/*
+ * tcm_mem_map
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to initialize the va and da fields of
+ *
+ * Given TCM bank entry, this callback will set device address for R5
+ * running on TCM and also setup virtual address for TCM bank
+ * remoteproc carveout.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int tcm_mem_map(struct rproc *rproc,
+ struct rproc_mem_entry *mem)
+{
+ void __iomem *va;
+
+ va = ioremap_wc(mem->dma, mem->len);
+ if (IS_ERR_OR_NULL(va))
+ return -ENOMEM;
+
+ /* Update memory entry va */
+ mem->va = (void *)va;
+
+ /* clear TCMs */
+ memset_io(va, 0, mem->len);
+
+ /*
+ * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
+ * while on the Linux side they are at 0xffexxxxx.
+ *
+ * Zero out the high 12 bits of the address. This will give
+ * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
+ */
+ mem->da &= 0x000fffff;
+
+ /*
+ * TCM Banks 1A and 1B still have to be translated.
+ *
+ * Below handle these two banks' absolute addresses (0xffe90000 and
+ * 0xffeb0000) and convert to the expected relative addresses
+ * (0x0 and 0x20000).
+ */
+ if (mem->da == 0x90000 || mem->da == 0xB0000)
+ mem->da -= 0x90000;
+
+ /* if translated TCM bank address is not valid report error */
+ if (mem->da != 0x0 && mem->da != 0x20000) {
+ dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
+ return -EINVAL;
+ }
+ return 0;
+}
+
+static int add_tcm_carveout_split_mode(struct rproc *rproc)
+{
+ int i, num_banks, ret;
+ struct rproc_mem_entry *mem;
+ enum pm_node_id pm_domain_id;
+ u32 bank_addr;
+ size_t bank_size = 0;
+ char *bank_name;
+ struct device *dev;
+ struct zynqmp_r5_core *r5_core;
+
+ r5_core = (struct zynqmp_r5_core *)rproc->priv;
+ if (!r5_core)
+ return -EINVAL;
+
+ dev = r5_core->dev;
+
+ /* go through zynqmp banks for r5 node */
+ num_banks = r5_core->tcm_bank_count;
+ if (num_banks <= 0) {
+ dev_err(dev, "need to specify TCM banks\n");
+ return -EINVAL;
+ }
+
+ for (i = 0; i < num_banks; i++) {
+ bank_addr = (u32)r5_core->tcm_banks[i].addr;
+ bank_name = r5_core->tcm_banks[i].bank_name;
+ bank_size = r5_core->tcm_banks[i].size;
+ pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
+
+ ret = zynqmp_pm_request_node(pm_domain_id,
+ ZYNQMP_PM_CAPABILITY_ACCESS, 0,
+ ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+ if (ret < 0) {
+ dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
+ return ret;
+ }
+
+ dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
+ bank_name, bank_addr, bank_size);
+
+ /* add carveout */
+ mem = rproc_mem_entry_init(dev, NULL, bank_addr,
+ bank_size, bank_addr,
+ tcm_mem_map, tcm_mem_unmap,
+ bank_name);
+ if (IS_ERR_OR_NULL(mem)) {
+ /* Turn off all TCM banks turned on before */
+ do {
+ pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
+ ret = zynqmp_pm_release_node((u32)pm_domain_id);
+ if (ret)
+ dev_warn(dev,
+ "fail to release node: %x, %x\n",
+ (u32)pm_domain_id, ret);
+ } while (i--);
+ return -ENOMEM;
+ }
+
+ rproc_add_carveout(rproc, mem);
+ }
+
+ return 0;
+}
+
+static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
+{
+ int i, num_banks, ret;
+ struct rproc_mem_entry *mem;
+ enum pm_node_id pm_domain_id;
+ u32 bank_addr;
+ size_t bank_size = 0;
+ char *bank_name;
+ struct device *dev;
+ struct platform_device *parent_pdev;
+ struct zynqmp_r5_cluster *cluster;
+ struct zynqmp_r5_core *r5_core;
+
+ r5_core = (struct zynqmp_r5_core *)rproc->priv;
+ if (!r5_core)
+ return -EINVAL;
+
+ dev = r5_core->dev;
+ if (!dev) {
+ pr_err("r5 core device unavailable\n");
+ return -ENODEV;
+ }
+
+ /* go through zynqmp banks for r5 node */
+ num_banks = r5_core->tcm_bank_count;
+ if (num_banks <= 0) {
+ dev_err(dev, "need to specify TCM banks\n");
+ return -EINVAL;
+ }
+
+ bank_addr = (u32)r5_core->tcm_banks[0].addr;
+ bank_name = r5_core->tcm_banks[0].bank_name;
+ for (i = 0; i < num_banks; i++) {
+ bank_size += r5_core->tcm_banks[i].size;
+ pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
+
+ ret = zynqmp_pm_request_node(pm_domain_id,
+ ZYNQMP_PM_CAPABILITY_ACCESS, 0,
+ ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+ if (ret < 0) {
+ dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
+ return ret;
+ }
+ }
+
+ dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
+ bank_name, bank_addr, bank_size);
+
+ /* add carveout */
+ mem = rproc_mem_entry_init(dev, NULL, bank_addr,
+ bank_size, bank_addr,
+ tcm_mem_map, tcm_mem_unmap,
+ bank_name);
+ if (IS_ERR_OR_NULL(mem)) {
+ for (i = 0; i < num_banks; i++) {
+ pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
+ ret = zynqmp_pm_release_node((u32)pm_domain_id);
+ if (ret)
+ dev_warn(dev,
+ "fail to release node: %x ret: %x\n",
+ (u32)pm_domain_id, ret);
+ }
+ return -ENOMEM;
+ }
+
+ rproc_add_carveout(rproc, mem);
+
+ return 0;
+}
+
+/*
+ * add_tcm_banks()
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Given R5 node in remoteproc instance
+ * allocate remoteproc carveout for TCM memory
+ * needed for firmware to be loaded
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int add_tcm_banks(struct rproc *rproc)
+{
+ struct device *dev;
+ struct platform_device *parent_pdev;
+ struct zynqmp_r5_cluster *cluster;
+ struct zynqmp_r5_core *r5_core;
+
+ r5_core = (struct zynqmp_r5_core *)rproc->priv;
+ if (!r5_core)
+ return -EINVAL;
+
+ dev = r5_core->dev;
+ if (!dev) {
+ pr_err("r5 core device unavailable\n");
+ return -ENODEV;
+ }
+
+ parent_pdev = to_platform_device(dev->parent);
+ if (!parent_pdev) {
+ dev_err(dev, "parent platform dev unavailable\n");
+ return -ENODEV;
+ }
+
+ cluster = platform_get_drvdata(parent_pdev);
+ if (!cluster) {
+ dev_err(&parent_pdev->dev, "Invalid driver data\n");
+ return -EINVAL;
+ }
+
+ if (cluster->mode == SPLIT_MODE)
+ return add_tcm_carveout_split_mode(rproc);
+ else if (cluster->mode == LOCKSTEP_MODE)
+ return add_tcm_carveout_lockstep_mode(rproc);
+
+ dev_err(cluster->dev, "invalid cluster mode\n");
+ return -EINVAL;
+}
+
+/*
+ * zynqmp_r5_parse_fw()
+ * @rproc: single R5 core's corresponding rproc instance
+ * @fw: ptr to firmware to be loaded onto r5 core
+ *
+ * When loading firmware, ensure the necessary carveouts are in remoteproc
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
+{
+ int ret;
+ struct zynqmp_r5_core *r5_core;
+ struct device *dev;
+
+ r5_core = rproc->priv;
+ if (!r5_core) {
+ dev_err(&rproc->dev, "r5 core not available\n");
+ return -EINVAL;
+ }
+
+ dev = r5_core->dev;
+
+ ret = add_tcm_banks(rproc);
+ if (ret) {
+ dev_err(dev, "failed to get TCM banks, err %d\n", ret);
+ return ret;
+ }
+
+ ret = add_mem_regions(rproc);
+ if (ret)
+ dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
+
+ ret = rproc_elf_load_rsc_table(rproc, fw);
+ if (ret == -EINVAL) {
+ /*
+ * resource table only required for IPC.
+ * if not present, this is not necessarily an error;
+ * for example, loading r5 hello world application
+ * so simply inform user and keep going.
+ */
+ dev_info(&rproc->dev, "no resource table found.\n");
+ ret = 0;
+ }
+ return ret;
+}
+
+static struct rproc_ops zynqmp_r5_rproc_ops = {
+ .start = zynqmp_r5_rproc_start,
+ .stop = zynqmp_r5_rproc_stop,
+ .load = rproc_elf_load_segments,
+ .parse_fw = zynqmp_r5_parse_fw,
+ .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
+ .sanity_check = rproc_elf_sanity_check,
+ .get_boot_addr = rproc_elf_get_boot_addr,
+};
+
+static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
+{
+ int i, j, k;
+ struct zynqmp_r5_core *r5_core;
+
+ dev_dbg(cluster->dev, "Printing dt node info\n");
+
+ pr_debug("cluster mode = %d\n", cluster->mode);
+ pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
+ cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
+ pr_debug("r5f num cores = %d\n", cluster->core_count);
+
+ for (i = 0; i < cluster->core_count; i++) {
+ r5_core = &cluster->r5_cores[i];
+ if (!r5_core) {
+ pr_err("can't get r5_core\n");
+ continue;
+ }
+
+ pr_debug("r5 core %d nodes\n", i);
+ pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
+ for (k = 0; k < r5_core->tcm_bank_count; k++) {
+ pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
+ k, r5_core->tcm_banks[k].addr,
+ r5_core->tcm_banks[k].size,
+ r5_core->tcm_banks[k].pm_domain_id,
+ r5_core->tcm_banks[k].bank_name);
+ }
+
+ pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
+
+ for (j = 0; j < r5_core->res_mem_count; j++) {
+ pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
+ j, r5_core->res_mem[j].base,
+ r5_core->res_mem[j].size,
+ r5_core->res_mem[j].name);
+ }
+ }
+}
+
+/**
+ * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
+ * this is called for each individual R5 core to
+ * set up mailbox, Xilinx platform manager unique ID,
+ * add to rproc core
+ *
+ * @r5_core: zynqmp_r5_core r5 core object to initialize
+ *
+ * Return: 0 for success, negative value for failure.
+ */
+static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
+{
+ int ret;
+ struct rproc *r5_rproc;
+ struct device *dev;
+
+ dev = r5_core->dev;
+
+ /* Set up DMA mask */
+ ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
+ if (ret)
+ return ret;
+
+ /* Allocate remoteproc instance */
+ r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
+ NULL, sizeof(struct zynqmp_r5_core));
+ if (IS_ERR_OR_NULL(r5_rproc))
+ return -ENOMEM;
+
+ r5_rproc->auto_boot = false;
+ r5_rproc->priv = r5_core;
+
+ /* Add R5 remoteproc */
+ ret = devm_rproc_add(dev, r5_rproc);
+ if (ret) {
+ pr_err("failed to add r5 remoteproc\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
+{
+ int tcm_bank_count, tcm_node;
+ int i = 0, j;
+ struct zynqmp_r5_core *r5_core;
+ const struct mem_bank_data *tcm = zynqmp_tcm_banks;
+ struct device *dev = cluster->dev;
+
+ /* ToDo: Use predefined TCM address space values from driver until
+ * system-dt spec is not final fot TCM
+ */
+ tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
+
+ /* count per core tcm banks */
+ tcm_bank_count = tcm_bank_count / cluster->core_count;
+
+ /* r5 core 0 will use all of TCM banks in lockstep mode.
+ * In split mode, r5 core0 will use 128k and r5 core1 will use another
+ * 128k. Assign TCM banks to each core accordingly
+ */
+ tcm_node = 0;
+ for (j = 0; j < cluster->core_count; j++) {
+ r5_core = &cluster->r5_cores[j];
+ r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
+ tcm_bank_count, GFP_KERNEL);
+ if (IS_ERR_OR_NULL(r5_core->tcm_banks))
+ return -ENOMEM;
+
+ for (i = 0; i < tcm_bank_count; i++) {
+ /* Use pre-defined TCM reg values.
+ * Eventually this should be replaced by values
+ * parsed from dts.
+ */
+ r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
+ r5_core->tcm_banks[i].size = tcm[tcm_node].size;
+ r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
+ r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
+ tcm_node++;
+ }
+
+ r5_core->tcm_bank_count = tcm_bank_count;
+ }
+
+ return 0;
+}
+
+static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
+{
+ int res_mem_count, i;
+ struct device *dev;
+ struct device_node *np, *rmem_np;
+ struct reserved_mem *rmem;
+
+ dev = r5_core->dev;
+
+ np = r5_core->np;
+ if (IS_ERR_OR_NULL(np)) {
+ pr_err("invalid device node of r5 core\n");
+ return -EINVAL;
+ }
+
+ res_mem_count = of_property_count_elems_of_size(np, "memory-region",
+ sizeof(phandle));
+ if (res_mem_count <= 0) {
+ dev_warn(dev, "failed to get memory-region property %d\n",
+ res_mem_count);
+ return -EINVAL;
+ }
+
+ r5_core->res_mem = devm_kzalloc(dev,
+ res_mem_count * sizeof(struct reserved_mem),
+ GFP_KERNEL);
+ if (!r5_core->res_mem) {
+ dev_err(dev, "failed to allocate mem region memory\n");
+ return -ENOMEM;
+ }
+
+ for (i = 0; i < res_mem_count; i++) {
+ rmem_np = of_parse_phandle(np, "memory-region", i);
+ if (!rmem_np)
+ return -EINVAL;
+
+ rmem = of_reserved_mem_lookup(rmem_np);
+ if (!rmem) {
+ of_node_put(rmem_np);
+ return -EINVAL;
+ }
+
+ memcpy(&r5_core->res_mem[i], rmem,
+ sizeof(struct reserved_mem));
+ of_node_put(rmem_np);
+ }
+
+ r5_core->res_mem_count = res_mem_count;
+
+ return 0;
+}
+
+static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
+{
+ int ret, i;
+ struct zynqmp_r5_core *r5_core;
+ struct device *dev = cluster->dev;
+
+ ret = zynqmp_r5_get_tcm_node(cluster);
+ if (ret < 0) {
+ dev_err(dev, "can't get tcm node, err %d\n", ret);
+ return ret;
+ }
+
+ for (i = 0; i < cluster->core_count; i++) {
+ r5_core = &cluster->r5_cores[i];
+ if (!r5_core) {
+ pr_err("invalid r5 core\n");
+ return -EINVAL;
+ }
+
+ ret = zynqmp_r5_get_mem_region_node(r5_core);
+ if (ret)
+ dev_warn(dev, "memory-region prop failed %d\n", ret);
+
+ ret = of_property_read_u32_index(r5_core->np, "power-domains",
+ 1, &r5_core->pm_domain_id);
+ if (ret) {
+ dev_err(dev, "failed to get power-domains property\n");
+ return ret;
+ }
+
+ ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
+ if (ret)
+ return ret;
+
+ ret = zynqmp_r5_add_rproc_core(r5_core);
+ if (ret) {
+ dev_err(dev, "failed to init r5 core %d\n", i);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
+{
+ struct device *dev = cluster->dev;
+ struct device_node *dev_node = dev_of_node(dev);
+ struct device_node *child;
+ struct platform_device *child_pdev;
+ int core_count = 0, ret, i;
+ enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
+ struct zynqmp_r5_core *r5_cores;
+
+ ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
+
+ /* on success returns 0, if not defined then returns -EINVAL,
+ * In that case, default is LOCKSTEP mode
+ */
+ if (ret != -EINVAL && ret != 0) {
+ dev_err(dev, "Invalid xlnx,cluster-mode property\n");
+ return -EINVAL;
+ }
+
+ if (cluster_mode == SINGLE_CPU_MODE) {
+ dev_err(dev, "driver does not support single cpu mode\n");
+ return -EINVAL;
+ } else if ((cluster_mode != SPLIT_MODE &&
+ cluster_mode != LOCKSTEP_MODE)) {
+ dev_err(dev, "Invalid cluster mode\n");
+ return -EINVAL;
+ }
+
+ core_count = of_get_available_child_count(dev_node);
+ if (core_count <= 0) {
+ dev_err(dev, "Invalid number of r5 cores %d", core_count);
+ return -EINVAL;
+ } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
+ dev_err(dev, "Invalid number of r5 cores for split mode\n");
+ return -EINVAL;
+ } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
+ dev_warn(dev, "Only r5 core0 will be used\n");
+ core_count = 1;
+ }
+
+ r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
+ core_count, GFP_KERNEL);
+ if (IS_ERR_OR_NULL(r5_cores)) {
+ dev_err(dev, "can't allocate memory for cores\n");
+ return -ENOMEM;
+ }
+
+ i = 0;
+ for_each_available_child_of_node(dev_node, child) {
+ child_pdev = of_find_device_by_node(child);
+ if (!child_pdev)
+ return -ENODEV;
+
+ r5_cores[i].dev = &child_pdev->dev;
+ if (!r5_cores[i].dev) {
+ pr_err("can't get device for r5 core %d\n", i);
+ return -ENODEV;
+ }
+
+ r5_cores[i].np = dev_of_node(r5_cores[i].dev);
+ if (!r5_cores[i].np) {
+ pr_err("can't get device node for r5 core %d\n", i);
+ return -ENODEV;
+ }
+
+ i++;
+ if (i == core_count)
+ break;
+ }
+
+ cluster->mode = cluster_mode;
+ cluster->core_count = core_count;
+ cluster->r5_cores = r5_cores;
+
+ ret = zynqmp_r5_core_init(cluster);
+ if (ret < 0) {
+ dev_err(dev, "failed to init r5 core err %d\n", ret);
+ return ret;
+ }
+
+ zynqmp_r5_print_dt_node_info(cluster);
+
+ return 0;
+}
+
+static void zynqmp_r5_cluster_exit(void *data)
+{
+ struct platform_device *pdev = (struct platform_device *)data;
+
+ platform_set_drvdata(pdev, NULL);
+
+ pr_info("Exit r5f subsystem driver\n");
+}
+
+/*
+ * zynqmp_r5_remoteproc_probe()
+ *
+ * @pdev: domain platform device for R5 cluster
+ *
+ * called when driver is probed, for each R5 core specified in DT,
+ * setup as needed to do remoteproc-related operations
+ *
+ * Return: 0 for success, negative value for failure.
+ */
+static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
+{
+ int ret;
+ struct zynqmp_r5_cluster *cluster;
+ struct device *dev = &pdev->dev;
+
+ cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
+ if (IS_ERR_OR_NULL(cluster))
+ return -ENOMEM;
+
+ cluster->dev = dev;
+
+ ret = devm_of_platform_populate(dev);
+ if (ret) {
+ dev_err(dev, "failed to populate platform dev %d\n", ret);
+ return ret;
+ }
+
+ /* wire in so each core can be cleaned up at driver remove */
+ platform_set_drvdata(pdev, cluster);
+
+ ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
+ if (ret)
+ return ret;
+
+ ret = zynqmp_r5_cluster_init(cluster);
+ if (ret) {
+ dev_err(dev, "Invalid r5f subsystem device tree\n");
+ return ret;
+ }
+
+ dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
+ return 0;
+}
+
+/* Match table for OF platform binding */
+static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
+ { .compatible = "xlnx,zynqmp-r5fss", },
+ { /* end of list */ },
+};
+MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
+
+static struct platform_driver zynqmp_r5_remoteproc_driver = {
+ .probe = zynqmp_r5_remoteproc_probe,
+ .driver = {
+ .name = "zynqmp_r5_remoteproc",
+ .of_match_table = zynqmp_r5_remoteproc_match,
+ },
+};
+module_platform_driver(zynqmp_r5_remoteproc_driver);
+
+MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
+MODULE_AUTHOR("Xilinx Inc.");
+MODULE_LICENSE("GPL v2");
--
2.25.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-11-23 6:20 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-23 6:20 UTC (permalink / raw)
To: Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
Tanmay Shah, linux-remoteproc, linux-kernel, devicetree,
linux-arm-kernel
This driver enables r5f dual core Real time Processing Unit subsystem
available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
(cluster) can be configured in different modes e.g. split mode in which
two r5f cores work independent of each other and lock-step mode in which
both r5f cores execute same code clock-for-clock and notify if the
result is different.
The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
Platform Management Unit that handles the R5 configuration, memory access
and R5 lifecycle management. The interface to this manager is done in this
driver via zynqmp_pm_* function calls.
Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
---
drivers/remoteproc/Kconfig | 12 +
drivers/remoteproc/Makefile | 1 +
drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
3 files changed, 972 insertions(+)
create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
index f30d00a3aabe..27f66910d8d3 100644
--- a/drivers/remoteproc/Kconfig
+++ b/drivers/remoteproc/Kconfig
@@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
It's safe to say N here if you're not interested in utilizing
a slave processor.
+config XLNX_R5_REMOTEPROC
+ tristate "Xilinx R5 remoteproc support"
+ depends on PM && ARCH_ZYNQMP
+ depends on ZYNQMP_FIRMWARE
+ select RPMSG_VIRTIO
+ select ZYNQMP_IPI_MBOX
+ help
+ Say y or m here to support Xilinx R5 remote processors via the remote
+ processor framework.
+
+ It's safe to say N if not interested in using RPU r5f cores.
+
endif # REMOTEPROC
endmenu
diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
index bb26c9e4ef9c..334a8bed4c14 100644
--- a/drivers/remoteproc/Makefile
+++ b/drivers/remoteproc/Makefile
@@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
+obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
new file mode 100644
index 000000000000..c2167fd3869d
--- /dev/null
+++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
@@ -0,0 +1,959 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ZynqMP R5 Remote Processor driver
+ *
+ */
+
+#include <dt-bindings/power/xlnx-zynqmp-power.h>
+#include <linux/firmware/xlnx-zynqmp.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/mailbox_client.h>
+#include <linux/mailbox/zynqmp-ipi-message.h>
+#include <linux/module.h>
+#include <linux/of_address.h>
+#include <linux/of_platform.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/remoteproc.h>
+#include <linux/skbuff.h>
+#include <linux/sysfs.h>
+
+#include "remoteproc_internal.h"
+
+/* settings for RPU cluster mode */
+enum zynqmp_r5_cluster_mode {
+ SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
+ LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
+ SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
+};
+
+/**
+ * struct mem_bank_data - Memory Bank description
+ *
+ * @addr: Start address of memory bank
+ * @size: Size of Memory bank
+ * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
+ * @bank_name: name of the bank for remoteproc framework
+ */
+struct mem_bank_data {
+ phys_addr_t addr;
+ size_t size;
+ enum pm_node_id pm_domain_id;
+ char *bank_name;
+};
+
+static const struct mem_bank_data zynqmp_tcm_banks[] = {
+ {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
+ {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
+ {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
+ {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
+};
+
+/**
+ * struct zynqmp_r5_core - ZynqMP R5 core structure
+ *
+ * @dev: device of RPU instance
+ * @np: device node of RPU instance
+ * @tcm_bank_count: number TCM banks accessible to this RPU
+ * @tcm_banks: array of each TCM bank data
+ * @res_mem_count: number of Reserved Memory regions per core
+ * @res_mem: array of reserved memory regions
+ * @rproc: rproc handle
+ * @pm_domain_id: RPU CPU power domain id
+ */
+struct zynqmp_r5_core {
+ struct device *dev;
+ struct device_node *np;
+ int tcm_bank_count;
+ struct mem_bank_data *tcm_banks;
+ int res_mem_count;
+ struct reserved_mem *res_mem;
+ struct rproc *rproc;
+ enum pm_node_id pm_domain_id;
+};
+
+/**
+ * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
+ *
+ * @dev: r5f subsystem cluster device node
+ * @mode: cluster mode of type zynqmp_r5_cluster_mode
+ * @core_count: number of r5 cores used for this cluster mode
+ * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
+ */
+struct zynqmp_r5_cluster {
+ struct device *dev;
+ enum zynqmp_r5_cluster_mode mode;
+ int core_count;
+ struct zynqmp_r5_core *r5_cores;
+};
+
+/*
+ * zynqmp_r5_set_mode - set RPU operation mode
+ *
+ * set RPU operation mode
+ *
+ * Return: 0 for success, negative value for failure
+ */
+static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
+ enum zynqmp_r5_cluster_mode rpu_mode)
+{
+ enum rpu_tcm_comb tcm_mode;
+ int ret, reg_val;
+
+ reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
+
+ ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
+ if (ret < 0) {
+ pr_err("failed to set RPU mode\n");
+ return ret;
+ }
+
+ tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
+ PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
+ ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
+ if (ret < 0)
+ pr_err("failed to configure TCM\n");
+
+ return ret;
+}
+
+/*
+ * zynqmp_r5_rproc_start
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Start R5 Core from designated boot address.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_start(struct rproc *rproc)
+{
+ struct zynqmp_r5_core *r5_core = rproc->priv;
+ enum rpu_boot_mem bootmem;
+ int ret;
+
+ if (!r5_core) {
+ pr_err("can't get r5 core\n");
+ return -EINVAL;
+ }
+
+ bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
+ PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
+
+ dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
+ bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
+
+ ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
+ bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
+ if (ret)
+ pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
+ return ret;
+}
+
+/*
+ * zynqmp_r5_rproc_stop
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Power down R5 Core.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_stop(struct rproc *rproc)
+{
+ struct zynqmp_r5_core *r5_core = rproc->priv;
+ int ret;
+
+ ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
+ ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+ if (ret)
+ pr_err("failed to stop remoteproc RPU %d\n", ret);
+
+ return ret;
+}
+
+/*
+ * zynqmp_r5_rproc_mem_map
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to map
+ *
+ * Callback to map va for memory-region's carveout.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
+ struct rproc_mem_entry *mem)
+{
+ void __iomem *va;
+
+ va = ioremap_wc(mem->dma, mem->len);
+ if (IS_ERR_OR_NULL(va))
+ return -ENOMEM;
+
+ mem->va = (void *)va;
+
+ return 0;
+}
+
+/*
+ * zynqmp_r5_rproc_mem_unmap
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to unmap
+ *
+ * Unmap memory-region carveout
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
+ struct rproc_mem_entry *mem)
+{
+ iounmap((void __iomem *)mem->va);
+ return 0;
+}
+
+/*
+ * add_mem_regions
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Construct rproc mem carveouts from carveout provided in
+ * memory-region property
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int add_mem_regions(struct rproc *rproc)
+{
+ struct device *dev;
+ struct rproc_mem_entry *mem;
+ struct reserved_mem *rmem;
+ struct zynqmp_r5_core *r5_core;
+ int i;
+
+ r5_core = rproc->priv;
+ dev = r5_core->dev;
+
+ /* Register associated reserved memory regions */
+ for (i = 0; i < r5_core->res_mem_count; i++) {
+ rmem = &r5_core->res_mem[i];
+ mem = rproc_mem_entry_init(dev, NULL,
+ (dma_addr_t)rmem->base,
+ rmem->size, rmem->base,
+ zynqmp_r5_rproc_mem_map,
+ zynqmp_r5_rproc_mem_unmap,
+ rmem->name);
+ if (IS_ERR_OR_NULL(mem))
+ return -ENOMEM;
+
+ rproc_add_carveout(rproc, mem);
+ }
+
+ return 0;
+}
+
+/*
+ * zynqmp_r5_rproc_mem_unmap
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to unmap
+ *
+ * Unmap TCM banks when powering down R5 core.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
+{
+ struct zynqmp_r5_core *r5_core;
+ int i;
+ enum pm_node_id pm_domain_id;
+
+ r5_core = rproc->priv;
+ if (!r5_core) {
+ pr_err("r5 core is not available\n");
+ return -EINVAL;
+ }
+
+ iounmap((void __iomem *)mem->va);
+
+ for (i = 0; i < r5_core->tcm_bank_count; i++) {
+ pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
+ if (zynqmp_pm_release_node(pm_domain_id))
+ pr_warn("can't turn off TCM bank %d", pm_domain_id);
+ }
+
+ return 0;
+}
+
+/*
+ * tcm_mem_map
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to initialize the va and da fields of
+ *
+ * Given TCM bank entry, this callback will set device address for R5
+ * running on TCM and also setup virtual address for TCM bank
+ * remoteproc carveout.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int tcm_mem_map(struct rproc *rproc,
+ struct rproc_mem_entry *mem)
+{
+ void __iomem *va;
+
+ va = ioremap_wc(mem->dma, mem->len);
+ if (IS_ERR_OR_NULL(va))
+ return -ENOMEM;
+
+ /* Update memory entry va */
+ mem->va = (void *)va;
+
+ /* clear TCMs */
+ memset_io(va, 0, mem->len);
+
+ /*
+ * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
+ * while on the Linux side they are at 0xffexxxxx.
+ *
+ * Zero out the high 12 bits of the address. This will give
+ * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
+ */
+ mem->da &= 0x000fffff;
+
+ /*
+ * TCM Banks 1A and 1B still have to be translated.
+ *
+ * Below handle these two banks' absolute addresses (0xffe90000 and
+ * 0xffeb0000) and convert to the expected relative addresses
+ * (0x0 and 0x20000).
+ */
+ if (mem->da == 0x90000 || mem->da == 0xB0000)
+ mem->da -= 0x90000;
+
+ /* if translated TCM bank address is not valid report error */
+ if (mem->da != 0x0 && mem->da != 0x20000) {
+ dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
+ return -EINVAL;
+ }
+ return 0;
+}
+
+static int add_tcm_carveout_split_mode(struct rproc *rproc)
+{
+ int i, num_banks, ret;
+ struct rproc_mem_entry *mem;
+ enum pm_node_id pm_domain_id;
+ u32 bank_addr;
+ size_t bank_size = 0;
+ char *bank_name;
+ struct device *dev;
+ struct zynqmp_r5_core *r5_core;
+
+ r5_core = (struct zynqmp_r5_core *)rproc->priv;
+ if (!r5_core)
+ return -EINVAL;
+
+ dev = r5_core->dev;
+
+ /* go through zynqmp banks for r5 node */
+ num_banks = r5_core->tcm_bank_count;
+ if (num_banks <= 0) {
+ dev_err(dev, "need to specify TCM banks\n");
+ return -EINVAL;
+ }
+
+ for (i = 0; i < num_banks; i++) {
+ bank_addr = (u32)r5_core->tcm_banks[i].addr;
+ bank_name = r5_core->tcm_banks[i].bank_name;
+ bank_size = r5_core->tcm_banks[i].size;
+ pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
+
+ ret = zynqmp_pm_request_node(pm_domain_id,
+ ZYNQMP_PM_CAPABILITY_ACCESS, 0,
+ ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+ if (ret < 0) {
+ dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
+ return ret;
+ }
+
+ dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
+ bank_name, bank_addr, bank_size);
+
+ /* add carveout */
+ mem = rproc_mem_entry_init(dev, NULL, bank_addr,
+ bank_size, bank_addr,
+ tcm_mem_map, tcm_mem_unmap,
+ bank_name);
+ if (IS_ERR_OR_NULL(mem)) {
+ /* Turn off all TCM banks turned on before */
+ do {
+ pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
+ ret = zynqmp_pm_release_node((u32)pm_domain_id);
+ if (ret)
+ dev_warn(dev,
+ "fail to release node: %x, %x\n",
+ (u32)pm_domain_id, ret);
+ } while (i--);
+ return -ENOMEM;
+ }
+
+ rproc_add_carveout(rproc, mem);
+ }
+
+ return 0;
+}
+
+static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
+{
+ int i, num_banks, ret;
+ struct rproc_mem_entry *mem;
+ enum pm_node_id pm_domain_id;
+ u32 bank_addr;
+ size_t bank_size = 0;
+ char *bank_name;
+ struct device *dev;
+ struct platform_device *parent_pdev;
+ struct zynqmp_r5_cluster *cluster;
+ struct zynqmp_r5_core *r5_core;
+
+ r5_core = (struct zynqmp_r5_core *)rproc->priv;
+ if (!r5_core)
+ return -EINVAL;
+
+ dev = r5_core->dev;
+ if (!dev) {
+ pr_err("r5 core device unavailable\n");
+ return -ENODEV;
+ }
+
+ /* go through zynqmp banks for r5 node */
+ num_banks = r5_core->tcm_bank_count;
+ if (num_banks <= 0) {
+ dev_err(dev, "need to specify TCM banks\n");
+ return -EINVAL;
+ }
+
+ bank_addr = (u32)r5_core->tcm_banks[0].addr;
+ bank_name = r5_core->tcm_banks[0].bank_name;
+ for (i = 0; i < num_banks; i++) {
+ bank_size += r5_core->tcm_banks[i].size;
+ pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
+
+ ret = zynqmp_pm_request_node(pm_domain_id,
+ ZYNQMP_PM_CAPABILITY_ACCESS, 0,
+ ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+ if (ret < 0) {
+ dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
+ return ret;
+ }
+ }
+
+ dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
+ bank_name, bank_addr, bank_size);
+
+ /* add carveout */
+ mem = rproc_mem_entry_init(dev, NULL, bank_addr,
+ bank_size, bank_addr,
+ tcm_mem_map, tcm_mem_unmap,
+ bank_name);
+ if (IS_ERR_OR_NULL(mem)) {
+ for (i = 0; i < num_banks; i++) {
+ pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
+ ret = zynqmp_pm_release_node((u32)pm_domain_id);
+ if (ret)
+ dev_warn(dev,
+ "fail to release node: %x ret: %x\n",
+ (u32)pm_domain_id, ret);
+ }
+ return -ENOMEM;
+ }
+
+ rproc_add_carveout(rproc, mem);
+
+ return 0;
+}
+
+/*
+ * add_tcm_banks()
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Given R5 node in remoteproc instance
+ * allocate remoteproc carveout for TCM memory
+ * needed for firmware to be loaded
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int add_tcm_banks(struct rproc *rproc)
+{
+ struct device *dev;
+ struct platform_device *parent_pdev;
+ struct zynqmp_r5_cluster *cluster;
+ struct zynqmp_r5_core *r5_core;
+
+ r5_core = (struct zynqmp_r5_core *)rproc->priv;
+ if (!r5_core)
+ return -EINVAL;
+
+ dev = r5_core->dev;
+ if (!dev) {
+ pr_err("r5 core device unavailable\n");
+ return -ENODEV;
+ }
+
+ parent_pdev = to_platform_device(dev->parent);
+ if (!parent_pdev) {
+ dev_err(dev, "parent platform dev unavailable\n");
+ return -ENODEV;
+ }
+
+ cluster = platform_get_drvdata(parent_pdev);
+ if (!cluster) {
+ dev_err(&parent_pdev->dev, "Invalid driver data\n");
+ return -EINVAL;
+ }
+
+ if (cluster->mode == SPLIT_MODE)
+ return add_tcm_carveout_split_mode(rproc);
+ else if (cluster->mode == LOCKSTEP_MODE)
+ return add_tcm_carveout_lockstep_mode(rproc);
+
+ dev_err(cluster->dev, "invalid cluster mode\n");
+ return -EINVAL;
+}
+
+/*
+ * zynqmp_r5_parse_fw()
+ * @rproc: single R5 core's corresponding rproc instance
+ * @fw: ptr to firmware to be loaded onto r5 core
+ *
+ * When loading firmware, ensure the necessary carveouts are in remoteproc
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
+{
+ int ret;
+ struct zynqmp_r5_core *r5_core;
+ struct device *dev;
+
+ r5_core = rproc->priv;
+ if (!r5_core) {
+ dev_err(&rproc->dev, "r5 core not available\n");
+ return -EINVAL;
+ }
+
+ dev = r5_core->dev;
+
+ ret = add_tcm_banks(rproc);
+ if (ret) {
+ dev_err(dev, "failed to get TCM banks, err %d\n", ret);
+ return ret;
+ }
+
+ ret = add_mem_regions(rproc);
+ if (ret)
+ dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
+
+ ret = rproc_elf_load_rsc_table(rproc, fw);
+ if (ret == -EINVAL) {
+ /*
+ * resource table only required for IPC.
+ * if not present, this is not necessarily an error;
+ * for example, loading r5 hello world application
+ * so simply inform user and keep going.
+ */
+ dev_info(&rproc->dev, "no resource table found.\n");
+ ret = 0;
+ }
+ return ret;
+}
+
+static struct rproc_ops zynqmp_r5_rproc_ops = {
+ .start = zynqmp_r5_rproc_start,
+ .stop = zynqmp_r5_rproc_stop,
+ .load = rproc_elf_load_segments,
+ .parse_fw = zynqmp_r5_parse_fw,
+ .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
+ .sanity_check = rproc_elf_sanity_check,
+ .get_boot_addr = rproc_elf_get_boot_addr,
+};
+
+static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
+{
+ int i, j, k;
+ struct zynqmp_r5_core *r5_core;
+
+ dev_dbg(cluster->dev, "Printing dt node info\n");
+
+ pr_debug("cluster mode = %d\n", cluster->mode);
+ pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
+ cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
+ pr_debug("r5f num cores = %d\n", cluster->core_count);
+
+ for (i = 0; i < cluster->core_count; i++) {
+ r5_core = &cluster->r5_cores[i];
+ if (!r5_core) {
+ pr_err("can't get r5_core\n");
+ continue;
+ }
+
+ pr_debug("r5 core %d nodes\n", i);
+ pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
+ for (k = 0; k < r5_core->tcm_bank_count; k++) {
+ pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
+ k, r5_core->tcm_banks[k].addr,
+ r5_core->tcm_banks[k].size,
+ r5_core->tcm_banks[k].pm_domain_id,
+ r5_core->tcm_banks[k].bank_name);
+ }
+
+ pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
+
+ for (j = 0; j < r5_core->res_mem_count; j++) {
+ pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
+ j, r5_core->res_mem[j].base,
+ r5_core->res_mem[j].size,
+ r5_core->res_mem[j].name);
+ }
+ }
+}
+
+/**
+ * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
+ * this is called for each individual R5 core to
+ * set up mailbox, Xilinx platform manager unique ID,
+ * add to rproc core
+ *
+ * @r5_core: zynqmp_r5_core r5 core object to initialize
+ *
+ * Return: 0 for success, negative value for failure.
+ */
+static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
+{
+ int ret;
+ struct rproc *r5_rproc;
+ struct device *dev;
+
+ dev = r5_core->dev;
+
+ /* Set up DMA mask */
+ ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
+ if (ret)
+ return ret;
+
+ /* Allocate remoteproc instance */
+ r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
+ NULL, sizeof(struct zynqmp_r5_core));
+ if (IS_ERR_OR_NULL(r5_rproc))
+ return -ENOMEM;
+
+ r5_rproc->auto_boot = false;
+ r5_rproc->priv = r5_core;
+
+ /* Add R5 remoteproc */
+ ret = devm_rproc_add(dev, r5_rproc);
+ if (ret) {
+ pr_err("failed to add r5 remoteproc\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
+{
+ int tcm_bank_count, tcm_node;
+ int i = 0, j;
+ struct zynqmp_r5_core *r5_core;
+ const struct mem_bank_data *tcm = zynqmp_tcm_banks;
+ struct device *dev = cluster->dev;
+
+ /* ToDo: Use predefined TCM address space values from driver until
+ * system-dt spec is not final fot TCM
+ */
+ tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
+
+ /* count per core tcm banks */
+ tcm_bank_count = tcm_bank_count / cluster->core_count;
+
+ /* r5 core 0 will use all of TCM banks in lockstep mode.
+ * In split mode, r5 core0 will use 128k and r5 core1 will use another
+ * 128k. Assign TCM banks to each core accordingly
+ */
+ tcm_node = 0;
+ for (j = 0; j < cluster->core_count; j++) {
+ r5_core = &cluster->r5_cores[j];
+ r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
+ tcm_bank_count, GFP_KERNEL);
+ if (IS_ERR_OR_NULL(r5_core->tcm_banks))
+ return -ENOMEM;
+
+ for (i = 0; i < tcm_bank_count; i++) {
+ /* Use pre-defined TCM reg values.
+ * Eventually this should be replaced by values
+ * parsed from dts.
+ */
+ r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
+ r5_core->tcm_banks[i].size = tcm[tcm_node].size;
+ r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
+ r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
+ tcm_node++;
+ }
+
+ r5_core->tcm_bank_count = tcm_bank_count;
+ }
+
+ return 0;
+}
+
+static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
+{
+ int res_mem_count, i;
+ struct device *dev;
+ struct device_node *np, *rmem_np;
+ struct reserved_mem *rmem;
+
+ dev = r5_core->dev;
+
+ np = r5_core->np;
+ if (IS_ERR_OR_NULL(np)) {
+ pr_err("invalid device node of r5 core\n");
+ return -EINVAL;
+ }
+
+ res_mem_count = of_property_count_elems_of_size(np, "memory-region",
+ sizeof(phandle));
+ if (res_mem_count <= 0) {
+ dev_warn(dev, "failed to get memory-region property %d\n",
+ res_mem_count);
+ return -EINVAL;
+ }
+
+ r5_core->res_mem = devm_kzalloc(dev,
+ res_mem_count * sizeof(struct reserved_mem),
+ GFP_KERNEL);
+ if (!r5_core->res_mem) {
+ dev_err(dev, "failed to allocate mem region memory\n");
+ return -ENOMEM;
+ }
+
+ for (i = 0; i < res_mem_count; i++) {
+ rmem_np = of_parse_phandle(np, "memory-region", i);
+ if (!rmem_np)
+ return -EINVAL;
+
+ rmem = of_reserved_mem_lookup(rmem_np);
+ if (!rmem) {
+ of_node_put(rmem_np);
+ return -EINVAL;
+ }
+
+ memcpy(&r5_core->res_mem[i], rmem,
+ sizeof(struct reserved_mem));
+ of_node_put(rmem_np);
+ }
+
+ r5_core->res_mem_count = res_mem_count;
+
+ return 0;
+}
+
+static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
+{
+ int ret, i;
+ struct zynqmp_r5_core *r5_core;
+ struct device *dev = cluster->dev;
+
+ ret = zynqmp_r5_get_tcm_node(cluster);
+ if (ret < 0) {
+ dev_err(dev, "can't get tcm node, err %d\n", ret);
+ return ret;
+ }
+
+ for (i = 0; i < cluster->core_count; i++) {
+ r5_core = &cluster->r5_cores[i];
+ if (!r5_core) {
+ pr_err("invalid r5 core\n");
+ return -EINVAL;
+ }
+
+ ret = zynqmp_r5_get_mem_region_node(r5_core);
+ if (ret)
+ dev_warn(dev, "memory-region prop failed %d\n", ret);
+
+ ret = of_property_read_u32_index(r5_core->np, "power-domains",
+ 1, &r5_core->pm_domain_id);
+ if (ret) {
+ dev_err(dev, "failed to get power-domains property\n");
+ return ret;
+ }
+
+ ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
+ if (ret)
+ return ret;
+
+ ret = zynqmp_r5_add_rproc_core(r5_core);
+ if (ret) {
+ dev_err(dev, "failed to init r5 core %d\n", i);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
+{
+ struct device *dev = cluster->dev;
+ struct device_node *dev_node = dev_of_node(dev);
+ struct device_node *child;
+ struct platform_device *child_pdev;
+ int core_count = 0, ret, i;
+ enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
+ struct zynqmp_r5_core *r5_cores;
+
+ ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
+
+ /* on success returns 0, if not defined then returns -EINVAL,
+ * In that case, default is LOCKSTEP mode
+ */
+ if (ret != -EINVAL && ret != 0) {
+ dev_err(dev, "Invalid xlnx,cluster-mode property\n");
+ return -EINVAL;
+ }
+
+ if (cluster_mode == SINGLE_CPU_MODE) {
+ dev_err(dev, "driver does not support single cpu mode\n");
+ return -EINVAL;
+ } else if ((cluster_mode != SPLIT_MODE &&
+ cluster_mode != LOCKSTEP_MODE)) {
+ dev_err(dev, "Invalid cluster mode\n");
+ return -EINVAL;
+ }
+
+ core_count = of_get_available_child_count(dev_node);
+ if (core_count <= 0) {
+ dev_err(dev, "Invalid number of r5 cores %d", core_count);
+ return -EINVAL;
+ } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
+ dev_err(dev, "Invalid number of r5 cores for split mode\n");
+ return -EINVAL;
+ } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
+ dev_warn(dev, "Only r5 core0 will be used\n");
+ core_count = 1;
+ }
+
+ r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
+ core_count, GFP_KERNEL);
+ if (IS_ERR_OR_NULL(r5_cores)) {
+ dev_err(dev, "can't allocate memory for cores\n");
+ return -ENOMEM;
+ }
+
+ i = 0;
+ for_each_available_child_of_node(dev_node, child) {
+ child_pdev = of_find_device_by_node(child);
+ if (!child_pdev)
+ return -ENODEV;
+
+ r5_cores[i].dev = &child_pdev->dev;
+ if (!r5_cores[i].dev) {
+ pr_err("can't get device for r5 core %d\n", i);
+ return -ENODEV;
+ }
+
+ r5_cores[i].np = dev_of_node(r5_cores[i].dev);
+ if (!r5_cores[i].np) {
+ pr_err("can't get device node for r5 core %d\n", i);
+ return -ENODEV;
+ }
+
+ i++;
+ if (i == core_count)
+ break;
+ }
+
+ cluster->mode = cluster_mode;
+ cluster->core_count = core_count;
+ cluster->r5_cores = r5_cores;
+
+ ret = zynqmp_r5_core_init(cluster);
+ if (ret < 0) {
+ dev_err(dev, "failed to init r5 core err %d\n", ret);
+ return ret;
+ }
+
+ zynqmp_r5_print_dt_node_info(cluster);
+
+ return 0;
+}
+
+static void zynqmp_r5_cluster_exit(void *data)
+{
+ struct platform_device *pdev = (struct platform_device *)data;
+
+ platform_set_drvdata(pdev, NULL);
+
+ pr_info("Exit r5f subsystem driver\n");
+}
+
+/*
+ * zynqmp_r5_remoteproc_probe()
+ *
+ * @pdev: domain platform device for R5 cluster
+ *
+ * called when driver is probed, for each R5 core specified in DT,
+ * setup as needed to do remoteproc-related operations
+ *
+ * Return: 0 for success, negative value for failure.
+ */
+static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
+{
+ int ret;
+ struct zynqmp_r5_cluster *cluster;
+ struct device *dev = &pdev->dev;
+
+ cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
+ if (IS_ERR_OR_NULL(cluster))
+ return -ENOMEM;
+
+ cluster->dev = dev;
+
+ ret = devm_of_platform_populate(dev);
+ if (ret) {
+ dev_err(dev, "failed to populate platform dev %d\n", ret);
+ return ret;
+ }
+
+ /* wire in so each core can be cleaned up at driver remove */
+ platform_set_drvdata(pdev, cluster);
+
+ ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
+ if (ret)
+ return ret;
+
+ ret = zynqmp_r5_cluster_init(cluster);
+ if (ret) {
+ dev_err(dev, "Invalid r5f subsystem device tree\n");
+ return ret;
+ }
+
+ dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
+ return 0;
+}
+
+/* Match table for OF platform binding */
+static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
+ { .compatible = "xlnx,zynqmp-r5fss", },
+ { /* end of list */ },
+};
+MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
+
+static struct platform_driver zynqmp_r5_remoteproc_driver = {
+ .probe = zynqmp_r5_remoteproc_probe,
+ .driver = {
+ .name = "zynqmp_r5_remoteproc",
+ .of_match_table = zynqmp_r5_remoteproc_match,
+ },
+};
+module_platform_driver(zynqmp_r5_remoteproc_driver);
+
+MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
+MODULE_AUTHOR("Xilinx Inc.");
+MODULE_LICENSE("GPL v2");
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 56+ messages in thread
* Re: [PATCH v2 2/6] arm64: dts: xilinx: zynqmp: Add RPU subsystem device node
2021-11-23 6:20 ` Tanmay Shah
@ 2021-11-29 17:58 ` Mathieu Poirier
-1 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-11-29 17:58 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On Mon, Nov 22, 2021 at 10:20:46PM -0800, Tanmay Shah wrote:
> RPU subsystem can be configured in cluster-mode or split mode.
> Also each r5 core has separate power domains.
>
> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
> ---
> arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 17 +++++++++++++++++
> 1 file changed, 17 insertions(+)
>
> diff --git a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
> index 28dccb891a53..f4fb98ccb1b5 100644
> --- a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
> +++ b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
> @@ -654,6 +654,23 @@ qspi: spi@ff0f0000 {
> power-domains = <&zynqmp_firmware PD_QSPI>;
> };
>
> + r5fss: r5fss@ff9a0000 {
> + compatible = "xlnx,zynqmp-r5fss";
> + xlnx,cluster-mode = <1>;
> +
> + reg = <0x0 0Xff9a0000 0x0 0x228>;
This line is giving me an obvious checkpatch warning. Please make sure you run
checkpatch.pl before sending out a patchset.
> +
> + r5f_core0: r5f-0 {
> + compatible = "xlnx,zynqmp-r5f";
> + power-domains = <&zynqmp_firmware PD_RPU_0>;
> + };
> +
> + r5f_core1: r5f-1 {
> + compatible = "xlnx,zynqmp-r5f";
> + power-domains = <&zynqmp_firmware PD_RPU_1>;
> + };
> + };
> +
> psgtr: phy@fd400000 {
> compatible = "xlnx,zynqmp-psgtr-v1.1";
> status = "disabled";
> --
> 2.25.1
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 2/6] arm64: dts: xilinx: zynqmp: Add RPU subsystem device node
@ 2021-11-29 17:58 ` Mathieu Poirier
0 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-11-29 17:58 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On Mon, Nov 22, 2021 at 10:20:46PM -0800, Tanmay Shah wrote:
> RPU subsystem can be configured in cluster-mode or split mode.
> Also each r5 core has separate power domains.
>
> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
> ---
> arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 17 +++++++++++++++++
> 1 file changed, 17 insertions(+)
>
> diff --git a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
> index 28dccb891a53..f4fb98ccb1b5 100644
> --- a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
> +++ b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
> @@ -654,6 +654,23 @@ qspi: spi@ff0f0000 {
> power-domains = <&zynqmp_firmware PD_QSPI>;
> };
>
> + r5fss: r5fss@ff9a0000 {
> + compatible = "xlnx,zynqmp-r5fss";
> + xlnx,cluster-mode = <1>;
> +
> + reg = <0x0 0Xff9a0000 0x0 0x228>;
This line is giving me an obvious checkpatch warning. Please make sure you run
checkpatch.pl before sending out a patchset.
> +
> + r5f_core0: r5f-0 {
> + compatible = "xlnx,zynqmp-r5f";
> + power-domains = <&zynqmp_firmware PD_RPU_0>;
> + };
> +
> + r5f_core1: r5f-1 {
> + compatible = "xlnx,zynqmp-r5f";
> + power-domains = <&zynqmp_firmware PD_RPU_1>;
> + };
> + };
> +
> psgtr: phy@fd400000 {
> compatible = "xlnx,zynqmp-psgtr-v1.1";
> status = "disabled";
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-11-23 6:20 ` Tanmay Shah
@ 2021-11-29 18:42 ` Mathieu Poirier
-1 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-11-29 18:42 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
> This driver enables r5f dual core Real time Processing Unit subsystem
> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
> (cluster) can be configured in different modes e.g. split mode in which
> two r5f cores work independent of each other and lock-step mode in which
> both r5f cores execute same code clock-for-clock and notify if the
> result is different.
>
> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
>
> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
> ---
> drivers/remoteproc/Kconfig | 12 +
> drivers/remoteproc/Makefile | 1 +
> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
> 3 files changed, 972 insertions(+)
> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
...and this patch gives me complation warnings:
CC drivers/remoteproc/xlnx_r5_remoteproc.o
kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c: In function ‘add_tcm_carveout_lockstep_mode’:
kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:412:28: warning: unused variable ‘cluster’ [-Wunused-variable]
412 | struct zynqmp_r5_cluster *cluster;
| ^~~~~~~
kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:411:26: warning: unused variable ‘parent_pdev’ [-Wunused-variable]
411 | struct platform_device *parent_pdev;
| ^~~~~~~~~~~
The above leads me to believe this patchset was not compiled before it was sent
out.
Being new to this I can understand that checkpatch.pl was omitted (albeit amply
documented) but obvious compilation warnings can't be excused. As such I
am dropping this set and will not review another version until January.
Mathieu
>
> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> index f30d00a3aabe..27f66910d8d3 100644
> --- a/drivers/remoteproc/Kconfig
> +++ b/drivers/remoteproc/Kconfig
> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
> It's safe to say N here if you're not interested in utilizing
> a slave processor.
>
> +config XLNX_R5_REMOTEPROC
> + tristate "Xilinx R5 remoteproc support"
> + depends on PM && ARCH_ZYNQMP
> + depends on ZYNQMP_FIRMWARE
> + select RPMSG_VIRTIO
> + select ZYNQMP_IPI_MBOX
> + help
> + Say y or m here to support Xilinx R5 remote processors via the remote
> + processor framework.
> +
> + It's safe to say N if not interested in using RPU r5f cores.
> +
> endif # REMOTEPROC
>
> endmenu
> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> index bb26c9e4ef9c..334a8bed4c14 100644
> --- a/drivers/remoteproc/Makefile
> +++ b/drivers/remoteproc/Makefile
> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> new file mode 100644
> index 000000000000..c2167fd3869d
> --- /dev/null
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -0,0 +1,959 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ZynqMP R5 Remote Processor driver
> + *
> + */
> +
> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
> +#include <linux/firmware/xlnx-zynqmp.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/mailbox_client.h>
> +#include <linux/mailbox/zynqmp-ipi-message.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/remoteproc.h>
> +#include <linux/skbuff.h>
> +#include <linux/sysfs.h>
> +
> +#include "remoteproc_internal.h"
> +
> +/* settings for RPU cluster mode */
> +enum zynqmp_r5_cluster_mode {
> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
> +};
> +
> +/**
> + * struct mem_bank_data - Memory Bank description
> + *
> + * @addr: Start address of memory bank
> + * @size: Size of Memory bank
> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
> + * @bank_name: name of the bank for remoteproc framework
> + */
> +struct mem_bank_data {
> + phys_addr_t addr;
> + size_t size;
> + enum pm_node_id pm_domain_id;
> + char *bank_name;
> +};
> +
> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
> +};
> +
> +/**
> + * struct zynqmp_r5_core - ZynqMP R5 core structure
> + *
> + * @dev: device of RPU instance
> + * @np: device node of RPU instance
> + * @tcm_bank_count: number TCM banks accessible to this RPU
> + * @tcm_banks: array of each TCM bank data
> + * @res_mem_count: number of Reserved Memory regions per core
> + * @res_mem: array of reserved memory regions
> + * @rproc: rproc handle
> + * @pm_domain_id: RPU CPU power domain id
> + */
> +struct zynqmp_r5_core {
> + struct device *dev;
> + struct device_node *np;
> + int tcm_bank_count;
> + struct mem_bank_data *tcm_banks;
> + int res_mem_count;
> + struct reserved_mem *res_mem;
> + struct rproc *rproc;
> + enum pm_node_id pm_domain_id;
> +};
> +
> +/**
> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
> + *
> + * @dev: r5f subsystem cluster device node
> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
> + * @core_count: number of r5 cores used for this cluster mode
> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
> + */
> +struct zynqmp_r5_cluster {
> + struct device *dev;
> + enum zynqmp_r5_cluster_mode mode;
> + int core_count;
> + struct zynqmp_r5_core *r5_cores;
> +};
> +
> +/*
> + * zynqmp_r5_set_mode - set RPU operation mode
> + *
> + * set RPU operation mode
> + *
> + * Return: 0 for success, negative value for failure
> + */
> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
> + enum zynqmp_r5_cluster_mode rpu_mode)
> +{
> + enum rpu_tcm_comb tcm_mode;
> + int ret, reg_val;
> +
> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
> +
> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
> + if (ret < 0) {
> + pr_err("failed to set RPU mode\n");
> + return ret;
> + }
> +
> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
> + if (ret < 0)
> + pr_err("failed to configure TCM\n");
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_start
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Start R5 Core from designated boot address.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + enum rpu_boot_mem bootmem;
> + int ret;
> +
> + if (!r5_core) {
> + pr_err("can't get r5 core\n");
> + return -EINVAL;
> + }
> +
> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> +
> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> +
> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> + if (ret)
> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_stop
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Power down R5 Core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + int ret;
> +
> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret)
> + pr_err("failed to stop remoteproc RPU %d\n", ret);
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + mem->va = (void *)va;
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap memory-region carveout
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + iounmap((void __iomem *)mem->va);
> + return 0;
> +}
> +
> +/*
> + * add_mem_regions
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Construct rproc mem carveouts from carveout provided in
> + * memory-region property
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_mem_regions(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct rproc_mem_entry *mem;
> + struct reserved_mem *rmem;
> + struct zynqmp_r5_core *r5_core;
> + int i;
> +
> + r5_core = rproc->priv;
> + dev = r5_core->dev;
> +
> + /* Register associated reserved memory regions */
> + for (i = 0; i < r5_core->res_mem_count; i++) {
> + rmem = &r5_core->res_mem[i];
> + mem = rproc_mem_entry_init(dev, NULL,
> + (dma_addr_t)rmem->base,
> + rmem->size, rmem->base,
> + zynqmp_r5_rproc_mem_map,
> + zynqmp_r5_rproc_mem_unmap,
> + rmem->name);
> + if (IS_ERR_OR_NULL(mem))
> + return -ENOMEM;
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap TCM banks when powering down R5 core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> + struct zynqmp_r5_core *r5_core;
> + int i;
> + enum pm_node_id pm_domain_id;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + pr_err("r5 core is not available\n");
> + return -EINVAL;
> + }
> +
> + iounmap((void __iomem *)mem->va);
> +
> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + if (zynqmp_pm_release_node(pm_domain_id))
> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * tcm_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to initialize the va and da fields of
> + *
> + * Given TCM bank entry, this callback will set device address for R5
> + * running on TCM and also setup virtual address for TCM bank
> + * remoteproc carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + /* Update memory entry va */
> + mem->va = (void *)va;
> +
> + /* clear TCMs */
> + memset_io(va, 0, mem->len);
> +
> + /*
> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> + * while on the Linux side they are at 0xffexxxxx.
> + *
> + * Zero out the high 12 bits of the address. This will give
> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> + */
> + mem->da &= 0x000fffff;
> +
> + /*
> + * TCM Banks 1A and 1B still have to be translated.
> + *
> + * Below handle these two banks' absolute addresses (0xffe90000 and
> + * 0xffeb0000) and convert to the expected relative addresses
> + * (0x0 and 0x20000).
> + */
> + if (mem->da == 0x90000 || mem->da == 0xB0000)
> + mem->da -= 0x90000;
> +
> + /* if translated TCM bank address is not valid report error */
> + if (mem->da != 0x0 && mem->da != 0x20000) {
> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
> + return -EINVAL;
> + }
> + return 0;
> +}
> +
> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
> + char *bank_name;
> + struct device *dev;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + for (i = 0; i < num_banks; i++) {
> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
> + bank_name = r5_core->tcm_banks[i].bank_name;
> + bank_size = r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> +
> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + /* Turn off all TCM banks turned on before */
> + do {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x, %x\n",
> + (u32)pm_domain_id, ret);
> + } while (i--);
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
> + char *bank_name;
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
> + bank_name = r5_core->tcm_banks[0].bank_name;
> + for (i = 0; i < num_banks; i++) {
> + bank_size += r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> + }
> +
> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + for (i = 0; i < num_banks; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x ret: %x\n",
> + (u32)pm_domain_id, ret);
> + }
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> +
> + return 0;
> +}
> +
> +/*
> + * add_tcm_banks()
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Given R5 node in remoteproc instance
> + * allocate remoteproc carveout for TCM memory
> + * needed for firmware to be loaded
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_tcm_banks(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + parent_pdev = to_platform_device(dev->parent);
> + if (!parent_pdev) {
> + dev_err(dev, "parent platform dev unavailable\n");
> + return -ENODEV;
> + }
> +
> + cluster = platform_get_drvdata(parent_pdev);
> + if (!cluster) {
> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
> + return -EINVAL;
> + }
> +
> + if (cluster->mode == SPLIT_MODE)
> + return add_tcm_carveout_split_mode(rproc);
> + else if (cluster->mode == LOCKSTEP_MODE)
> + return add_tcm_carveout_lockstep_mode(rproc);
> +
> + dev_err(cluster->dev, "invalid cluster mode\n");
> + return -EINVAL;
> +}
> +
> +/*
> + * zynqmp_r5_parse_fw()
> + * @rproc: single R5 core's corresponding rproc instance
> + * @fw: ptr to firmware to be loaded onto r5 core
> + *
> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> + int ret;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + dev_err(&rproc->dev, "r5 core not available\n");
> + return -EINVAL;
> + }
> +
> + dev = r5_core->dev;
> +
> + ret = add_tcm_banks(rproc);
> + if (ret) {
> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
> + return ret;
> + }
> +
> + ret = add_mem_regions(rproc);
> + if (ret)
> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
> +
> + ret = rproc_elf_load_rsc_table(rproc, fw);
> + if (ret == -EINVAL) {
> + /*
> + * resource table only required for IPC.
> + * if not present, this is not necessarily an error;
> + * for example, loading r5 hello world application
> + * so simply inform user and keep going.
> + */
> + dev_info(&rproc->dev, "no resource table found.\n");
> + ret = 0;
> + }
> + return ret;
> +}
> +
> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> + .start = zynqmp_r5_rproc_start,
> + .stop = zynqmp_r5_rproc_stop,
> + .load = rproc_elf_load_segments,
> + .parse_fw = zynqmp_r5_parse_fw,
> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> + .sanity_check = rproc_elf_sanity_check,
> + .get_boot_addr = rproc_elf_get_boot_addr,
> +};
> +
> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
> +{
> + int i, j, k;
> + struct zynqmp_r5_core *r5_core;
> +
> + dev_dbg(cluster->dev, "Printing dt node info\n");
> +
> + pr_debug("cluster mode = %d\n", cluster->mode);
> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
> + pr_debug("r5f num cores = %d\n", cluster->core_count);
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("can't get r5_core\n");
> + continue;
> + }
> +
> + pr_debug("r5 core %d nodes\n", i);
> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
> + k, r5_core->tcm_banks[k].addr,
> + r5_core->tcm_banks[k].size,
> + r5_core->tcm_banks[k].pm_domain_id,
> + r5_core->tcm_banks[k].bank_name);
> + }
> +
> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
> +
> + for (j = 0; j < r5_core->res_mem_count; j++) {
> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
> + j, r5_core->res_mem[j].base,
> + r5_core->res_mem[j].size,
> + r5_core->res_mem[j].name);
> + }
> + }
> +}
> +
> +/**
> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
> + * this is called for each individual R5 core to
> + * set up mailbox, Xilinx platform manager unique ID,
> + * add to rproc core
> + *
> + * @r5_core: zynqmp_r5_core r5 core object to initialize
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
> +{
> + int ret;
> + struct rproc *r5_rproc;
> + struct device *dev;
> +
> + dev = r5_core->dev;
> +
> + /* Set up DMA mask */
> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> + if (ret)
> + return ret;
> +
> + /* Allocate remoteproc instance */
> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> + NULL, sizeof(struct zynqmp_r5_core));
> + if (IS_ERR_OR_NULL(r5_rproc))
> + return -ENOMEM;
> +
> + r5_rproc->auto_boot = false;
> + r5_rproc->priv = r5_core;
> +
> + /* Add R5 remoteproc */
> + ret = devm_rproc_add(dev, r5_rproc);
> + if (ret) {
> + pr_err("failed to add r5 remoteproc\n");
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> +{
> + int tcm_bank_count, tcm_node;
> + int i = 0, j;
> + struct zynqmp_r5_core *r5_core;
> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> + struct device *dev = cluster->dev;
> +
> + /* ToDo: Use predefined TCM address space values from driver until
> + * system-dt spec is not final fot TCM
> + */
> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> +
> + /* count per core tcm banks */
> + tcm_bank_count = tcm_bank_count / cluster->core_count;
> +
> + /* r5 core 0 will use all of TCM banks in lockstep mode.
> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> + * 128k. Assign TCM banks to each core accordingly
> + */
> + tcm_node = 0;
> + for (j = 0; j < cluster->core_count; j++) {
> + r5_core = &cluster->r5_cores[j];
> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> + tcm_bank_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> + return -ENOMEM;
> +
> + for (i = 0; i < tcm_bank_count; i++) {
> + /* Use pre-defined TCM reg values.
> + * Eventually this should be replaced by values
> + * parsed from dts.
> + */
> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> + tcm_node++;
> + }
> +
> + r5_core->tcm_bank_count = tcm_bank_count;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> +{
> + int res_mem_count, i;
> + struct device *dev;
> + struct device_node *np, *rmem_np;
> + struct reserved_mem *rmem;
> +
> + dev = r5_core->dev;
> +
> + np = r5_core->np;
> + if (IS_ERR_OR_NULL(np)) {
> + pr_err("invalid device node of r5 core\n");
> + return -EINVAL;
> + }
> +
> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> + sizeof(phandle));
> + if (res_mem_count <= 0) {
> + dev_warn(dev, "failed to get memory-region property %d\n",
> + res_mem_count);
> + return -EINVAL;
> + }
> +
> + r5_core->res_mem = devm_kzalloc(dev,
> + res_mem_count * sizeof(struct reserved_mem),
> + GFP_KERNEL);
> + if (!r5_core->res_mem) {
> + dev_err(dev, "failed to allocate mem region memory\n");
> + return -ENOMEM;
> + }
> +
> + for (i = 0; i < res_mem_count; i++) {
> + rmem_np = of_parse_phandle(np, "memory-region", i);
> + if (!rmem_np)
> + return -EINVAL;
> +
> + rmem = of_reserved_mem_lookup(rmem_np);
> + if (!rmem) {
> + of_node_put(rmem_np);
> + return -EINVAL;
> + }
> +
> + memcpy(&r5_core->res_mem[i], rmem,
> + sizeof(struct reserved_mem));
> + of_node_put(rmem_np);
> + }
> +
> + r5_core->res_mem_count = res_mem_count;
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> +{
> + int ret, i;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev = cluster->dev;
> +
> + ret = zynqmp_r5_get_tcm_node(cluster);
> + if (ret < 0) {
> + dev_err(dev, "can't get tcm node, err %d\n", ret);
> + return ret;
> + }
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("invalid r5 core\n");
> + return -EINVAL;
> + }
> +
> + ret = zynqmp_r5_get_mem_region_node(r5_core);
> + if (ret)
> + dev_warn(dev, "memory-region prop failed %d\n", ret);
> +
> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> + 1, &r5_core->pm_domain_id);
> + if (ret) {
> + dev_err(dev, "failed to get power-domains property\n");
> + return ret;
> + }
> +
> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_add_rproc_core(r5_core);
> + if (ret) {
> + dev_err(dev, "failed to init r5 core %d\n", i);
> + return ret;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> +{
> + struct device *dev = cluster->dev;
> + struct device_node *dev_node = dev_of_node(dev);
> + struct device_node *child;
> + struct platform_device *child_pdev;
> + int core_count = 0, ret, i;
> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> + struct zynqmp_r5_core *r5_cores;
> +
> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> +
> + /* on success returns 0, if not defined then returns -EINVAL,
> + * In that case, default is LOCKSTEP mode
> + */
> + if (ret != -EINVAL && ret != 0) {
> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> + return -EINVAL;
> + }
> +
> + if (cluster_mode == SINGLE_CPU_MODE) {
> + dev_err(dev, "driver does not support single cpu mode\n");
> + return -EINVAL;
> + } else if ((cluster_mode != SPLIT_MODE &&
> + cluster_mode != LOCKSTEP_MODE)) {
> + dev_err(dev, "Invalid cluster mode\n");
> + return -EINVAL;
> + }
> +
> + core_count = of_get_available_child_count(dev_node);
> + if (core_count <= 0) {
> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> + return -EINVAL;
> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> + return -EINVAL;
> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> + dev_warn(dev, "Only r5 core0 will be used\n");
> + core_count = 1;
> + }
> +
> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> + core_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_cores)) {
> + dev_err(dev, "can't allocate memory for cores\n");
> + return -ENOMEM;
> + }
> +
> + i = 0;
> + for_each_available_child_of_node(dev_node, child) {
> + child_pdev = of_find_device_by_node(child);
> + if (!child_pdev)
> + return -ENODEV;
> +
> + r5_cores[i].dev = &child_pdev->dev;
> + if (!r5_cores[i].dev) {
> + pr_err("can't get device for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> + if (!r5_cores[i].np) {
> + pr_err("can't get device node for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + i++;
> + if (i == core_count)
> + break;
> + }
> +
> + cluster->mode = cluster_mode;
> + cluster->core_count = core_count;
> + cluster->r5_cores = r5_cores;
> +
> + ret = zynqmp_r5_core_init(cluster);
> + if (ret < 0) {
> + dev_err(dev, "failed to init r5 core err %d\n", ret);
> + return ret;
> + }
> +
> + zynqmp_r5_print_dt_node_info(cluster);
> +
> + return 0;
> +}
> +
> +static void zynqmp_r5_cluster_exit(void *data)
> +{
> + struct platform_device *pdev = (struct platform_device *)data;
> +
> + platform_set_drvdata(pdev, NULL);
> +
> + pr_info("Exit r5f subsystem driver\n");
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_probe()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * called when driver is probed, for each R5 core specified in DT,
> + * setup as needed to do remoteproc-related operations
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> +{
> + int ret;
> + struct zynqmp_r5_cluster *cluster;
> + struct device *dev = &pdev->dev;
> +
> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> + if (IS_ERR_OR_NULL(cluster))
> + return -ENOMEM;
> +
> + cluster->dev = dev;
> +
> + ret = devm_of_platform_populate(dev);
> + if (ret) {
> + dev_err(dev, "failed to populate platform dev %d\n", ret);
> + return ret;
> + }
> +
> + /* wire in so each core can be cleaned up at driver remove */
> + platform_set_drvdata(pdev, cluster);
> +
> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_cluster_init(cluster);
> + if (ret) {
> + dev_err(dev, "Invalid r5f subsystem device tree\n");
> + return ret;
> + }
> +
> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> + return 0;
> +}
> +
> +/* Match table for OF platform binding */
> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> + { .compatible = "xlnx,zynqmp-r5fss", },
> + { /* end of list */ },
> +};
> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> +
> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> + .probe = zynqmp_r5_remoteproc_probe,
> + .driver = {
> + .name = "zynqmp_r5_remoteproc",
> + .of_match_table = zynqmp_r5_remoteproc_match,
> + },
> +};
> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> +
> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> +MODULE_AUTHOR("Xilinx Inc.");
> +MODULE_LICENSE("GPL v2");
> --
> 2.25.1
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-11-29 18:42 ` Mathieu Poirier
0 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-11-29 18:42 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
> This driver enables r5f dual core Real time Processing Unit subsystem
> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
> (cluster) can be configured in different modes e.g. split mode in which
> two r5f cores work independent of each other and lock-step mode in which
> both r5f cores execute same code clock-for-clock and notify if the
> result is different.
>
> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
>
> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
> ---
> drivers/remoteproc/Kconfig | 12 +
> drivers/remoteproc/Makefile | 1 +
> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
> 3 files changed, 972 insertions(+)
> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
...and this patch gives me complation warnings:
CC drivers/remoteproc/xlnx_r5_remoteproc.o
kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c: In function ‘add_tcm_carveout_lockstep_mode’:
kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:412:28: warning: unused variable ‘cluster’ [-Wunused-variable]
412 | struct zynqmp_r5_cluster *cluster;
| ^~~~~~~
kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:411:26: warning: unused variable ‘parent_pdev’ [-Wunused-variable]
411 | struct platform_device *parent_pdev;
| ^~~~~~~~~~~
The above leads me to believe this patchset was not compiled before it was sent
out.
Being new to this I can understand that checkpatch.pl was omitted (albeit amply
documented) but obvious compilation warnings can't be excused. As such I
am dropping this set and will not review another version until January.
Mathieu
>
> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> index f30d00a3aabe..27f66910d8d3 100644
> --- a/drivers/remoteproc/Kconfig
> +++ b/drivers/remoteproc/Kconfig
> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
> It's safe to say N here if you're not interested in utilizing
> a slave processor.
>
> +config XLNX_R5_REMOTEPROC
> + tristate "Xilinx R5 remoteproc support"
> + depends on PM && ARCH_ZYNQMP
> + depends on ZYNQMP_FIRMWARE
> + select RPMSG_VIRTIO
> + select ZYNQMP_IPI_MBOX
> + help
> + Say y or m here to support Xilinx R5 remote processors via the remote
> + processor framework.
> +
> + It's safe to say N if not interested in using RPU r5f cores.
> +
> endif # REMOTEPROC
>
> endmenu
> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> index bb26c9e4ef9c..334a8bed4c14 100644
> --- a/drivers/remoteproc/Makefile
> +++ b/drivers/remoteproc/Makefile
> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> new file mode 100644
> index 000000000000..c2167fd3869d
> --- /dev/null
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -0,0 +1,959 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ZynqMP R5 Remote Processor driver
> + *
> + */
> +
> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
> +#include <linux/firmware/xlnx-zynqmp.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/mailbox_client.h>
> +#include <linux/mailbox/zynqmp-ipi-message.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/remoteproc.h>
> +#include <linux/skbuff.h>
> +#include <linux/sysfs.h>
> +
> +#include "remoteproc_internal.h"
> +
> +/* settings for RPU cluster mode */
> +enum zynqmp_r5_cluster_mode {
> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
> +};
> +
> +/**
> + * struct mem_bank_data - Memory Bank description
> + *
> + * @addr: Start address of memory bank
> + * @size: Size of Memory bank
> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
> + * @bank_name: name of the bank for remoteproc framework
> + */
> +struct mem_bank_data {
> + phys_addr_t addr;
> + size_t size;
> + enum pm_node_id pm_domain_id;
> + char *bank_name;
> +};
> +
> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
> +};
> +
> +/**
> + * struct zynqmp_r5_core - ZynqMP R5 core structure
> + *
> + * @dev: device of RPU instance
> + * @np: device node of RPU instance
> + * @tcm_bank_count: number TCM banks accessible to this RPU
> + * @tcm_banks: array of each TCM bank data
> + * @res_mem_count: number of Reserved Memory regions per core
> + * @res_mem: array of reserved memory regions
> + * @rproc: rproc handle
> + * @pm_domain_id: RPU CPU power domain id
> + */
> +struct zynqmp_r5_core {
> + struct device *dev;
> + struct device_node *np;
> + int tcm_bank_count;
> + struct mem_bank_data *tcm_banks;
> + int res_mem_count;
> + struct reserved_mem *res_mem;
> + struct rproc *rproc;
> + enum pm_node_id pm_domain_id;
> +};
> +
> +/**
> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
> + *
> + * @dev: r5f subsystem cluster device node
> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
> + * @core_count: number of r5 cores used for this cluster mode
> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
> + */
> +struct zynqmp_r5_cluster {
> + struct device *dev;
> + enum zynqmp_r5_cluster_mode mode;
> + int core_count;
> + struct zynqmp_r5_core *r5_cores;
> +};
> +
> +/*
> + * zynqmp_r5_set_mode - set RPU operation mode
> + *
> + * set RPU operation mode
> + *
> + * Return: 0 for success, negative value for failure
> + */
> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
> + enum zynqmp_r5_cluster_mode rpu_mode)
> +{
> + enum rpu_tcm_comb tcm_mode;
> + int ret, reg_val;
> +
> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
> +
> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
> + if (ret < 0) {
> + pr_err("failed to set RPU mode\n");
> + return ret;
> + }
> +
> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
> + if (ret < 0)
> + pr_err("failed to configure TCM\n");
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_start
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Start R5 Core from designated boot address.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + enum rpu_boot_mem bootmem;
> + int ret;
> +
> + if (!r5_core) {
> + pr_err("can't get r5 core\n");
> + return -EINVAL;
> + }
> +
> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> +
> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> +
> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> + if (ret)
> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_stop
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Power down R5 Core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + int ret;
> +
> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret)
> + pr_err("failed to stop remoteproc RPU %d\n", ret);
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + mem->va = (void *)va;
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap memory-region carveout
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + iounmap((void __iomem *)mem->va);
> + return 0;
> +}
> +
> +/*
> + * add_mem_regions
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Construct rproc mem carveouts from carveout provided in
> + * memory-region property
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_mem_regions(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct rproc_mem_entry *mem;
> + struct reserved_mem *rmem;
> + struct zynqmp_r5_core *r5_core;
> + int i;
> +
> + r5_core = rproc->priv;
> + dev = r5_core->dev;
> +
> + /* Register associated reserved memory regions */
> + for (i = 0; i < r5_core->res_mem_count; i++) {
> + rmem = &r5_core->res_mem[i];
> + mem = rproc_mem_entry_init(dev, NULL,
> + (dma_addr_t)rmem->base,
> + rmem->size, rmem->base,
> + zynqmp_r5_rproc_mem_map,
> + zynqmp_r5_rproc_mem_unmap,
> + rmem->name);
> + if (IS_ERR_OR_NULL(mem))
> + return -ENOMEM;
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap TCM banks when powering down R5 core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> + struct zynqmp_r5_core *r5_core;
> + int i;
> + enum pm_node_id pm_domain_id;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + pr_err("r5 core is not available\n");
> + return -EINVAL;
> + }
> +
> + iounmap((void __iomem *)mem->va);
> +
> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + if (zynqmp_pm_release_node(pm_domain_id))
> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * tcm_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to initialize the va and da fields of
> + *
> + * Given TCM bank entry, this callback will set device address for R5
> + * running on TCM and also setup virtual address for TCM bank
> + * remoteproc carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + /* Update memory entry va */
> + mem->va = (void *)va;
> +
> + /* clear TCMs */
> + memset_io(va, 0, mem->len);
> +
> + /*
> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> + * while on the Linux side they are at 0xffexxxxx.
> + *
> + * Zero out the high 12 bits of the address. This will give
> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> + */
> + mem->da &= 0x000fffff;
> +
> + /*
> + * TCM Banks 1A and 1B still have to be translated.
> + *
> + * Below handle these two banks' absolute addresses (0xffe90000 and
> + * 0xffeb0000) and convert to the expected relative addresses
> + * (0x0 and 0x20000).
> + */
> + if (mem->da == 0x90000 || mem->da == 0xB0000)
> + mem->da -= 0x90000;
> +
> + /* if translated TCM bank address is not valid report error */
> + if (mem->da != 0x0 && mem->da != 0x20000) {
> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
> + return -EINVAL;
> + }
> + return 0;
> +}
> +
> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
> + char *bank_name;
> + struct device *dev;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + for (i = 0; i < num_banks; i++) {
> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
> + bank_name = r5_core->tcm_banks[i].bank_name;
> + bank_size = r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> +
> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + /* Turn off all TCM banks turned on before */
> + do {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x, %x\n",
> + (u32)pm_domain_id, ret);
> + } while (i--);
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
> + char *bank_name;
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
> + bank_name = r5_core->tcm_banks[0].bank_name;
> + for (i = 0; i < num_banks; i++) {
> + bank_size += r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> + }
> +
> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + for (i = 0; i < num_banks; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x ret: %x\n",
> + (u32)pm_domain_id, ret);
> + }
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> +
> + return 0;
> +}
> +
> +/*
> + * add_tcm_banks()
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Given R5 node in remoteproc instance
> + * allocate remoteproc carveout for TCM memory
> + * needed for firmware to be loaded
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_tcm_banks(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + parent_pdev = to_platform_device(dev->parent);
> + if (!parent_pdev) {
> + dev_err(dev, "parent platform dev unavailable\n");
> + return -ENODEV;
> + }
> +
> + cluster = platform_get_drvdata(parent_pdev);
> + if (!cluster) {
> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
> + return -EINVAL;
> + }
> +
> + if (cluster->mode == SPLIT_MODE)
> + return add_tcm_carveout_split_mode(rproc);
> + else if (cluster->mode == LOCKSTEP_MODE)
> + return add_tcm_carveout_lockstep_mode(rproc);
> +
> + dev_err(cluster->dev, "invalid cluster mode\n");
> + return -EINVAL;
> +}
> +
> +/*
> + * zynqmp_r5_parse_fw()
> + * @rproc: single R5 core's corresponding rproc instance
> + * @fw: ptr to firmware to be loaded onto r5 core
> + *
> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> + int ret;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + dev_err(&rproc->dev, "r5 core not available\n");
> + return -EINVAL;
> + }
> +
> + dev = r5_core->dev;
> +
> + ret = add_tcm_banks(rproc);
> + if (ret) {
> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
> + return ret;
> + }
> +
> + ret = add_mem_regions(rproc);
> + if (ret)
> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
> +
> + ret = rproc_elf_load_rsc_table(rproc, fw);
> + if (ret == -EINVAL) {
> + /*
> + * resource table only required for IPC.
> + * if not present, this is not necessarily an error;
> + * for example, loading r5 hello world application
> + * so simply inform user and keep going.
> + */
> + dev_info(&rproc->dev, "no resource table found.\n");
> + ret = 0;
> + }
> + return ret;
> +}
> +
> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> + .start = zynqmp_r5_rproc_start,
> + .stop = zynqmp_r5_rproc_stop,
> + .load = rproc_elf_load_segments,
> + .parse_fw = zynqmp_r5_parse_fw,
> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> + .sanity_check = rproc_elf_sanity_check,
> + .get_boot_addr = rproc_elf_get_boot_addr,
> +};
> +
> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
> +{
> + int i, j, k;
> + struct zynqmp_r5_core *r5_core;
> +
> + dev_dbg(cluster->dev, "Printing dt node info\n");
> +
> + pr_debug("cluster mode = %d\n", cluster->mode);
> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
> + pr_debug("r5f num cores = %d\n", cluster->core_count);
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("can't get r5_core\n");
> + continue;
> + }
> +
> + pr_debug("r5 core %d nodes\n", i);
> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
> + k, r5_core->tcm_banks[k].addr,
> + r5_core->tcm_banks[k].size,
> + r5_core->tcm_banks[k].pm_domain_id,
> + r5_core->tcm_banks[k].bank_name);
> + }
> +
> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
> +
> + for (j = 0; j < r5_core->res_mem_count; j++) {
> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
> + j, r5_core->res_mem[j].base,
> + r5_core->res_mem[j].size,
> + r5_core->res_mem[j].name);
> + }
> + }
> +}
> +
> +/**
> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
> + * this is called for each individual R5 core to
> + * set up mailbox, Xilinx platform manager unique ID,
> + * add to rproc core
> + *
> + * @r5_core: zynqmp_r5_core r5 core object to initialize
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
> +{
> + int ret;
> + struct rproc *r5_rproc;
> + struct device *dev;
> +
> + dev = r5_core->dev;
> +
> + /* Set up DMA mask */
> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> + if (ret)
> + return ret;
> +
> + /* Allocate remoteproc instance */
> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> + NULL, sizeof(struct zynqmp_r5_core));
> + if (IS_ERR_OR_NULL(r5_rproc))
> + return -ENOMEM;
> +
> + r5_rproc->auto_boot = false;
> + r5_rproc->priv = r5_core;
> +
> + /* Add R5 remoteproc */
> + ret = devm_rproc_add(dev, r5_rproc);
> + if (ret) {
> + pr_err("failed to add r5 remoteproc\n");
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> +{
> + int tcm_bank_count, tcm_node;
> + int i = 0, j;
> + struct zynqmp_r5_core *r5_core;
> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> + struct device *dev = cluster->dev;
> +
> + /* ToDo: Use predefined TCM address space values from driver until
> + * system-dt spec is not final fot TCM
> + */
> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> +
> + /* count per core tcm banks */
> + tcm_bank_count = tcm_bank_count / cluster->core_count;
> +
> + /* r5 core 0 will use all of TCM banks in lockstep mode.
> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> + * 128k. Assign TCM banks to each core accordingly
> + */
> + tcm_node = 0;
> + for (j = 0; j < cluster->core_count; j++) {
> + r5_core = &cluster->r5_cores[j];
> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> + tcm_bank_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> + return -ENOMEM;
> +
> + for (i = 0; i < tcm_bank_count; i++) {
> + /* Use pre-defined TCM reg values.
> + * Eventually this should be replaced by values
> + * parsed from dts.
> + */
> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> + tcm_node++;
> + }
> +
> + r5_core->tcm_bank_count = tcm_bank_count;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> +{
> + int res_mem_count, i;
> + struct device *dev;
> + struct device_node *np, *rmem_np;
> + struct reserved_mem *rmem;
> +
> + dev = r5_core->dev;
> +
> + np = r5_core->np;
> + if (IS_ERR_OR_NULL(np)) {
> + pr_err("invalid device node of r5 core\n");
> + return -EINVAL;
> + }
> +
> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> + sizeof(phandle));
> + if (res_mem_count <= 0) {
> + dev_warn(dev, "failed to get memory-region property %d\n",
> + res_mem_count);
> + return -EINVAL;
> + }
> +
> + r5_core->res_mem = devm_kzalloc(dev,
> + res_mem_count * sizeof(struct reserved_mem),
> + GFP_KERNEL);
> + if (!r5_core->res_mem) {
> + dev_err(dev, "failed to allocate mem region memory\n");
> + return -ENOMEM;
> + }
> +
> + for (i = 0; i < res_mem_count; i++) {
> + rmem_np = of_parse_phandle(np, "memory-region", i);
> + if (!rmem_np)
> + return -EINVAL;
> +
> + rmem = of_reserved_mem_lookup(rmem_np);
> + if (!rmem) {
> + of_node_put(rmem_np);
> + return -EINVAL;
> + }
> +
> + memcpy(&r5_core->res_mem[i], rmem,
> + sizeof(struct reserved_mem));
> + of_node_put(rmem_np);
> + }
> +
> + r5_core->res_mem_count = res_mem_count;
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> +{
> + int ret, i;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev = cluster->dev;
> +
> + ret = zynqmp_r5_get_tcm_node(cluster);
> + if (ret < 0) {
> + dev_err(dev, "can't get tcm node, err %d\n", ret);
> + return ret;
> + }
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("invalid r5 core\n");
> + return -EINVAL;
> + }
> +
> + ret = zynqmp_r5_get_mem_region_node(r5_core);
> + if (ret)
> + dev_warn(dev, "memory-region prop failed %d\n", ret);
> +
> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> + 1, &r5_core->pm_domain_id);
> + if (ret) {
> + dev_err(dev, "failed to get power-domains property\n");
> + return ret;
> + }
> +
> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_add_rproc_core(r5_core);
> + if (ret) {
> + dev_err(dev, "failed to init r5 core %d\n", i);
> + return ret;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> +{
> + struct device *dev = cluster->dev;
> + struct device_node *dev_node = dev_of_node(dev);
> + struct device_node *child;
> + struct platform_device *child_pdev;
> + int core_count = 0, ret, i;
> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> + struct zynqmp_r5_core *r5_cores;
> +
> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> +
> + /* on success returns 0, if not defined then returns -EINVAL,
> + * In that case, default is LOCKSTEP mode
> + */
> + if (ret != -EINVAL && ret != 0) {
> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> + return -EINVAL;
> + }
> +
> + if (cluster_mode == SINGLE_CPU_MODE) {
> + dev_err(dev, "driver does not support single cpu mode\n");
> + return -EINVAL;
> + } else if ((cluster_mode != SPLIT_MODE &&
> + cluster_mode != LOCKSTEP_MODE)) {
> + dev_err(dev, "Invalid cluster mode\n");
> + return -EINVAL;
> + }
> +
> + core_count = of_get_available_child_count(dev_node);
> + if (core_count <= 0) {
> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> + return -EINVAL;
> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> + return -EINVAL;
> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> + dev_warn(dev, "Only r5 core0 will be used\n");
> + core_count = 1;
> + }
> +
> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> + core_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_cores)) {
> + dev_err(dev, "can't allocate memory for cores\n");
> + return -ENOMEM;
> + }
> +
> + i = 0;
> + for_each_available_child_of_node(dev_node, child) {
> + child_pdev = of_find_device_by_node(child);
> + if (!child_pdev)
> + return -ENODEV;
> +
> + r5_cores[i].dev = &child_pdev->dev;
> + if (!r5_cores[i].dev) {
> + pr_err("can't get device for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> + if (!r5_cores[i].np) {
> + pr_err("can't get device node for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + i++;
> + if (i == core_count)
> + break;
> + }
> +
> + cluster->mode = cluster_mode;
> + cluster->core_count = core_count;
> + cluster->r5_cores = r5_cores;
> +
> + ret = zynqmp_r5_core_init(cluster);
> + if (ret < 0) {
> + dev_err(dev, "failed to init r5 core err %d\n", ret);
> + return ret;
> + }
> +
> + zynqmp_r5_print_dt_node_info(cluster);
> +
> + return 0;
> +}
> +
> +static void zynqmp_r5_cluster_exit(void *data)
> +{
> + struct platform_device *pdev = (struct platform_device *)data;
> +
> + platform_set_drvdata(pdev, NULL);
> +
> + pr_info("Exit r5f subsystem driver\n");
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_probe()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * called when driver is probed, for each R5 core specified in DT,
> + * setup as needed to do remoteproc-related operations
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> +{
> + int ret;
> + struct zynqmp_r5_cluster *cluster;
> + struct device *dev = &pdev->dev;
> +
> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> + if (IS_ERR_OR_NULL(cluster))
> + return -ENOMEM;
> +
> + cluster->dev = dev;
> +
> + ret = devm_of_platform_populate(dev);
> + if (ret) {
> + dev_err(dev, "failed to populate platform dev %d\n", ret);
> + return ret;
> + }
> +
> + /* wire in so each core can be cleaned up at driver remove */
> + platform_set_drvdata(pdev, cluster);
> +
> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_cluster_init(cluster);
> + if (ret) {
> + dev_err(dev, "Invalid r5f subsystem device tree\n");
> + return ret;
> + }
> +
> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> + return 0;
> +}
> +
> +/* Match table for OF platform binding */
> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> + { .compatible = "xlnx,zynqmp-r5fss", },
> + { /* end of list */ },
> +};
> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> +
> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> + .probe = zynqmp_r5_remoteproc_probe,
> + .driver = {
> + .name = "zynqmp_r5_remoteproc",
> + .of_match_table = zynqmp_r5_remoteproc_match,
> + },
> +};
> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> +
> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> +MODULE_AUTHOR("Xilinx Inc.");
> +MODULE_LICENSE("GPL v2");
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 2/6] arm64: dts: xilinx: zynqmp: Add RPU subsystem device node
2021-11-29 17:58 ` Mathieu Poirier
@ 2021-11-30 8:01 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-30 8:01 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
Hi Mathieu,
Thanks for reviews. Please find my responses inline.
On 11/29/21 11:28 PM, Mathieu Poirier wrote:
> On Mon, Nov 22, 2021 at 10:20:46PM -0800, Tanmay Shah wrote:
>> RPU subsystem can be configured in cluster-mode or split mode.
>> Also each r5 core has separate power domains.
>>
>> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
>> ---
>> arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 17 +++++++++++++++++
>> 1 file changed, 17 insertions(+)
>>
>> diff --git a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
>> index 28dccb891a53..f4fb98ccb1b5 100644
>> --- a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
>> +++ b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
>> @@ -654,6 +654,23 @@ qspi: spi@ff0f0000 {
>> power-domains = <&zynqmp_firmware PD_QSPI>;
>> };
>>
>> + r5fss: r5fss@ff9a0000 {
>> + compatible = "xlnx,zynqmp-r5fss";
>> + xlnx,cluster-mode = <1>;
>> +
>> + reg = <0x0 0Xff9a0000 0x0 0x228>;
> This line is giving me an obvious checkpatch warning. Please make sure you run
> checkpatch.pl before sending out a patchset.
>
I ran checkpatch.pl on patches that contains .c files and .h files and I
had fixed lot of style related warnings using checkpatch.pl.
However, I had misunderstanding that checkpatch.pl should not be used
for patches with .yaml and .dtsi files.
So I skipped it for bindings patch and dtsi patch. Thanks for correcting
me here. I will make sure to run checkpatch.pl for all patches from
here. Also, I will fix this error in next revision.
>> +
>> + r5f_core0: r5f-0 {
>> + compatible = "xlnx,zynqmp-r5f";
>> + power-domains = <&zynqmp_firmware PD_RPU_0>;
>> + };
>> +
>> + r5f_core1: r5f-1 {
>> + compatible = "xlnx,zynqmp-r5f";
>> + power-domains = <&zynqmp_firmware PD_RPU_1>;
>> + };
>> + };
>> +
>> psgtr: phy@fd400000 {
>> compatible = "xlnx,zynqmp-psgtr-v1.1";
>> status = "disabled";
>> --
>> 2.25.1
>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 2/6] arm64: dts: xilinx: zynqmp: Add RPU subsystem device node
@ 2021-11-30 8:01 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-30 8:01 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
Hi Mathieu,
Thanks for reviews. Please find my responses inline.
On 11/29/21 11:28 PM, Mathieu Poirier wrote:
> On Mon, Nov 22, 2021 at 10:20:46PM -0800, Tanmay Shah wrote:
>> RPU subsystem can be configured in cluster-mode or split mode.
>> Also each r5 core has separate power domains.
>>
>> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
>> ---
>> arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 17 +++++++++++++++++
>> 1 file changed, 17 insertions(+)
>>
>> diff --git a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
>> index 28dccb891a53..f4fb98ccb1b5 100644
>> --- a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
>> +++ b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
>> @@ -654,6 +654,23 @@ qspi: spi@ff0f0000 {
>> power-domains = <&zynqmp_firmware PD_QSPI>;
>> };
>>
>> + r5fss: r5fss@ff9a0000 {
>> + compatible = "xlnx,zynqmp-r5fss";
>> + xlnx,cluster-mode = <1>;
>> +
>> + reg = <0x0 0Xff9a0000 0x0 0x228>;
> This line is giving me an obvious checkpatch warning. Please make sure you run
> checkpatch.pl before sending out a patchset.
>
I ran checkpatch.pl on patches that contains .c files and .h files and I
had fixed lot of style related warnings using checkpatch.pl.
However, I had misunderstanding that checkpatch.pl should not be used
for patches with .yaml and .dtsi files.
So I skipped it for bindings patch and dtsi patch. Thanks for correcting
me here. I will make sure to run checkpatch.pl for all patches from
here. Also, I will fix this error in next revision.
>> +
>> + r5f_core0: r5f-0 {
>> + compatible = "xlnx,zynqmp-r5f";
>> + power-domains = <&zynqmp_firmware PD_RPU_0>;
>> + };
>> +
>> + r5f_core1: r5f-1 {
>> + compatible = "xlnx,zynqmp-r5f";
>> + power-domains = <&zynqmp_firmware PD_RPU_1>;
>> + };
>> + };
>> +
>> psgtr: phy@fd400000 {
>> compatible = "xlnx,zynqmp-psgtr-v1.1";
>> status = "disabled";
>> --
>> 2.25.1
>>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-11-29 18:42 ` Mathieu Poirier
@ 2021-11-30 8:01 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-30 8:01 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On 11/30/21 12:12 AM, Mathieu Poirier wrote:
> On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
>> This driver enables r5f dual core Real time Processing Unit subsystem
>> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
>> (cluster) can be configured in different modes e.g. split mode in which
>> two r5f cores work independent of each other and lock-step mode in which
>> both r5f cores execute same code clock-for-clock and notify if the
>> result is different.
>>
>> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
>> Platform Management Unit that handles the R5 configuration, memory access
>> and R5 lifecycle management. The interface to this manager is done in this
>> driver via zynqmp_pm_* function calls.
>>
>> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
>> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
>> ---
>> drivers/remoteproc/Kconfig | 12 +
>> drivers/remoteproc/Makefile | 1 +
>> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
>> 3 files changed, 972 insertions(+)
>> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
> ...and this patch gives me complation warnings:
>
> CC drivers/remoteproc/xlnx_r5_remoteproc.o
> kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c: In function ‘add_tcm_carveout_lockstep_mode’:
> kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:412:28: warning: unused variable ‘cluster’ [-Wunused-variable]
> 412 | struct zynqmp_r5_cluster *cluster;
> | ^~~~~~~
> kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:411:26: warning: unused variable ‘parent_pdev’ [-Wunused-variable]
> 411 | struct platform_device *parent_pdev;
> | ^~~~~~~~~~~
>
> The above leads me to believe this patchset was not compiled before it was sent
> out.
Please don't assume that this patch-set was not compiled. This driver
was compiled and tested on Xilinx QEMU and zynqmp platform for its
functionality.
This driver went through multiple internal reviews and I had to
re-architecture it multiple times.
I simply missed to fix above warnings before submitting driver. I have
compiled driver with following command:
//make ARCH=arm64 W=1 C=1 CROSS_COMPILE="aarch64-linux-gnu-" -j32 -Rr
O=$zynqmp_kernel_build/rproc-next
I did my best to make sure driver stays warning free, however few
warnings were still missed. That is not intentional and by mistake. I
fully intend to comply with Linux Kernel community guideline and
checklist before submitting patches.
Apart from above warnings, two more warnings are there in v2 i.e. due to
typecast issue between (void __iomem *) and (void *). But, I had not
solution before. Also they were used before in different driver so, I
chose to use them anyway.
I just found that using memremap set of functions will fix them. I will
use those functions in next patch set instead of ioremap_wc as explained
in this article: https://lwn.net/Articles/653585/
> Being new to this I can understand that checkpatch.pl was omitted (albeit amply
> documented) but obvious compilation warnings can't be excused. As such I
> am dropping this set and will not review another version until January.
I understand having warnings in driver may lead to frustration and may
cause trust issues for the rest of the code and it can't be excused at all.
I am aware of guidelines of sending patches upstream
(https://www.kernel.org/doc/html/latest/process/submitting-patches.html)
and fully intend to comply with that and like I said, if I miss
something it is not by intention but human error.
I constantly try to improve processes to upstream patches so we don't
face above type of issues.
I did run checkpatch.pl as per best of my knowledge and fixed lot of
style related warnings reported by it. Also I enabled W=1 option in my
compilation command and fixed lots of warnings reported by compiler too.
As I explained earlier, I wasn't aware of unused-variable warnings, and
so I missed to fix them.
With this, I request to review driver from functionality point of view
as well along with style errors. So, I can address more number of
comments / concerns in less number of patches. If you want these
warnings to be fixed, I will send v3 and you can put more comments on
v3. However, I highly appreciate if we can continue reviews and not
postpone till January. Please let me know your thoughts.
Thanks,
Tanmay
> Mathieu
>> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
>> index f30d00a3aabe..27f66910d8d3 100644
>> --- a/drivers/remoteproc/Kconfig
>> +++ b/drivers/remoteproc/Kconfig
>> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
>> It's safe to say N here if you're not interested in utilizing
>> a slave processor.
>>
>> +config XLNX_R5_REMOTEPROC
>> + tristate "Xilinx R5 remoteproc support"
>> + depends on PM && ARCH_ZYNQMP
>> + depends on ZYNQMP_FIRMWARE
>> + select RPMSG_VIRTIO
>> + select ZYNQMP_IPI_MBOX
>> + help
>> + Say y or m here to support Xilinx R5 remote processors via the remote
>> + processor framework.
>> +
>> + It's safe to say N if not interested in using RPU r5f cores.
>> +
>> endif # REMOTEPROC
>>
>> endmenu
>> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
>> index bb26c9e4ef9c..334a8bed4c14 100644
>> --- a/drivers/remoteproc/Makefile
>> +++ b/drivers/remoteproc/Makefile
>> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
>> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
>> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
>> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
>> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
>> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> new file mode 100644
>> index 000000000000..c2167fd3869d
>> --- /dev/null
>> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> @@ -0,0 +1,959 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ZynqMP R5 Remote Processor driver
>> + *
>> + */
>> +
>> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
>> +#include <linux/firmware/xlnx-zynqmp.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/kernel.h>
>> +#include <linux/list.h>
>> +#include <linux/mailbox_client.h>
>> +#include <linux/mailbox/zynqmp-ipi-message.h>
>> +#include <linux/module.h>
>> +#include <linux/of_address.h>
>> +#include <linux/of_platform.h>
>> +#include <linux/of_reserved_mem.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/remoteproc.h>
>> +#include <linux/skbuff.h>
>> +#include <linux/sysfs.h>
>> +
>> +#include "remoteproc_internal.h"
>> +
>> +/* settings for RPU cluster mode */
>> +enum zynqmp_r5_cluster_mode {
>> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
>> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
>> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
>> +};
>> +
>> +/**
>> + * struct mem_bank_data - Memory Bank description
>> + *
>> + * @addr: Start address of memory bank
>> + * @size: Size of Memory bank
>> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
>> + * @bank_name: name of the bank for remoteproc framework
>> + */
>> +struct mem_bank_data {
>> + phys_addr_t addr;
>> + size_t size;
>> + enum pm_node_id pm_domain_id;
>> + char *bank_name;
>> +};
>> +
>> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
>> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
>> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
>> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
>> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
>> +};
>> +
>> +/**
>> + * struct zynqmp_r5_core - ZynqMP R5 core structure
>> + *
>> + * @dev: device of RPU instance
>> + * @np: device node of RPU instance
>> + * @tcm_bank_count: number TCM banks accessible to this RPU
>> + * @tcm_banks: array of each TCM bank data
>> + * @res_mem_count: number of Reserved Memory regions per core
>> + * @res_mem: array of reserved memory regions
>> + * @rproc: rproc handle
>> + * @pm_domain_id: RPU CPU power domain id
>> + */
>> +struct zynqmp_r5_core {
>> + struct device *dev;
>> + struct device_node *np;
>> + int tcm_bank_count;
>> + struct mem_bank_data *tcm_banks;
>> + int res_mem_count;
>> + struct reserved_mem *res_mem;
>> + struct rproc *rproc;
>> + enum pm_node_id pm_domain_id;
>> +};
>> +
>> +/**
>> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
>> + *
>> + * @dev: r5f subsystem cluster device node
>> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
>> + * @core_count: number of r5 cores used for this cluster mode
>> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
>> + */
>> +struct zynqmp_r5_cluster {
>> + struct device *dev;
>> + enum zynqmp_r5_cluster_mode mode;
>> + int core_count;
>> + struct zynqmp_r5_core *r5_cores;
>> +};
>> +
>> +/*
>> + * zynqmp_r5_set_mode - set RPU operation mode
>> + *
>> + * set RPU operation mode
>> + *
>> + * Return: 0 for success, negative value for failure
>> + */
>> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
>> + enum zynqmp_r5_cluster_mode rpu_mode)
>> +{
>> + enum rpu_tcm_comb tcm_mode;
>> + int ret, reg_val;
>> +
>> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
>> +
>> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
>> + if (ret < 0) {
>> + pr_err("failed to set RPU mode\n");
>> + return ret;
>> + }
>> +
>> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
>> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
>> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
>> + if (ret < 0)
>> + pr_err("failed to configure TCM\n");
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_start
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Start R5 Core from designated boot address.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + enum rpu_boot_mem bootmem;
>> + int ret;
>> +
>> + if (!r5_core) {
>> + pr_err("can't get r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
>> +
>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>> +
>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>> + if (ret)
>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_stop
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Power down R5 Core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + int ret;
>> +
>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret)
>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap memory-region carveout
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + iounmap((void __iomem *)mem->va);
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_mem_regions
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Construct rproc mem carveouts from carveout provided in
>> + * memory-region property
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_mem_regions(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct rproc_mem_entry *mem;
>> + struct reserved_mem *rmem;
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> +
>> + r5_core = rproc->priv;
>> + dev = r5_core->dev;
>> +
>> + /* Register associated reserved memory regions */
>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>> + rmem = &r5_core->res_mem[i];
>> + mem = rproc_mem_entry_init(dev, NULL,
>> + (dma_addr_t)rmem->base,
>> + rmem->size, rmem->base,
>> + zynqmp_r5_rproc_mem_map,
>> + zynqmp_r5_rproc_mem_unmap,
>> + rmem->name);
>> + if (IS_ERR_OR_NULL(mem))
>> + return -ENOMEM;
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap TCM banks when powering down R5 core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>> +{
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> + enum pm_node_id pm_domain_id;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + pr_err("r5 core is not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + iounmap((void __iomem *)mem->va);
>> +
>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + if (zynqmp_pm_release_node(pm_domain_id))
>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * tcm_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to initialize the va and da fields of
>> + *
>> + * Given TCM bank entry, this callback will set device address for R5
>> + * running on TCM and also setup virtual address for TCM bank
>> + * remoteproc carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + /* Update memory entry va */
>> + mem->va = (void *)va;
>> +
>> + /* clear TCMs */
>> + memset_io(va, 0, mem->len);
>> +
>> + /*
>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>> + * while on the Linux side they are at 0xffexxxxx.
>> + *
>> + * Zero out the high 12 bits of the address. This will give
>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>> + */
>> + mem->da &= 0x000fffff;
>> +
>> + /*
>> + * TCM Banks 1A and 1B still have to be translated.
>> + *
>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>> + * 0xffeb0000) and convert to the expected relative addresses
>> + * (0x0 and 0x20000).
>> + */
>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>> + mem->da -= 0x90000;
>> +
>> + /* if translated TCM bank address is not valid report error */
>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>> + return -EINVAL;
>> + }
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; i < num_banks; i++) {
>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>> + bank_name = r5_core->tcm_banks[i].bank_name;
>> + bank_size = r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + /* Turn off all TCM banks turned on before */
>> + do {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x, %x\n",
>> + (u32)pm_domain_id, ret);
>> + } while (i--);
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>> + bank_name = r5_core->tcm_banks[0].bank_name;
>> + for (i = 0; i < num_banks; i++) {
>> + bank_size += r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> + }
>> +
>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + for (i = 0; i < num_banks; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x ret: %x\n",
>> + (u32)pm_domain_id, ret);
>> + }
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_tcm_banks()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Given R5 node in remoteproc instance
>> + * allocate remoteproc carveout for TCM memory
>> + * needed for firmware to be loaded
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_parse_fw()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @fw: ptr to firmware to be loaded onto r5 core
>> + *
>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>> +{
>> + int ret;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + dev_err(&rproc->dev, "r5 core not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + dev = r5_core->dev;
>> +
>> + ret = add_tcm_banks(rproc);
>> + if (ret) {
>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = add_mem_regions(rproc);
>> + if (ret)
>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>> +
>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>> + if (ret == -EINVAL) {
>> + /*
>> + * resource table only required for IPC.
>> + * if not present, this is not necessarily an error;
>> + * for example, loading r5 hello world application
>> + * so simply inform user and keep going.
>> + */
>> + dev_info(&rproc->dev, "no resource table found.\n");
>> + ret = 0;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> +
>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int i, j, k;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + dev_dbg(cluster->dev, "Printing dt node info\n");
>> +
>> + pr_debug("cluster mode = %d\n", cluster->mode);
>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("can't get r5_core\n");
>> + continue;
>> + }
>> +
>> + pr_debug("r5 core %d nodes\n", i);
>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>> + k, r5_core->tcm_banks[k].addr,
>> + r5_core->tcm_banks[k].size,
>> + r5_core->tcm_banks[k].pm_domain_id,
>> + r5_core->tcm_banks[k].bank_name);
>> + }
>> +
>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>> +
>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>> + j, r5_core->res_mem[j].base,
>> + r5_core->res_mem[j].size,
>> + r5_core->res_mem[j].name);
>> + }
>> + }
>> +}
>> +
>> +/**
>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>> + * this is called for each individual R5 core to
>> + * set up mailbox, Xilinx platform manager unique ID,
>> + * add to rproc core
>> + *
>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>> +{
>> + int ret;
>> + struct rproc *r5_rproc;
>> + struct device *dev;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* Set up DMA mask */
>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>> + if (ret)
>> + return ret;
>> +
>> + /* Allocate remoteproc instance */
>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>> + NULL, sizeof(struct zynqmp_r5_core));
>> + if (IS_ERR_OR_NULL(r5_rproc))
>> + return -ENOMEM;
>> +
>> + r5_rproc->auto_boot = false;
>> + r5_rproc->priv = r5_core;
>> +
>> + /* Add R5 remoteproc */
>> + ret = devm_rproc_add(dev, r5_rproc);
>> + if (ret) {
>> + pr_err("failed to add r5 remoteproc\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int tcm_bank_count, tcm_node;
>> + int i = 0, j;
>> + struct zynqmp_r5_core *r5_core;
>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>> + struct device *dev = cluster->dev;
>> +
>> + /* ToDo: Use predefined TCM address space values from driver until
>> + * system-dt spec is not final fot TCM
>> + */
>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>> +
>> + /* count per core tcm banks */
>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>> +
>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>> + * 128k. Assign TCM banks to each core accordingly
>> + */
>> + tcm_node = 0;
>> + for (j = 0; j < cluster->core_count; j++) {
>> + r5_core = &cluster->r5_cores[j];
>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>> + tcm_bank_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>> + return -ENOMEM;
>> +
>> + for (i = 0; i < tcm_bank_count; i++) {
>> + /* Use pre-defined TCM reg values.
>> + * Eventually this should be replaced by values
>> + * parsed from dts.
>> + */
>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>> + tcm_node++;
>> + }
>> +
>> + r5_core->tcm_bank_count = tcm_bank_count;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>> +{
>> + int res_mem_count, i;
>> + struct device *dev;
>> + struct device_node *np, *rmem_np;
>> + struct reserved_mem *rmem;
>> +
>> + dev = r5_core->dev;
>> +
>> + np = r5_core->np;
>> + if (IS_ERR_OR_NULL(np)) {
>> + pr_err("invalid device node of r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>> + sizeof(phandle));
>> + if (res_mem_count <= 0) {
>> + dev_warn(dev, "failed to get memory-region property %d\n",
>> + res_mem_count);
>> + return -EINVAL;
>> + }
>> +
>> + r5_core->res_mem = devm_kzalloc(dev,
>> + res_mem_count * sizeof(struct reserved_mem),
>> + GFP_KERNEL);
>> + if (!r5_core->res_mem) {
>> + dev_err(dev, "failed to allocate mem region memory\n");
>> + return -ENOMEM;
>> + }
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int ret, i;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev = cluster->dev;
>> +
>> + ret = zynqmp_r5_get_tcm_node(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("invalid r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>> + if (ret)
>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>> +
>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>> + 1, &r5_core->pm_domain_id);
>> + if (ret) {
>> + dev_err(dev, "failed to get power-domains property\n");
>> + return ret;
>> + }
>> +
>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>> + if (ret) {
>> + dev_err(dev, "failed to init r5 core %d\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + struct device *dev = cluster->dev;
>> + struct device_node *dev_node = dev_of_node(dev);
>> + struct device_node *child;
>> + struct platform_device *child_pdev;
>> + int core_count = 0, ret, i;
>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>> + struct zynqmp_r5_core *r5_cores;
>> +
>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>> +
>> + /* on success returns 0, if not defined then returns -EINVAL,
>> + * In that case, default is LOCKSTEP mode
>> + */
>> + if (ret != -EINVAL && ret != 0) {
>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster_mode == SINGLE_CPU_MODE) {
>> + dev_err(dev, "driver does not support single cpu mode\n");
>> + return -EINVAL;
>> + } else if ((cluster_mode != SPLIT_MODE &&
>> + cluster_mode != LOCKSTEP_MODE)) {
>> + dev_err(dev, "Invalid cluster mode\n");
>> + return -EINVAL;
>> + }
>> +
>> + core_count = of_get_available_child_count(dev_node);
>> + if (core_count <= 0) {
>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>> + return -EINVAL;
>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>> + return -EINVAL;
>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>> + dev_warn(dev, "Only r5 core0 will be used\n");
>> + core_count = 1;
>> + }
>> +
>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>> + core_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_cores)) {
>> + dev_err(dev, "can't allocate memory for cores\n");
>> + return -ENOMEM;
>> + }
>> +
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
>> + return -ENODEV;
>> +
>> + r5_cores[i].dev = &child_pdev->dev;
>> + if (!r5_cores[i].dev) {
>> + pr_err("can't get device for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>> + if (!r5_cores[i].np) {
>> + pr_err("can't get device node for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + i++;
>> + if (i == core_count)
>> + break;
>> + }
>> +
>> + cluster->mode = cluster_mode;
>> + cluster->core_count = core_count;
>> + cluster->r5_cores = r5_cores;
>> +
>> + ret = zynqmp_r5_core_init(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + zynqmp_r5_print_dt_node_info(cluster);
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
>> +
>> + pr_info("Exit r5f subsystem driver\n");
>> +}
>> +
>> +/*
>> + * zynqmp_r5_remoteproc_probe()
>> + *
>> + * @pdev: domain platform device for R5 cluster
>> + *
>> + * called when driver is probed, for each R5 core specified in DT,
>> + * setup as needed to do remoteproc-related operations
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>> +{
>> + int ret;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct device *dev = &pdev->dev;
>> +
>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(cluster))
>> + return -ENOMEM;
>> +
>> + cluster->dev = dev;
>> +
>> + ret = devm_of_platform_populate(dev);
>> + if (ret) {
>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /* wire in so each core can be cleaned up at driver remove */
>> + platform_set_drvdata(pdev, cluster);
>> +
>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_cluster_init(cluster);
>> + if (ret) {
>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>> + return ret;
>> + }
>> +
>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
>> + return 0;
>> +}
>> +
>> +/* Match table for OF platform binding */
>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>> + { .compatible = "xlnx,zynqmp-r5fss", },
>> + { /* end of list */ },
>> +};
>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>> +
>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>> + .probe = zynqmp_r5_remoteproc_probe,
>> + .driver = {
>> + .name = "zynqmp_r5_remoteproc",
>> + .of_match_table = zynqmp_r5_remoteproc_match,
>> + },
>> +};
>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>> +
>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>> +MODULE_AUTHOR("Xilinx Inc.");
>> +MODULE_LICENSE("GPL v2");
>> --
>> 2.25.1
>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-11-30 8:01 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-11-30 8:01 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On 11/30/21 12:12 AM, Mathieu Poirier wrote:
> On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
>> This driver enables r5f dual core Real time Processing Unit subsystem
>> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
>> (cluster) can be configured in different modes e.g. split mode in which
>> two r5f cores work independent of each other and lock-step mode in which
>> both r5f cores execute same code clock-for-clock and notify if the
>> result is different.
>>
>> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
>> Platform Management Unit that handles the R5 configuration, memory access
>> and R5 lifecycle management. The interface to this manager is done in this
>> driver via zynqmp_pm_* function calls.
>>
>> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
>> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
>> ---
>> drivers/remoteproc/Kconfig | 12 +
>> drivers/remoteproc/Makefile | 1 +
>> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
>> 3 files changed, 972 insertions(+)
>> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
> ...and this patch gives me complation warnings:
>
> CC drivers/remoteproc/xlnx_r5_remoteproc.o
> kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c: In function ‘add_tcm_carveout_lockstep_mode’:
> kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:412:28: warning: unused variable ‘cluster’ [-Wunused-variable]
> 412 | struct zynqmp_r5_cluster *cluster;
> | ^~~~~~~
> kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:411:26: warning: unused variable ‘parent_pdev’ [-Wunused-variable]
> 411 | struct platform_device *parent_pdev;
> | ^~~~~~~~~~~
>
> The above leads me to believe this patchset was not compiled before it was sent
> out.
Please don't assume that this patch-set was not compiled. This driver
was compiled and tested on Xilinx QEMU and zynqmp platform for its
functionality.
This driver went through multiple internal reviews and I had to
re-architecture it multiple times.
I simply missed to fix above warnings before submitting driver. I have
compiled driver with following command:
//make ARCH=arm64 W=1 C=1 CROSS_COMPILE="aarch64-linux-gnu-" -j32 -Rr
O=$zynqmp_kernel_build/rproc-next
I did my best to make sure driver stays warning free, however few
warnings were still missed. That is not intentional and by mistake. I
fully intend to comply with Linux Kernel community guideline and
checklist before submitting patches.
Apart from above warnings, two more warnings are there in v2 i.e. due to
typecast issue between (void __iomem *) and (void *). But, I had not
solution before. Also they were used before in different driver so, I
chose to use them anyway.
I just found that using memremap set of functions will fix them. I will
use those functions in next patch set instead of ioremap_wc as explained
in this article: https://lwn.net/Articles/653585/
> Being new to this I can understand that checkpatch.pl was omitted (albeit amply
> documented) but obvious compilation warnings can't be excused. As such I
> am dropping this set and will not review another version until January.
I understand having warnings in driver may lead to frustration and may
cause trust issues for the rest of the code and it can't be excused at all.
I am aware of guidelines of sending patches upstream
(https://www.kernel.org/doc/html/latest/process/submitting-patches.html)
and fully intend to comply with that and like I said, if I miss
something it is not by intention but human error.
I constantly try to improve processes to upstream patches so we don't
face above type of issues.
I did run checkpatch.pl as per best of my knowledge and fixed lot of
style related warnings reported by it. Also I enabled W=1 option in my
compilation command and fixed lots of warnings reported by compiler too.
As I explained earlier, I wasn't aware of unused-variable warnings, and
so I missed to fix them.
With this, I request to review driver from functionality point of view
as well along with style errors. So, I can address more number of
comments / concerns in less number of patches. If you want these
warnings to be fixed, I will send v3 and you can put more comments on
v3. However, I highly appreciate if we can continue reviews and not
postpone till January. Please let me know your thoughts.
Thanks,
Tanmay
> Mathieu
>> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
>> index f30d00a3aabe..27f66910d8d3 100644
>> --- a/drivers/remoteproc/Kconfig
>> +++ b/drivers/remoteproc/Kconfig
>> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
>> It's safe to say N here if you're not interested in utilizing
>> a slave processor.
>>
>> +config XLNX_R5_REMOTEPROC
>> + tristate "Xilinx R5 remoteproc support"
>> + depends on PM && ARCH_ZYNQMP
>> + depends on ZYNQMP_FIRMWARE
>> + select RPMSG_VIRTIO
>> + select ZYNQMP_IPI_MBOX
>> + help
>> + Say y or m here to support Xilinx R5 remote processors via the remote
>> + processor framework.
>> +
>> + It's safe to say N if not interested in using RPU r5f cores.
>> +
>> endif # REMOTEPROC
>>
>> endmenu
>> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
>> index bb26c9e4ef9c..334a8bed4c14 100644
>> --- a/drivers/remoteproc/Makefile
>> +++ b/drivers/remoteproc/Makefile
>> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
>> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
>> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
>> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
>> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
>> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> new file mode 100644
>> index 000000000000..c2167fd3869d
>> --- /dev/null
>> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> @@ -0,0 +1,959 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ZynqMP R5 Remote Processor driver
>> + *
>> + */
>> +
>> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
>> +#include <linux/firmware/xlnx-zynqmp.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/kernel.h>
>> +#include <linux/list.h>
>> +#include <linux/mailbox_client.h>
>> +#include <linux/mailbox/zynqmp-ipi-message.h>
>> +#include <linux/module.h>
>> +#include <linux/of_address.h>
>> +#include <linux/of_platform.h>
>> +#include <linux/of_reserved_mem.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/remoteproc.h>
>> +#include <linux/skbuff.h>
>> +#include <linux/sysfs.h>
>> +
>> +#include "remoteproc_internal.h"
>> +
>> +/* settings for RPU cluster mode */
>> +enum zynqmp_r5_cluster_mode {
>> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
>> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
>> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
>> +};
>> +
>> +/**
>> + * struct mem_bank_data - Memory Bank description
>> + *
>> + * @addr: Start address of memory bank
>> + * @size: Size of Memory bank
>> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
>> + * @bank_name: name of the bank for remoteproc framework
>> + */
>> +struct mem_bank_data {
>> + phys_addr_t addr;
>> + size_t size;
>> + enum pm_node_id pm_domain_id;
>> + char *bank_name;
>> +};
>> +
>> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
>> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
>> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
>> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
>> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
>> +};
>> +
>> +/**
>> + * struct zynqmp_r5_core - ZynqMP R5 core structure
>> + *
>> + * @dev: device of RPU instance
>> + * @np: device node of RPU instance
>> + * @tcm_bank_count: number TCM banks accessible to this RPU
>> + * @tcm_banks: array of each TCM bank data
>> + * @res_mem_count: number of Reserved Memory regions per core
>> + * @res_mem: array of reserved memory regions
>> + * @rproc: rproc handle
>> + * @pm_domain_id: RPU CPU power domain id
>> + */
>> +struct zynqmp_r5_core {
>> + struct device *dev;
>> + struct device_node *np;
>> + int tcm_bank_count;
>> + struct mem_bank_data *tcm_banks;
>> + int res_mem_count;
>> + struct reserved_mem *res_mem;
>> + struct rproc *rproc;
>> + enum pm_node_id pm_domain_id;
>> +};
>> +
>> +/**
>> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
>> + *
>> + * @dev: r5f subsystem cluster device node
>> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
>> + * @core_count: number of r5 cores used for this cluster mode
>> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
>> + */
>> +struct zynqmp_r5_cluster {
>> + struct device *dev;
>> + enum zynqmp_r5_cluster_mode mode;
>> + int core_count;
>> + struct zynqmp_r5_core *r5_cores;
>> +};
>> +
>> +/*
>> + * zynqmp_r5_set_mode - set RPU operation mode
>> + *
>> + * set RPU operation mode
>> + *
>> + * Return: 0 for success, negative value for failure
>> + */
>> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
>> + enum zynqmp_r5_cluster_mode rpu_mode)
>> +{
>> + enum rpu_tcm_comb tcm_mode;
>> + int ret, reg_val;
>> +
>> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
>> +
>> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
>> + if (ret < 0) {
>> + pr_err("failed to set RPU mode\n");
>> + return ret;
>> + }
>> +
>> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
>> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
>> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
>> + if (ret < 0)
>> + pr_err("failed to configure TCM\n");
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_start
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Start R5 Core from designated boot address.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + enum rpu_boot_mem bootmem;
>> + int ret;
>> +
>> + if (!r5_core) {
>> + pr_err("can't get r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
>> +
>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>> +
>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>> + if (ret)
>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_stop
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Power down R5 Core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + int ret;
>> +
>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret)
>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap memory-region carveout
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + iounmap((void __iomem *)mem->va);
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_mem_regions
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Construct rproc mem carveouts from carveout provided in
>> + * memory-region property
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_mem_regions(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct rproc_mem_entry *mem;
>> + struct reserved_mem *rmem;
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> +
>> + r5_core = rproc->priv;
>> + dev = r5_core->dev;
>> +
>> + /* Register associated reserved memory regions */
>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>> + rmem = &r5_core->res_mem[i];
>> + mem = rproc_mem_entry_init(dev, NULL,
>> + (dma_addr_t)rmem->base,
>> + rmem->size, rmem->base,
>> + zynqmp_r5_rproc_mem_map,
>> + zynqmp_r5_rproc_mem_unmap,
>> + rmem->name);
>> + if (IS_ERR_OR_NULL(mem))
>> + return -ENOMEM;
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap TCM banks when powering down R5 core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>> +{
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> + enum pm_node_id pm_domain_id;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + pr_err("r5 core is not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + iounmap((void __iomem *)mem->va);
>> +
>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + if (zynqmp_pm_release_node(pm_domain_id))
>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * tcm_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to initialize the va and da fields of
>> + *
>> + * Given TCM bank entry, this callback will set device address for R5
>> + * running on TCM and also setup virtual address for TCM bank
>> + * remoteproc carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + /* Update memory entry va */
>> + mem->va = (void *)va;
>> +
>> + /* clear TCMs */
>> + memset_io(va, 0, mem->len);
>> +
>> + /*
>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>> + * while on the Linux side they are at 0xffexxxxx.
>> + *
>> + * Zero out the high 12 bits of the address. This will give
>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>> + */
>> + mem->da &= 0x000fffff;
>> +
>> + /*
>> + * TCM Banks 1A and 1B still have to be translated.
>> + *
>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>> + * 0xffeb0000) and convert to the expected relative addresses
>> + * (0x0 and 0x20000).
>> + */
>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>> + mem->da -= 0x90000;
>> +
>> + /* if translated TCM bank address is not valid report error */
>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>> + return -EINVAL;
>> + }
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; i < num_banks; i++) {
>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>> + bank_name = r5_core->tcm_banks[i].bank_name;
>> + bank_size = r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + /* Turn off all TCM banks turned on before */
>> + do {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x, %x\n",
>> + (u32)pm_domain_id, ret);
>> + } while (i--);
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>> + bank_name = r5_core->tcm_banks[0].bank_name;
>> + for (i = 0; i < num_banks; i++) {
>> + bank_size += r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> + }
>> +
>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + for (i = 0; i < num_banks; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x ret: %x\n",
>> + (u32)pm_domain_id, ret);
>> + }
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_tcm_banks()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Given R5 node in remoteproc instance
>> + * allocate remoteproc carveout for TCM memory
>> + * needed for firmware to be loaded
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_parse_fw()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @fw: ptr to firmware to be loaded onto r5 core
>> + *
>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>> +{
>> + int ret;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + dev_err(&rproc->dev, "r5 core not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + dev = r5_core->dev;
>> +
>> + ret = add_tcm_banks(rproc);
>> + if (ret) {
>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = add_mem_regions(rproc);
>> + if (ret)
>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>> +
>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>> + if (ret == -EINVAL) {
>> + /*
>> + * resource table only required for IPC.
>> + * if not present, this is not necessarily an error;
>> + * for example, loading r5 hello world application
>> + * so simply inform user and keep going.
>> + */
>> + dev_info(&rproc->dev, "no resource table found.\n");
>> + ret = 0;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> +
>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int i, j, k;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + dev_dbg(cluster->dev, "Printing dt node info\n");
>> +
>> + pr_debug("cluster mode = %d\n", cluster->mode);
>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("can't get r5_core\n");
>> + continue;
>> + }
>> +
>> + pr_debug("r5 core %d nodes\n", i);
>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>> + k, r5_core->tcm_banks[k].addr,
>> + r5_core->tcm_banks[k].size,
>> + r5_core->tcm_banks[k].pm_domain_id,
>> + r5_core->tcm_banks[k].bank_name);
>> + }
>> +
>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>> +
>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>> + j, r5_core->res_mem[j].base,
>> + r5_core->res_mem[j].size,
>> + r5_core->res_mem[j].name);
>> + }
>> + }
>> +}
>> +
>> +/**
>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>> + * this is called for each individual R5 core to
>> + * set up mailbox, Xilinx platform manager unique ID,
>> + * add to rproc core
>> + *
>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>> +{
>> + int ret;
>> + struct rproc *r5_rproc;
>> + struct device *dev;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* Set up DMA mask */
>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>> + if (ret)
>> + return ret;
>> +
>> + /* Allocate remoteproc instance */
>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>> + NULL, sizeof(struct zynqmp_r5_core));
>> + if (IS_ERR_OR_NULL(r5_rproc))
>> + return -ENOMEM;
>> +
>> + r5_rproc->auto_boot = false;
>> + r5_rproc->priv = r5_core;
>> +
>> + /* Add R5 remoteproc */
>> + ret = devm_rproc_add(dev, r5_rproc);
>> + if (ret) {
>> + pr_err("failed to add r5 remoteproc\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int tcm_bank_count, tcm_node;
>> + int i = 0, j;
>> + struct zynqmp_r5_core *r5_core;
>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>> + struct device *dev = cluster->dev;
>> +
>> + /* ToDo: Use predefined TCM address space values from driver until
>> + * system-dt spec is not final fot TCM
>> + */
>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>> +
>> + /* count per core tcm banks */
>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>> +
>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>> + * 128k. Assign TCM banks to each core accordingly
>> + */
>> + tcm_node = 0;
>> + for (j = 0; j < cluster->core_count; j++) {
>> + r5_core = &cluster->r5_cores[j];
>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>> + tcm_bank_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>> + return -ENOMEM;
>> +
>> + for (i = 0; i < tcm_bank_count; i++) {
>> + /* Use pre-defined TCM reg values.
>> + * Eventually this should be replaced by values
>> + * parsed from dts.
>> + */
>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>> + tcm_node++;
>> + }
>> +
>> + r5_core->tcm_bank_count = tcm_bank_count;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>> +{
>> + int res_mem_count, i;
>> + struct device *dev;
>> + struct device_node *np, *rmem_np;
>> + struct reserved_mem *rmem;
>> +
>> + dev = r5_core->dev;
>> +
>> + np = r5_core->np;
>> + if (IS_ERR_OR_NULL(np)) {
>> + pr_err("invalid device node of r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>> + sizeof(phandle));
>> + if (res_mem_count <= 0) {
>> + dev_warn(dev, "failed to get memory-region property %d\n",
>> + res_mem_count);
>> + return -EINVAL;
>> + }
>> +
>> + r5_core->res_mem = devm_kzalloc(dev,
>> + res_mem_count * sizeof(struct reserved_mem),
>> + GFP_KERNEL);
>> + if (!r5_core->res_mem) {
>> + dev_err(dev, "failed to allocate mem region memory\n");
>> + return -ENOMEM;
>> + }
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int ret, i;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev = cluster->dev;
>> +
>> + ret = zynqmp_r5_get_tcm_node(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("invalid r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>> + if (ret)
>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>> +
>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>> + 1, &r5_core->pm_domain_id);
>> + if (ret) {
>> + dev_err(dev, "failed to get power-domains property\n");
>> + return ret;
>> + }
>> +
>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>> + if (ret) {
>> + dev_err(dev, "failed to init r5 core %d\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + struct device *dev = cluster->dev;
>> + struct device_node *dev_node = dev_of_node(dev);
>> + struct device_node *child;
>> + struct platform_device *child_pdev;
>> + int core_count = 0, ret, i;
>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>> + struct zynqmp_r5_core *r5_cores;
>> +
>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>> +
>> + /* on success returns 0, if not defined then returns -EINVAL,
>> + * In that case, default is LOCKSTEP mode
>> + */
>> + if (ret != -EINVAL && ret != 0) {
>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster_mode == SINGLE_CPU_MODE) {
>> + dev_err(dev, "driver does not support single cpu mode\n");
>> + return -EINVAL;
>> + } else if ((cluster_mode != SPLIT_MODE &&
>> + cluster_mode != LOCKSTEP_MODE)) {
>> + dev_err(dev, "Invalid cluster mode\n");
>> + return -EINVAL;
>> + }
>> +
>> + core_count = of_get_available_child_count(dev_node);
>> + if (core_count <= 0) {
>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>> + return -EINVAL;
>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>> + return -EINVAL;
>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>> + dev_warn(dev, "Only r5 core0 will be used\n");
>> + core_count = 1;
>> + }
>> +
>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>> + core_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_cores)) {
>> + dev_err(dev, "can't allocate memory for cores\n");
>> + return -ENOMEM;
>> + }
>> +
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
>> + return -ENODEV;
>> +
>> + r5_cores[i].dev = &child_pdev->dev;
>> + if (!r5_cores[i].dev) {
>> + pr_err("can't get device for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>> + if (!r5_cores[i].np) {
>> + pr_err("can't get device node for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + i++;
>> + if (i == core_count)
>> + break;
>> + }
>> +
>> + cluster->mode = cluster_mode;
>> + cluster->core_count = core_count;
>> + cluster->r5_cores = r5_cores;
>> +
>> + ret = zynqmp_r5_core_init(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + zynqmp_r5_print_dt_node_info(cluster);
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
>> +
>> + pr_info("Exit r5f subsystem driver\n");
>> +}
>> +
>> +/*
>> + * zynqmp_r5_remoteproc_probe()
>> + *
>> + * @pdev: domain platform device for R5 cluster
>> + *
>> + * called when driver is probed, for each R5 core specified in DT,
>> + * setup as needed to do remoteproc-related operations
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>> +{
>> + int ret;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct device *dev = &pdev->dev;
>> +
>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(cluster))
>> + return -ENOMEM;
>> +
>> + cluster->dev = dev;
>> +
>> + ret = devm_of_platform_populate(dev);
>> + if (ret) {
>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /* wire in so each core can be cleaned up at driver remove */
>> + platform_set_drvdata(pdev, cluster);
>> +
>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_cluster_init(cluster);
>> + if (ret) {
>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>> + return ret;
>> + }
>> +
>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
>> + return 0;
>> +}
>> +
>> +/* Match table for OF platform binding */
>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>> + { .compatible = "xlnx,zynqmp-r5fss", },
>> + { /* end of list */ },
>> +};
>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>> +
>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>> + .probe = zynqmp_r5_remoteproc_probe,
>> + .driver = {
>> + .name = "zynqmp_r5_remoteproc",
>> + .of_match_table = zynqmp_r5_remoteproc_match,
>> + },
>> +};
>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>> +
>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>> +MODULE_AUTHOR("Xilinx Inc.");
>> +MODULE_LICENSE("GPL v2");
>> --
>> 2.25.1
>>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-11-30 8:01 ` Tanmay Shah
@ 2021-11-30 20:14 ` Mathieu Poirier
-1 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-11-30 20:14 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On Tue, 30 Nov 2021 at 01:01, Tanmay Shah <tanmay.shah@xilinx.com> wrote:
>
>
> On 11/30/21 12:12 AM, Mathieu Poirier wrote:
> > On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
> >> This driver enables r5f dual core Real time Processing Unit subsystem
> >> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
> >> (cluster) can be configured in different modes e.g. split mode in which
> >> two r5f cores work independent of each other and lock-step mode in which
> >> both r5f cores execute same code clock-for-clock and notify if the
> >> result is different.
> >>
> >> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
> >> Platform Management Unit that handles the R5 configuration, memory access
> >> and R5 lifecycle management. The interface to this manager is done in this
> >> driver via zynqmp_pm_* function calls.
> >>
> >> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> >> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
> >> ---
> >> drivers/remoteproc/Kconfig | 12 +
> >> drivers/remoteproc/Makefile | 1 +
> >> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
> >> 3 files changed, 972 insertions(+)
> >> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
> > ...and this patch gives me complation warnings:
> >
> > CC drivers/remoteproc/xlnx_r5_remoteproc.o
> > kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c: In function ‘add_tcm_carveout_lockstep_mode’:
> > kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:412:28: warning: unused variable ‘cluster’ [-Wunused-variable]
> > 412 | struct zynqmp_r5_cluster *cluster;
> > | ^~~~~~~
> > kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:411:26: warning: unused variable ‘parent_pdev’ [-Wunused-variable]
> > 411 | struct platform_device *parent_pdev;
> > | ^~~~~~~~~~~
> >
> > The above leads me to believe this patchset was not compiled before it was sent
> > out.
>
> Please don't assume that this patch-set was not compiled. This driver
> was compiled and tested on Xilinx QEMU and zynqmp platform for its
> functionality.
>
> This driver went through multiple internal reviews and I had to
> re-architecture it multiple times.
And yet two unused variables (which is pretty easy to spot) got by.
>
> I simply missed to fix above warnings before submitting driver. I have
> compiled driver with following command:
>
> //make ARCH=arm64 W=1 C=1 CROSS_COMPILE="aarch64-linux-gnu-" -j32 -Rr
> O=$zynqmp_kernel_build/rproc-next
>
> I did my best to make sure driver stays warning free, however few
> warnings were still missed. That is not intentional and by mistake. I
> fully intend to comply with Linux Kernel community guideline and
> checklist before submitting patches.
>
I am sure you do.
> Apart from above warnings, two more warnings are there in v2 i.e. due to
> typecast issue between (void __iomem *) and (void *). But, I had not
> solution before. Also they were used before in different driver so, I
> chose to use them anyway.
>
The sparse warnings related to the "__iomem *" can be ignored. I also
tried to find a solution to those but realised it was more involved
than expected.
> I just found that using memremap set of functions will fix them. I will
> use those functions in next patch set instead of ioremap_wc as explained
> in this article: https://lwn.net/Articles/653585/
>
This is very interesting...
> > Being new to this I can understand that checkpatch.pl was omitted (albeit amply
> > documented) but obvious compilation warnings can't be excused. As such I
> > am dropping this set and will not review another version until January.
>
>
> I understand having warnings in driver may lead to frustration and may
> cause trust issues for the rest of the code and it can't be excused at all.
>
Very much so.
> I am aware of guidelines of sending patches upstream
> (https://www.kernel.org/doc/html/latest/process/submitting-patches.html)
> and fully intend to comply with that and like I said, if I miss
> something it is not by intention but human error.
>
> I constantly try to improve processes to upstream patches so we don't
> face above type of issues.
>
> I did run checkpatch.pl as per best of my knowledge and fixed lot of
> style related warnings reported by it. Also I enabled W=1 option in my
> compilation command and fixed lots of warnings reported by compiler too.
> As I explained earlier, I wasn't aware of unused-variable warnings, and
> so I missed to fix them.
>
Ok
> With this, I request to review driver from functionality point of view
> as well along with style errors. So, I can address more number of
> comments / concerns in less number of patches. If you want these
> warnings to be fixed, I will send v3 and you can put more comments on
> v3. However, I highly appreciate if we can continue reviews and not
> postpone till January. Please let me know your thoughts.
>
I will give this patchset another chance later this week.
>
> Thanks,
>
> Tanmay
>
>
> > Mathieu
>
>
> >> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> >> index f30d00a3aabe..27f66910d8d3 100644
> >> --- a/drivers/remoteproc/Kconfig
> >> +++ b/drivers/remoteproc/Kconfig
> >> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
> >> It's safe to say N here if you're not interested in utilizing
> >> a slave processor.
> >>
> >> +config XLNX_R5_REMOTEPROC
> >> + tristate "Xilinx R5 remoteproc support"
> >> + depends on PM && ARCH_ZYNQMP
> >> + depends on ZYNQMP_FIRMWARE
> >> + select RPMSG_VIRTIO
> >> + select ZYNQMP_IPI_MBOX
> >> + help
> >> + Say y or m here to support Xilinx R5 remote processors via the remote
> >> + processor framework.
> >> +
> >> + It's safe to say N if not interested in using RPU r5f cores.
> >> +
> >> endif # REMOTEPROC
> >>
> >> endmenu
> >> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> >> index bb26c9e4ef9c..334a8bed4c14 100644
> >> --- a/drivers/remoteproc/Makefile
> >> +++ b/drivers/remoteproc/Makefile
> >> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
> >> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
> >> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
> >> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
> >> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
> >> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> >> new file mode 100644
> >> index 000000000000..c2167fd3869d
> >> --- /dev/null
> >> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> >> @@ -0,0 +1,959 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +/*
> >> + * ZynqMP R5 Remote Processor driver
> >> + *
> >> + */
> >> +
> >> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
> >> +#include <linux/firmware/xlnx-zynqmp.h>
> >> +#include <linux/interrupt.h>
> >> +#include <linux/kernel.h>
> >> +#include <linux/list.h>
> >> +#include <linux/mailbox_client.h>
> >> +#include <linux/mailbox/zynqmp-ipi-message.h>
> >> +#include <linux/module.h>
> >> +#include <linux/of_address.h>
> >> +#include <linux/of_platform.h>
> >> +#include <linux/of_reserved_mem.h>
> >> +#include <linux/platform_device.h>
> >> +#include <linux/remoteproc.h>
> >> +#include <linux/skbuff.h>
> >> +#include <linux/sysfs.h>
> >> +
> >> +#include "remoteproc_internal.h"
> >> +
> >> +/* settings for RPU cluster mode */
> >> +enum zynqmp_r5_cluster_mode {
> >> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
> >> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
> >> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
> >> +};
> >> +
> >> +/**
> >> + * struct mem_bank_data - Memory Bank description
> >> + *
> >> + * @addr: Start address of memory bank
> >> + * @size: Size of Memory bank
> >> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
> >> + * @bank_name: name of the bank for remoteproc framework
> >> + */
> >> +struct mem_bank_data {
> >> + phys_addr_t addr;
> >> + size_t size;
> >> + enum pm_node_id pm_domain_id;
> >> + char *bank_name;
> >> +};
> >> +
> >> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
> >> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> >> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
> >> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
> >> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
> >> +};
> >> +
> >> +/**
> >> + * struct zynqmp_r5_core - ZynqMP R5 core structure
> >> + *
> >> + * @dev: device of RPU instance
> >> + * @np: device node of RPU instance
> >> + * @tcm_bank_count: number TCM banks accessible to this RPU
> >> + * @tcm_banks: array of each TCM bank data
> >> + * @res_mem_count: number of Reserved Memory regions per core
> >> + * @res_mem: array of reserved memory regions
> >> + * @rproc: rproc handle
> >> + * @pm_domain_id: RPU CPU power domain id
> >> + */
> >> +struct zynqmp_r5_core {
> >> + struct device *dev;
> >> + struct device_node *np;
> >> + int tcm_bank_count;
> >> + struct mem_bank_data *tcm_banks;
> >> + int res_mem_count;
> >> + struct reserved_mem *res_mem;
> >> + struct rproc *rproc;
> >> + enum pm_node_id pm_domain_id;
> >> +};
> >> +
> >> +/**
> >> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
> >> + *
> >> + * @dev: r5f subsystem cluster device node
> >> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
> >> + * @core_count: number of r5 cores used for this cluster mode
> >> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
> >> + */
> >> +struct zynqmp_r5_cluster {
> >> + struct device *dev;
> >> + enum zynqmp_r5_cluster_mode mode;
> >> + int core_count;
> >> + struct zynqmp_r5_core *r5_cores;
> >> +};
> >> +
> >> +/*
> >> + * zynqmp_r5_set_mode - set RPU operation mode
> >> + *
> >> + * set RPU operation mode
> >> + *
> >> + * Return: 0 for success, negative value for failure
> >> + */
> >> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
> >> + enum zynqmp_r5_cluster_mode rpu_mode)
> >> +{
> >> + enum rpu_tcm_comb tcm_mode;
> >> + int ret, reg_val;
> >> +
> >> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
> >> +
> >> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
> >> + if (ret < 0) {
> >> + pr_err("failed to set RPU mode\n");
> >> + return ret;
> >> + }
> >> +
> >> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
> >> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> >> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
> >> + if (ret < 0)
> >> + pr_err("failed to configure TCM\n");
> >> +
> >> + return ret;
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_rproc_start
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + *
> >> + * Start R5 Core from designated boot address.
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> >> +{
> >> + struct zynqmp_r5_core *r5_core = rproc->priv;
> >> + enum rpu_boot_mem bootmem;
> >> + int ret;
> >> +
> >> + if (!r5_core) {
> >> + pr_err("can't get r5 core\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
> >> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> >> +
> >> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
> >> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> >> +
> >> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
> >> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> >> + if (ret)
> >> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
> >> + return ret;
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_rproc_stop
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + *
> >> + * Power down R5 Core.
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> >> +{
> >> + struct zynqmp_r5_core *r5_core = rproc->priv;
> >> + int ret;
> >> +
> >> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
> >> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> >> + if (ret)
> >> + pr_err("failed to stop remoteproc RPU %d\n", ret);
> >> +
> >> + return ret;
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_rproc_mem_map
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + * @mem: mem entry to map
> >> + *
> >> + * Callback to map va for memory-region's carveout.
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
> >> + struct rproc_mem_entry *mem)
> >> +{
> >> + void __iomem *va;
> >> +
> >> + va = ioremap_wc(mem->dma, mem->len);
> >> + if (IS_ERR_OR_NULL(va))
> >> + return -ENOMEM;
> >> +
> >> + mem->va = (void *)va;
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_rproc_mem_unmap
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + * @mem: mem entry to unmap
> >> + *
> >> + * Unmap memory-region carveout
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
> >> + struct rproc_mem_entry *mem)
> >> +{
> >> + iounmap((void __iomem *)mem->va);
> >> + return 0;
> >> +}
> >> +
> >> +/*
> >> + * add_mem_regions
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + *
> >> + * Construct rproc mem carveouts from carveout provided in
> >> + * memory-region property
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int add_mem_regions(struct rproc *rproc)
> >> +{
> >> + struct device *dev;
> >> + struct rproc_mem_entry *mem;
> >> + struct reserved_mem *rmem;
> >> + struct zynqmp_r5_core *r5_core;
> >> + int i;
> >> +
> >> + r5_core = rproc->priv;
> >> + dev = r5_core->dev;
> >> +
> >> + /* Register associated reserved memory regions */
> >> + for (i = 0; i < r5_core->res_mem_count; i++) {
> >> + rmem = &r5_core->res_mem[i];
> >> + mem = rproc_mem_entry_init(dev, NULL,
> >> + (dma_addr_t)rmem->base,
> >> + rmem->size, rmem->base,
> >> + zynqmp_r5_rproc_mem_map,
> >> + zynqmp_r5_rproc_mem_unmap,
> >> + rmem->name);
> >> + if (IS_ERR_OR_NULL(mem))
> >> + return -ENOMEM;
> >> +
> >> + rproc_add_carveout(rproc, mem);
> >> + }
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_rproc_mem_unmap
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + * @mem: mem entry to unmap
> >> + *
> >> + * Unmap TCM banks when powering down R5 core.
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
> >> +{
> >> + struct zynqmp_r5_core *r5_core;
> >> + int i;
> >> + enum pm_node_id pm_domain_id;
> >> +
> >> + r5_core = rproc->priv;
> >> + if (!r5_core) {
> >> + pr_err("r5 core is not available\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + iounmap((void __iomem *)mem->va);
> >> +
> >> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
> >> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> >> + if (zynqmp_pm_release_node(pm_domain_id))
> >> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
> >> + }
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +/*
> >> + * tcm_mem_map
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + * @mem: mem entry to initialize the va and da fields of
> >> + *
> >> + * Given TCM bank entry, this callback will set device address for R5
> >> + * running on TCM and also setup virtual address for TCM bank
> >> + * remoteproc carveout.
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int tcm_mem_map(struct rproc *rproc,
> >> + struct rproc_mem_entry *mem)
> >> +{
> >> + void __iomem *va;
> >> +
> >> + va = ioremap_wc(mem->dma, mem->len);
> >> + if (IS_ERR_OR_NULL(va))
> >> + return -ENOMEM;
> >> +
> >> + /* Update memory entry va */
> >> + mem->va = (void *)va;
> >> +
> >> + /* clear TCMs */
> >> + memset_io(va, 0, mem->len);
> >> +
> >> + /*
> >> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> >> + * while on the Linux side they are at 0xffexxxxx.
> >> + *
> >> + * Zero out the high 12 bits of the address. This will give
> >> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> >> + */
> >> + mem->da &= 0x000fffff;
> >> +
> >> + /*
> >> + * TCM Banks 1A and 1B still have to be translated.
> >> + *
> >> + * Below handle these two banks' absolute addresses (0xffe90000 and
> >> + * 0xffeb0000) and convert to the expected relative addresses
> >> + * (0x0 and 0x20000).
> >> + */
> >> + if (mem->da == 0x90000 || mem->da == 0xB0000)
> >> + mem->da -= 0x90000;
> >> +
> >> + /* if translated TCM bank address is not valid report error */
> >> + if (mem->da != 0x0 && mem->da != 0x20000) {
> >> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
> >> + return -EINVAL;
> >> + }
> >> + return 0;
> >> +}
> >> +
> >> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
> >> +{
> >> + int i, num_banks, ret;
> >> + struct rproc_mem_entry *mem;
> >> + enum pm_node_id pm_domain_id;
> >> + u32 bank_addr;
> >> + size_t bank_size = 0;
> >> + char *bank_name;
> >> + struct device *dev;
> >> + struct zynqmp_r5_core *r5_core;
> >> +
> >> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> >> + if (!r5_core)
> >> + return -EINVAL;
> >> +
> >> + dev = r5_core->dev;
> >> +
> >> + /* go through zynqmp banks for r5 node */
> >> + num_banks = r5_core->tcm_bank_count;
> >> + if (num_banks <= 0) {
> >> + dev_err(dev, "need to specify TCM banks\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + for (i = 0; i < num_banks; i++) {
> >> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
> >> + bank_name = r5_core->tcm_banks[i].bank_name;
> >> + bank_size = r5_core->tcm_banks[i].size;
> >> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> >> +
> >> + ret = zynqmp_pm_request_node(pm_domain_id,
> >> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> >> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> >> + if (ret < 0) {
> >> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> >> + return ret;
> >> + }
> >> +
> >> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
> >> + bank_name, bank_addr, bank_size);
> >> +
> >> + /* add carveout */
> >> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> >> + bank_size, bank_addr,
> >> + tcm_mem_map, tcm_mem_unmap,
> >> + bank_name);
> >> + if (IS_ERR_OR_NULL(mem)) {
> >> + /* Turn off all TCM banks turned on before */
> >> + do {
> >> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> >> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> >> + if (ret)
> >> + dev_warn(dev,
> >> + "fail to release node: %x, %x\n",
> >> + (u32)pm_domain_id, ret);
> >> + } while (i--);
> >> + return -ENOMEM;
> >> + }
> >> +
> >> + rproc_add_carveout(rproc, mem);
> >> + }
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> >> +{
> >> + int i, num_banks, ret;
> >> + struct rproc_mem_entry *mem;
> >> + enum pm_node_id pm_domain_id;
> >> + u32 bank_addr;
> >> + size_t bank_size = 0;
> >> + char *bank_name;
> >> + struct device *dev;
> >> + struct platform_device *parent_pdev;
> >> + struct zynqmp_r5_cluster *cluster;
> >> + struct zynqmp_r5_core *r5_core;
> >> +
> >> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> >> + if (!r5_core)
> >> + return -EINVAL;
> >> +
> >> + dev = r5_core->dev;
> >> + if (!dev) {
> >> + pr_err("r5 core device unavailable\n");
> >> + return -ENODEV;
> >> + }
> >> +
> >> + /* go through zynqmp banks for r5 node */
> >> + num_banks = r5_core->tcm_bank_count;
> >> + if (num_banks <= 0) {
> >> + dev_err(dev, "need to specify TCM banks\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
> >> + bank_name = r5_core->tcm_banks[0].bank_name;
> >> + for (i = 0; i < num_banks; i++) {
> >> + bank_size += r5_core->tcm_banks[i].size;
> >> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> >> +
> >> + ret = zynqmp_pm_request_node(pm_domain_id,
> >> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> >> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> >> + if (ret < 0) {
> >> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> >> + return ret;
> >> + }
> >> + }
> >> +
> >> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
> >> + bank_name, bank_addr, bank_size);
> >> +
> >> + /* add carveout */
> >> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> >> + bank_size, bank_addr,
> >> + tcm_mem_map, tcm_mem_unmap,
> >> + bank_name);
> >> + if (IS_ERR_OR_NULL(mem)) {
> >> + for (i = 0; i < num_banks; i++) {
> >> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> >> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> >> + if (ret)
> >> + dev_warn(dev,
> >> + "fail to release node: %x ret: %x\n",
> >> + (u32)pm_domain_id, ret);
> >> + }
> >> + return -ENOMEM;
> >> + }
> >> +
> >> + rproc_add_carveout(rproc, mem);
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +/*
> >> + * add_tcm_banks()
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + *
> >> + * Given R5 node in remoteproc instance
> >> + * allocate remoteproc carveout for TCM memory
> >> + * needed for firmware to be loaded
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int add_tcm_banks(struct rproc *rproc)
> >> +{
> >> + struct device *dev;
> >> + struct platform_device *parent_pdev;
> >> + struct zynqmp_r5_cluster *cluster;
> >> + struct zynqmp_r5_core *r5_core;
> >> +
> >> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> >> + if (!r5_core)
> >> + return -EINVAL;
> >> +
> >> + dev = r5_core->dev;
> >> + if (!dev) {
> >> + pr_err("r5 core device unavailable\n");
> >> + return -ENODEV;
> >> + }
> >> +
> >> + parent_pdev = to_platform_device(dev->parent);
> >> + if (!parent_pdev) {
> >> + dev_err(dev, "parent platform dev unavailable\n");
> >> + return -ENODEV;
> >> + }
> >> +
> >> + cluster = platform_get_drvdata(parent_pdev);
> >> + if (!cluster) {
> >> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + if (cluster->mode == SPLIT_MODE)
> >> + return add_tcm_carveout_split_mode(rproc);
> >> + else if (cluster->mode == LOCKSTEP_MODE)
> >> + return add_tcm_carveout_lockstep_mode(rproc);
> >> +
> >> + dev_err(cluster->dev, "invalid cluster mode\n");
> >> + return -EINVAL;
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_parse_fw()
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + * @fw: ptr to firmware to be loaded onto r5 core
> >> + *
> >> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> >> +{
> >> + int ret;
> >> + struct zynqmp_r5_core *r5_core;
> >> + struct device *dev;
> >> +
> >> + r5_core = rproc->priv;
> >> + if (!r5_core) {
> >> + dev_err(&rproc->dev, "r5 core not available\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + dev = r5_core->dev;
> >> +
> >> + ret = add_tcm_banks(rproc);
> >> + if (ret) {
> >> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
> >> + return ret;
> >> + }
> >> +
> >> + ret = add_mem_regions(rproc);
> >> + if (ret)
> >> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
> >> +
> >> + ret = rproc_elf_load_rsc_table(rproc, fw);
> >> + if (ret == -EINVAL) {
> >> + /*
> >> + * resource table only required for IPC.
> >> + * if not present, this is not necessarily an error;
> >> + * for example, loading r5 hello world application
> >> + * so simply inform user and keep going.
> >> + */
> >> + dev_info(&rproc->dev, "no resource table found.\n");
> >> + ret = 0;
> >> + }
> >> + return ret;
> >> +}
> >> +
> >> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> >> + .start = zynqmp_r5_rproc_start,
> >> + .stop = zynqmp_r5_rproc_stop,
> >> + .load = rproc_elf_load_segments,
> >> + .parse_fw = zynqmp_r5_parse_fw,
> >> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> >> + .sanity_check = rproc_elf_sanity_check,
> >> + .get_boot_addr = rproc_elf_get_boot_addr,
> >> +};
> >> +
> >> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
> >> +{
> >> + int i, j, k;
> >> + struct zynqmp_r5_core *r5_core;
> >> +
> >> + dev_dbg(cluster->dev, "Printing dt node info\n");
> >> +
> >> + pr_debug("cluster mode = %d\n", cluster->mode);
> >> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
> >> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
> >> + pr_debug("r5f num cores = %d\n", cluster->core_count);
> >> +
> >> + for (i = 0; i < cluster->core_count; i++) {
> >> + r5_core = &cluster->r5_cores[i];
> >> + if (!r5_core) {
> >> + pr_err("can't get r5_core\n");
> >> + continue;
> >> + }
> >> +
> >> + pr_debug("r5 core %d nodes\n", i);
> >> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
> >> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
> >> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
> >> + k, r5_core->tcm_banks[k].addr,
> >> + r5_core->tcm_banks[k].size,
> >> + r5_core->tcm_banks[k].pm_domain_id,
> >> + r5_core->tcm_banks[k].bank_name);
> >> + }
> >> +
> >> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
> >> +
> >> + for (j = 0; j < r5_core->res_mem_count; j++) {
> >> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
> >> + j, r5_core->res_mem[j].base,
> >> + r5_core->res_mem[j].size,
> >> + r5_core->res_mem[j].name);
> >> + }
> >> + }
> >> +}
> >> +
> >> +/**
> >> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
> >> + * this is called for each individual R5 core to
> >> + * set up mailbox, Xilinx platform manager unique ID,
> >> + * add to rproc core
> >> + *
> >> + * @r5_core: zynqmp_r5_core r5 core object to initialize
> >> + *
> >> + * Return: 0 for success, negative value for failure.
> >> + */
> >> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
> >> +{
> >> + int ret;
> >> + struct rproc *r5_rproc;
> >> + struct device *dev;
> >> +
> >> + dev = r5_core->dev;
> >> +
> >> + /* Set up DMA mask */
> >> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> >> + if (ret)
> >> + return ret;
> >> +
> >> + /* Allocate remoteproc instance */
> >> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> >> + NULL, sizeof(struct zynqmp_r5_core));
> >> + if (IS_ERR_OR_NULL(r5_rproc))
> >> + return -ENOMEM;
> >> +
> >> + r5_rproc->auto_boot = false;
> >> + r5_rproc->priv = r5_core;
> >> +
> >> + /* Add R5 remoteproc */
> >> + ret = devm_rproc_add(dev, r5_rproc);
> >> + if (ret) {
> >> + pr_err("failed to add r5 remoteproc\n");
> >> + return ret;
> >> + }
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> >> +{
> >> + int tcm_bank_count, tcm_node;
> >> + int i = 0, j;
> >> + struct zynqmp_r5_core *r5_core;
> >> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> >> + struct device *dev = cluster->dev;
> >> +
> >> + /* ToDo: Use predefined TCM address space values from driver until
> >> + * system-dt spec is not final fot TCM
> >> + */
> >> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> >> +
> >> + /* count per core tcm banks */
> >> + tcm_bank_count = tcm_bank_count / cluster->core_count;
> >> +
> >> + /* r5 core 0 will use all of TCM banks in lockstep mode.
> >> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> >> + * 128k. Assign TCM banks to each core accordingly
> >> + */
> >> + tcm_node = 0;
> >> + for (j = 0; j < cluster->core_count; j++) {
> >> + r5_core = &cluster->r5_cores[j];
> >> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> >> + tcm_bank_count, GFP_KERNEL);
> >> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> >> + return -ENOMEM;
> >> +
> >> + for (i = 0; i < tcm_bank_count; i++) {
> >> + /* Use pre-defined TCM reg values.
> >> + * Eventually this should be replaced by values
> >> + * parsed from dts.
> >> + */
> >> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> >> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> >> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> >> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> >> + tcm_node++;
> >> + }
> >> +
> >> + r5_core->tcm_bank_count = tcm_bank_count;
> >> + }
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> >> +{
> >> + int res_mem_count, i;
> >> + struct device *dev;
> >> + struct device_node *np, *rmem_np;
> >> + struct reserved_mem *rmem;
> >> +
> >> + dev = r5_core->dev;
> >> +
> >> + np = r5_core->np;
> >> + if (IS_ERR_OR_NULL(np)) {
> >> + pr_err("invalid device node of r5 core\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> >> + sizeof(phandle));
> >> + if (res_mem_count <= 0) {
> >> + dev_warn(dev, "failed to get memory-region property %d\n",
> >> + res_mem_count);
> >> + return -EINVAL;
> >> + }
> >> +
> >> + r5_core->res_mem = devm_kzalloc(dev,
> >> + res_mem_count * sizeof(struct reserved_mem),
> >> + GFP_KERNEL);
> >> + if (!r5_core->res_mem) {
> >> + dev_err(dev, "failed to allocate mem region memory\n");
> >> + return -ENOMEM;
> >> + }
> >> +
> >> + for (i = 0; i < res_mem_count; i++) {
> >> + rmem_np = of_parse_phandle(np, "memory-region", i);
> >> + if (!rmem_np)
> >> + return -EINVAL;
> >> +
> >> + rmem = of_reserved_mem_lookup(rmem_np);
> >> + if (!rmem) {
> >> + of_node_put(rmem_np);
> >> + return -EINVAL;
> >> + }
> >> +
> >> + memcpy(&r5_core->res_mem[i], rmem,
> >> + sizeof(struct reserved_mem));
> >> + of_node_put(rmem_np);
> >> + }
> >> +
> >> + r5_core->res_mem_count = res_mem_count;
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> >> +{
> >> + int ret, i;
> >> + struct zynqmp_r5_core *r5_core;
> >> + struct device *dev = cluster->dev;
> >> +
> >> + ret = zynqmp_r5_get_tcm_node(cluster);
> >> + if (ret < 0) {
> >> + dev_err(dev, "can't get tcm node, err %d\n", ret);
> >> + return ret;
> >> + }
> >> +
> >> + for (i = 0; i < cluster->core_count; i++) {
> >> + r5_core = &cluster->r5_cores[i];
> >> + if (!r5_core) {
> >> + pr_err("invalid r5 core\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + ret = zynqmp_r5_get_mem_region_node(r5_core);
> >> + if (ret)
> >> + dev_warn(dev, "memory-region prop failed %d\n", ret);
> >> +
> >> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> >> + 1, &r5_core->pm_domain_id);
> >> + if (ret) {
> >> + dev_err(dev, "failed to get power-domains property\n");
> >> + return ret;
> >> + }
> >> +
> >> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> >> + if (ret)
> >> + return ret;
> >> +
> >> + ret = zynqmp_r5_add_rproc_core(r5_core);
> >> + if (ret) {
> >> + dev_err(dev, "failed to init r5 core %d\n", i);
> >> + return ret;
> >> + }
> >> + }
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> >> +{
> >> + struct device *dev = cluster->dev;
> >> + struct device_node *dev_node = dev_of_node(dev);
> >> + struct device_node *child;
> >> + struct platform_device *child_pdev;
> >> + int core_count = 0, ret, i;
> >> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> >> + struct zynqmp_r5_core *r5_cores;
> >> +
> >> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> >> +
> >> + /* on success returns 0, if not defined then returns -EINVAL,
> >> + * In that case, default is LOCKSTEP mode
> >> + */
> >> + if (ret != -EINVAL && ret != 0) {
> >> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + if (cluster_mode == SINGLE_CPU_MODE) {
> >> + dev_err(dev, "driver does not support single cpu mode\n");
> >> + return -EINVAL;
> >> + } else if ((cluster_mode != SPLIT_MODE &&
> >> + cluster_mode != LOCKSTEP_MODE)) {
> >> + dev_err(dev, "Invalid cluster mode\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + core_count = of_get_available_child_count(dev_node);
> >> + if (core_count <= 0) {
> >> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> >> + return -EINVAL;
> >> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> >> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> >> + return -EINVAL;
> >> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> >> + dev_warn(dev, "Only r5 core0 will be used\n");
> >> + core_count = 1;
> >> + }
> >> +
> >> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> >> + core_count, GFP_KERNEL);
> >> + if (IS_ERR_OR_NULL(r5_cores)) {
> >> + dev_err(dev, "can't allocate memory for cores\n");
> >> + return -ENOMEM;
> >> + }
> >> +
> >> + i = 0;
> >> + for_each_available_child_of_node(dev_node, child) {
> >> + child_pdev = of_find_device_by_node(child);
> >> + if (!child_pdev)
> >> + return -ENODEV;
> >> +
> >> + r5_cores[i].dev = &child_pdev->dev;
> >> + if (!r5_cores[i].dev) {
> >> + pr_err("can't get device for r5 core %d\n", i);
> >> + return -ENODEV;
> >> + }
> >> +
> >> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> >> + if (!r5_cores[i].np) {
> >> + pr_err("can't get device node for r5 core %d\n", i);
> >> + return -ENODEV;
> >> + }
> >> +
> >> + i++;
> >> + if (i == core_count)
> >> + break;
> >> + }
> >> +
> >> + cluster->mode = cluster_mode;
> >> + cluster->core_count = core_count;
> >> + cluster->r5_cores = r5_cores;
> >> +
> >> + ret = zynqmp_r5_core_init(cluster);
> >> + if (ret < 0) {
> >> + dev_err(dev, "failed to init r5 core err %d\n", ret);
> >> + return ret;
> >> + }
> >> +
> >> + zynqmp_r5_print_dt_node_info(cluster);
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +static void zynqmp_r5_cluster_exit(void *data)
> >> +{
> >> + struct platform_device *pdev = (struct platform_device *)data;
> >> +
> >> + platform_set_drvdata(pdev, NULL);
> >> +
> >> + pr_info("Exit r5f subsystem driver\n");
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_remoteproc_probe()
> >> + *
> >> + * @pdev: domain platform device for R5 cluster
> >> + *
> >> + * called when driver is probed, for each R5 core specified in DT,
> >> + * setup as needed to do remoteproc-related operations
> >> + *
> >> + * Return: 0 for success, negative value for failure.
> >> + */
> >> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> >> +{
> >> + int ret;
> >> + struct zynqmp_r5_cluster *cluster;
> >> + struct device *dev = &pdev->dev;
> >> +
> >> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> >> + if (IS_ERR_OR_NULL(cluster))
> >> + return -ENOMEM;
> >> +
> >> + cluster->dev = dev;
> >> +
> >> + ret = devm_of_platform_populate(dev);
> >> + if (ret) {
> >> + dev_err(dev, "failed to populate platform dev %d\n", ret);
> >> + return ret;
> >> + }
> >> +
> >> + /* wire in so each core can be cleaned up at driver remove */
> >> + platform_set_drvdata(pdev, cluster);
> >> +
> >> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> >> + if (ret)
> >> + return ret;
> >> +
> >> + ret = zynqmp_r5_cluster_init(cluster);
> >> + if (ret) {
> >> + dev_err(dev, "Invalid r5f subsystem device tree\n");
> >> + return ret;
> >> + }
> >> +
> >> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> >> + return 0;
> >> +}
> >> +
> >> +/* Match table for OF platform binding */
> >> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> >> + { .compatible = "xlnx,zynqmp-r5fss", },
> >> + { /* end of list */ },
> >> +};
> >> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> >> +
> >> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> >> + .probe = zynqmp_r5_remoteproc_probe,
> >> + .driver = {
> >> + .name = "zynqmp_r5_remoteproc",
> >> + .of_match_table = zynqmp_r5_remoteproc_match,
> >> + },
> >> +};
> >> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> >> +
> >> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> >> +MODULE_AUTHOR("Xilinx Inc.");
> >> +MODULE_LICENSE("GPL v2");
> >> --
> >> 2.25.1
> >>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-11-30 20:14 ` Mathieu Poirier
0 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-11-30 20:14 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On Tue, 30 Nov 2021 at 01:01, Tanmay Shah <tanmay.shah@xilinx.com> wrote:
>
>
> On 11/30/21 12:12 AM, Mathieu Poirier wrote:
> > On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
> >> This driver enables r5f dual core Real time Processing Unit subsystem
> >> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
> >> (cluster) can be configured in different modes e.g. split mode in which
> >> two r5f cores work independent of each other and lock-step mode in which
> >> both r5f cores execute same code clock-for-clock and notify if the
> >> result is different.
> >>
> >> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
> >> Platform Management Unit that handles the R5 configuration, memory access
> >> and R5 lifecycle management. The interface to this manager is done in this
> >> driver via zynqmp_pm_* function calls.
> >>
> >> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> >> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
> >> ---
> >> drivers/remoteproc/Kconfig | 12 +
> >> drivers/remoteproc/Makefile | 1 +
> >> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
> >> 3 files changed, 972 insertions(+)
> >> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
> > ...and this patch gives me complation warnings:
> >
> > CC drivers/remoteproc/xlnx_r5_remoteproc.o
> > kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c: In function ‘add_tcm_carveout_lockstep_mode’:
> > kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:412:28: warning: unused variable ‘cluster’ [-Wunused-variable]
> > 412 | struct zynqmp_r5_cluster *cluster;
> > | ^~~~~~~
> > kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:411:26: warning: unused variable ‘parent_pdev’ [-Wunused-variable]
> > 411 | struct platform_device *parent_pdev;
> > | ^~~~~~~~~~~
> >
> > The above leads me to believe this patchset was not compiled before it was sent
> > out.
>
> Please don't assume that this patch-set was not compiled. This driver
> was compiled and tested on Xilinx QEMU and zynqmp platform for its
> functionality.
>
> This driver went through multiple internal reviews and I had to
> re-architecture it multiple times.
And yet two unused variables (which is pretty easy to spot) got by.
>
> I simply missed to fix above warnings before submitting driver. I have
> compiled driver with following command:
>
> //make ARCH=arm64 W=1 C=1 CROSS_COMPILE="aarch64-linux-gnu-" -j32 -Rr
> O=$zynqmp_kernel_build/rproc-next
>
> I did my best to make sure driver stays warning free, however few
> warnings were still missed. That is not intentional and by mistake. I
> fully intend to comply with Linux Kernel community guideline and
> checklist before submitting patches.
>
I am sure you do.
> Apart from above warnings, two more warnings are there in v2 i.e. due to
> typecast issue between (void __iomem *) and (void *). But, I had not
> solution before. Also they were used before in different driver so, I
> chose to use them anyway.
>
The sparse warnings related to the "__iomem *" can be ignored. I also
tried to find a solution to those but realised it was more involved
than expected.
> I just found that using memremap set of functions will fix them. I will
> use those functions in next patch set instead of ioremap_wc as explained
> in this article: https://lwn.net/Articles/653585/
>
This is very interesting...
> > Being new to this I can understand that checkpatch.pl was omitted (albeit amply
> > documented) but obvious compilation warnings can't be excused. As such I
> > am dropping this set and will not review another version until January.
>
>
> I understand having warnings in driver may lead to frustration and may
> cause trust issues for the rest of the code and it can't be excused at all.
>
Very much so.
> I am aware of guidelines of sending patches upstream
> (https://www.kernel.org/doc/html/latest/process/submitting-patches.html)
> and fully intend to comply with that and like I said, if I miss
> something it is not by intention but human error.
>
> I constantly try to improve processes to upstream patches so we don't
> face above type of issues.
>
> I did run checkpatch.pl as per best of my knowledge and fixed lot of
> style related warnings reported by it. Also I enabled W=1 option in my
> compilation command and fixed lots of warnings reported by compiler too.
> As I explained earlier, I wasn't aware of unused-variable warnings, and
> so I missed to fix them.
>
Ok
> With this, I request to review driver from functionality point of view
> as well along with style errors. So, I can address more number of
> comments / concerns in less number of patches. If you want these
> warnings to be fixed, I will send v3 and you can put more comments on
> v3. However, I highly appreciate if we can continue reviews and not
> postpone till January. Please let me know your thoughts.
>
I will give this patchset another chance later this week.
>
> Thanks,
>
> Tanmay
>
>
> > Mathieu
>
>
> >> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> >> index f30d00a3aabe..27f66910d8d3 100644
> >> --- a/drivers/remoteproc/Kconfig
> >> +++ b/drivers/remoteproc/Kconfig
> >> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
> >> It's safe to say N here if you're not interested in utilizing
> >> a slave processor.
> >>
> >> +config XLNX_R5_REMOTEPROC
> >> + tristate "Xilinx R5 remoteproc support"
> >> + depends on PM && ARCH_ZYNQMP
> >> + depends on ZYNQMP_FIRMWARE
> >> + select RPMSG_VIRTIO
> >> + select ZYNQMP_IPI_MBOX
> >> + help
> >> + Say y or m here to support Xilinx R5 remote processors via the remote
> >> + processor framework.
> >> +
> >> + It's safe to say N if not interested in using RPU r5f cores.
> >> +
> >> endif # REMOTEPROC
> >>
> >> endmenu
> >> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> >> index bb26c9e4ef9c..334a8bed4c14 100644
> >> --- a/drivers/remoteproc/Makefile
> >> +++ b/drivers/remoteproc/Makefile
> >> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
> >> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
> >> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
> >> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
> >> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
> >> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> >> new file mode 100644
> >> index 000000000000..c2167fd3869d
> >> --- /dev/null
> >> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> >> @@ -0,0 +1,959 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +/*
> >> + * ZynqMP R5 Remote Processor driver
> >> + *
> >> + */
> >> +
> >> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
> >> +#include <linux/firmware/xlnx-zynqmp.h>
> >> +#include <linux/interrupt.h>
> >> +#include <linux/kernel.h>
> >> +#include <linux/list.h>
> >> +#include <linux/mailbox_client.h>
> >> +#include <linux/mailbox/zynqmp-ipi-message.h>
> >> +#include <linux/module.h>
> >> +#include <linux/of_address.h>
> >> +#include <linux/of_platform.h>
> >> +#include <linux/of_reserved_mem.h>
> >> +#include <linux/platform_device.h>
> >> +#include <linux/remoteproc.h>
> >> +#include <linux/skbuff.h>
> >> +#include <linux/sysfs.h>
> >> +
> >> +#include "remoteproc_internal.h"
> >> +
> >> +/* settings for RPU cluster mode */
> >> +enum zynqmp_r5_cluster_mode {
> >> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
> >> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
> >> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
> >> +};
> >> +
> >> +/**
> >> + * struct mem_bank_data - Memory Bank description
> >> + *
> >> + * @addr: Start address of memory bank
> >> + * @size: Size of Memory bank
> >> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
> >> + * @bank_name: name of the bank for remoteproc framework
> >> + */
> >> +struct mem_bank_data {
> >> + phys_addr_t addr;
> >> + size_t size;
> >> + enum pm_node_id pm_domain_id;
> >> + char *bank_name;
> >> +};
> >> +
> >> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
> >> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> >> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
> >> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
> >> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
> >> +};
> >> +
> >> +/**
> >> + * struct zynqmp_r5_core - ZynqMP R5 core structure
> >> + *
> >> + * @dev: device of RPU instance
> >> + * @np: device node of RPU instance
> >> + * @tcm_bank_count: number TCM banks accessible to this RPU
> >> + * @tcm_banks: array of each TCM bank data
> >> + * @res_mem_count: number of Reserved Memory regions per core
> >> + * @res_mem: array of reserved memory regions
> >> + * @rproc: rproc handle
> >> + * @pm_domain_id: RPU CPU power domain id
> >> + */
> >> +struct zynqmp_r5_core {
> >> + struct device *dev;
> >> + struct device_node *np;
> >> + int tcm_bank_count;
> >> + struct mem_bank_data *tcm_banks;
> >> + int res_mem_count;
> >> + struct reserved_mem *res_mem;
> >> + struct rproc *rproc;
> >> + enum pm_node_id pm_domain_id;
> >> +};
> >> +
> >> +/**
> >> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
> >> + *
> >> + * @dev: r5f subsystem cluster device node
> >> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
> >> + * @core_count: number of r5 cores used for this cluster mode
> >> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
> >> + */
> >> +struct zynqmp_r5_cluster {
> >> + struct device *dev;
> >> + enum zynqmp_r5_cluster_mode mode;
> >> + int core_count;
> >> + struct zynqmp_r5_core *r5_cores;
> >> +};
> >> +
> >> +/*
> >> + * zynqmp_r5_set_mode - set RPU operation mode
> >> + *
> >> + * set RPU operation mode
> >> + *
> >> + * Return: 0 for success, negative value for failure
> >> + */
> >> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
> >> + enum zynqmp_r5_cluster_mode rpu_mode)
> >> +{
> >> + enum rpu_tcm_comb tcm_mode;
> >> + int ret, reg_val;
> >> +
> >> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
> >> +
> >> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
> >> + if (ret < 0) {
> >> + pr_err("failed to set RPU mode\n");
> >> + return ret;
> >> + }
> >> +
> >> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
> >> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> >> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
> >> + if (ret < 0)
> >> + pr_err("failed to configure TCM\n");
> >> +
> >> + return ret;
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_rproc_start
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + *
> >> + * Start R5 Core from designated boot address.
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> >> +{
> >> + struct zynqmp_r5_core *r5_core = rproc->priv;
> >> + enum rpu_boot_mem bootmem;
> >> + int ret;
> >> +
> >> + if (!r5_core) {
> >> + pr_err("can't get r5 core\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
> >> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> >> +
> >> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
> >> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> >> +
> >> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
> >> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> >> + if (ret)
> >> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
> >> + return ret;
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_rproc_stop
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + *
> >> + * Power down R5 Core.
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> >> +{
> >> + struct zynqmp_r5_core *r5_core = rproc->priv;
> >> + int ret;
> >> +
> >> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
> >> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> >> + if (ret)
> >> + pr_err("failed to stop remoteproc RPU %d\n", ret);
> >> +
> >> + return ret;
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_rproc_mem_map
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + * @mem: mem entry to map
> >> + *
> >> + * Callback to map va for memory-region's carveout.
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
> >> + struct rproc_mem_entry *mem)
> >> +{
> >> + void __iomem *va;
> >> +
> >> + va = ioremap_wc(mem->dma, mem->len);
> >> + if (IS_ERR_OR_NULL(va))
> >> + return -ENOMEM;
> >> +
> >> + mem->va = (void *)va;
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_rproc_mem_unmap
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + * @mem: mem entry to unmap
> >> + *
> >> + * Unmap memory-region carveout
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
> >> + struct rproc_mem_entry *mem)
> >> +{
> >> + iounmap((void __iomem *)mem->va);
> >> + return 0;
> >> +}
> >> +
> >> +/*
> >> + * add_mem_regions
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + *
> >> + * Construct rproc mem carveouts from carveout provided in
> >> + * memory-region property
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int add_mem_regions(struct rproc *rproc)
> >> +{
> >> + struct device *dev;
> >> + struct rproc_mem_entry *mem;
> >> + struct reserved_mem *rmem;
> >> + struct zynqmp_r5_core *r5_core;
> >> + int i;
> >> +
> >> + r5_core = rproc->priv;
> >> + dev = r5_core->dev;
> >> +
> >> + /* Register associated reserved memory regions */
> >> + for (i = 0; i < r5_core->res_mem_count; i++) {
> >> + rmem = &r5_core->res_mem[i];
> >> + mem = rproc_mem_entry_init(dev, NULL,
> >> + (dma_addr_t)rmem->base,
> >> + rmem->size, rmem->base,
> >> + zynqmp_r5_rproc_mem_map,
> >> + zynqmp_r5_rproc_mem_unmap,
> >> + rmem->name);
> >> + if (IS_ERR_OR_NULL(mem))
> >> + return -ENOMEM;
> >> +
> >> + rproc_add_carveout(rproc, mem);
> >> + }
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_rproc_mem_unmap
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + * @mem: mem entry to unmap
> >> + *
> >> + * Unmap TCM banks when powering down R5 core.
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
> >> +{
> >> + struct zynqmp_r5_core *r5_core;
> >> + int i;
> >> + enum pm_node_id pm_domain_id;
> >> +
> >> + r5_core = rproc->priv;
> >> + if (!r5_core) {
> >> + pr_err("r5 core is not available\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + iounmap((void __iomem *)mem->va);
> >> +
> >> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
> >> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> >> + if (zynqmp_pm_release_node(pm_domain_id))
> >> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
> >> + }
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +/*
> >> + * tcm_mem_map
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + * @mem: mem entry to initialize the va and da fields of
> >> + *
> >> + * Given TCM bank entry, this callback will set device address for R5
> >> + * running on TCM and also setup virtual address for TCM bank
> >> + * remoteproc carveout.
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int tcm_mem_map(struct rproc *rproc,
> >> + struct rproc_mem_entry *mem)
> >> +{
> >> + void __iomem *va;
> >> +
> >> + va = ioremap_wc(mem->dma, mem->len);
> >> + if (IS_ERR_OR_NULL(va))
> >> + return -ENOMEM;
> >> +
> >> + /* Update memory entry va */
> >> + mem->va = (void *)va;
> >> +
> >> + /* clear TCMs */
> >> + memset_io(va, 0, mem->len);
> >> +
> >> + /*
> >> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> >> + * while on the Linux side they are at 0xffexxxxx.
> >> + *
> >> + * Zero out the high 12 bits of the address. This will give
> >> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> >> + */
> >> + mem->da &= 0x000fffff;
> >> +
> >> + /*
> >> + * TCM Banks 1A and 1B still have to be translated.
> >> + *
> >> + * Below handle these two banks' absolute addresses (0xffe90000 and
> >> + * 0xffeb0000) and convert to the expected relative addresses
> >> + * (0x0 and 0x20000).
> >> + */
> >> + if (mem->da == 0x90000 || mem->da == 0xB0000)
> >> + mem->da -= 0x90000;
> >> +
> >> + /* if translated TCM bank address is not valid report error */
> >> + if (mem->da != 0x0 && mem->da != 0x20000) {
> >> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
> >> + return -EINVAL;
> >> + }
> >> + return 0;
> >> +}
> >> +
> >> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
> >> +{
> >> + int i, num_banks, ret;
> >> + struct rproc_mem_entry *mem;
> >> + enum pm_node_id pm_domain_id;
> >> + u32 bank_addr;
> >> + size_t bank_size = 0;
> >> + char *bank_name;
> >> + struct device *dev;
> >> + struct zynqmp_r5_core *r5_core;
> >> +
> >> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> >> + if (!r5_core)
> >> + return -EINVAL;
> >> +
> >> + dev = r5_core->dev;
> >> +
> >> + /* go through zynqmp banks for r5 node */
> >> + num_banks = r5_core->tcm_bank_count;
> >> + if (num_banks <= 0) {
> >> + dev_err(dev, "need to specify TCM banks\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + for (i = 0; i < num_banks; i++) {
> >> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
> >> + bank_name = r5_core->tcm_banks[i].bank_name;
> >> + bank_size = r5_core->tcm_banks[i].size;
> >> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> >> +
> >> + ret = zynqmp_pm_request_node(pm_domain_id,
> >> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> >> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> >> + if (ret < 0) {
> >> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> >> + return ret;
> >> + }
> >> +
> >> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
> >> + bank_name, bank_addr, bank_size);
> >> +
> >> + /* add carveout */
> >> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> >> + bank_size, bank_addr,
> >> + tcm_mem_map, tcm_mem_unmap,
> >> + bank_name);
> >> + if (IS_ERR_OR_NULL(mem)) {
> >> + /* Turn off all TCM banks turned on before */
> >> + do {
> >> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> >> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> >> + if (ret)
> >> + dev_warn(dev,
> >> + "fail to release node: %x, %x\n",
> >> + (u32)pm_domain_id, ret);
> >> + } while (i--);
> >> + return -ENOMEM;
> >> + }
> >> +
> >> + rproc_add_carveout(rproc, mem);
> >> + }
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> >> +{
> >> + int i, num_banks, ret;
> >> + struct rproc_mem_entry *mem;
> >> + enum pm_node_id pm_domain_id;
> >> + u32 bank_addr;
> >> + size_t bank_size = 0;
> >> + char *bank_name;
> >> + struct device *dev;
> >> + struct platform_device *parent_pdev;
> >> + struct zynqmp_r5_cluster *cluster;
> >> + struct zynqmp_r5_core *r5_core;
> >> +
> >> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> >> + if (!r5_core)
> >> + return -EINVAL;
> >> +
> >> + dev = r5_core->dev;
> >> + if (!dev) {
> >> + pr_err("r5 core device unavailable\n");
> >> + return -ENODEV;
> >> + }
> >> +
> >> + /* go through zynqmp banks for r5 node */
> >> + num_banks = r5_core->tcm_bank_count;
> >> + if (num_banks <= 0) {
> >> + dev_err(dev, "need to specify TCM banks\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
> >> + bank_name = r5_core->tcm_banks[0].bank_name;
> >> + for (i = 0; i < num_banks; i++) {
> >> + bank_size += r5_core->tcm_banks[i].size;
> >> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> >> +
> >> + ret = zynqmp_pm_request_node(pm_domain_id,
> >> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> >> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> >> + if (ret < 0) {
> >> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> >> + return ret;
> >> + }
> >> + }
> >> +
> >> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
> >> + bank_name, bank_addr, bank_size);
> >> +
> >> + /* add carveout */
> >> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> >> + bank_size, bank_addr,
> >> + tcm_mem_map, tcm_mem_unmap,
> >> + bank_name);
> >> + if (IS_ERR_OR_NULL(mem)) {
> >> + for (i = 0; i < num_banks; i++) {
> >> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> >> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> >> + if (ret)
> >> + dev_warn(dev,
> >> + "fail to release node: %x ret: %x\n",
> >> + (u32)pm_domain_id, ret);
> >> + }
> >> + return -ENOMEM;
> >> + }
> >> +
> >> + rproc_add_carveout(rproc, mem);
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +/*
> >> + * add_tcm_banks()
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + *
> >> + * Given R5 node in remoteproc instance
> >> + * allocate remoteproc carveout for TCM memory
> >> + * needed for firmware to be loaded
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int add_tcm_banks(struct rproc *rproc)
> >> +{
> >> + struct device *dev;
> >> + struct platform_device *parent_pdev;
> >> + struct zynqmp_r5_cluster *cluster;
> >> + struct zynqmp_r5_core *r5_core;
> >> +
> >> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> >> + if (!r5_core)
> >> + return -EINVAL;
> >> +
> >> + dev = r5_core->dev;
> >> + if (!dev) {
> >> + pr_err("r5 core device unavailable\n");
> >> + return -ENODEV;
> >> + }
> >> +
> >> + parent_pdev = to_platform_device(dev->parent);
> >> + if (!parent_pdev) {
> >> + dev_err(dev, "parent platform dev unavailable\n");
> >> + return -ENODEV;
> >> + }
> >> +
> >> + cluster = platform_get_drvdata(parent_pdev);
> >> + if (!cluster) {
> >> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + if (cluster->mode == SPLIT_MODE)
> >> + return add_tcm_carveout_split_mode(rproc);
> >> + else if (cluster->mode == LOCKSTEP_MODE)
> >> + return add_tcm_carveout_lockstep_mode(rproc);
> >> +
> >> + dev_err(cluster->dev, "invalid cluster mode\n");
> >> + return -EINVAL;
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_parse_fw()
> >> + * @rproc: single R5 core's corresponding rproc instance
> >> + * @fw: ptr to firmware to be loaded onto r5 core
> >> + *
> >> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> >> + *
> >> + * return 0 on success, otherwise non-zero value on failure
> >> + */
> >> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> >> +{
> >> + int ret;
> >> + struct zynqmp_r5_core *r5_core;
> >> + struct device *dev;
> >> +
> >> + r5_core = rproc->priv;
> >> + if (!r5_core) {
> >> + dev_err(&rproc->dev, "r5 core not available\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + dev = r5_core->dev;
> >> +
> >> + ret = add_tcm_banks(rproc);
> >> + if (ret) {
> >> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
> >> + return ret;
> >> + }
> >> +
> >> + ret = add_mem_regions(rproc);
> >> + if (ret)
> >> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
> >> +
> >> + ret = rproc_elf_load_rsc_table(rproc, fw);
> >> + if (ret == -EINVAL) {
> >> + /*
> >> + * resource table only required for IPC.
> >> + * if not present, this is not necessarily an error;
> >> + * for example, loading r5 hello world application
> >> + * so simply inform user and keep going.
> >> + */
> >> + dev_info(&rproc->dev, "no resource table found.\n");
> >> + ret = 0;
> >> + }
> >> + return ret;
> >> +}
> >> +
> >> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> >> + .start = zynqmp_r5_rproc_start,
> >> + .stop = zynqmp_r5_rproc_stop,
> >> + .load = rproc_elf_load_segments,
> >> + .parse_fw = zynqmp_r5_parse_fw,
> >> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> >> + .sanity_check = rproc_elf_sanity_check,
> >> + .get_boot_addr = rproc_elf_get_boot_addr,
> >> +};
> >> +
> >> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
> >> +{
> >> + int i, j, k;
> >> + struct zynqmp_r5_core *r5_core;
> >> +
> >> + dev_dbg(cluster->dev, "Printing dt node info\n");
> >> +
> >> + pr_debug("cluster mode = %d\n", cluster->mode);
> >> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
> >> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
> >> + pr_debug("r5f num cores = %d\n", cluster->core_count);
> >> +
> >> + for (i = 0; i < cluster->core_count; i++) {
> >> + r5_core = &cluster->r5_cores[i];
> >> + if (!r5_core) {
> >> + pr_err("can't get r5_core\n");
> >> + continue;
> >> + }
> >> +
> >> + pr_debug("r5 core %d nodes\n", i);
> >> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
> >> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
> >> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
> >> + k, r5_core->tcm_banks[k].addr,
> >> + r5_core->tcm_banks[k].size,
> >> + r5_core->tcm_banks[k].pm_domain_id,
> >> + r5_core->tcm_banks[k].bank_name);
> >> + }
> >> +
> >> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
> >> +
> >> + for (j = 0; j < r5_core->res_mem_count; j++) {
> >> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
> >> + j, r5_core->res_mem[j].base,
> >> + r5_core->res_mem[j].size,
> >> + r5_core->res_mem[j].name);
> >> + }
> >> + }
> >> +}
> >> +
> >> +/**
> >> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
> >> + * this is called for each individual R5 core to
> >> + * set up mailbox, Xilinx platform manager unique ID,
> >> + * add to rproc core
> >> + *
> >> + * @r5_core: zynqmp_r5_core r5 core object to initialize
> >> + *
> >> + * Return: 0 for success, negative value for failure.
> >> + */
> >> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
> >> +{
> >> + int ret;
> >> + struct rproc *r5_rproc;
> >> + struct device *dev;
> >> +
> >> + dev = r5_core->dev;
> >> +
> >> + /* Set up DMA mask */
> >> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> >> + if (ret)
> >> + return ret;
> >> +
> >> + /* Allocate remoteproc instance */
> >> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> >> + NULL, sizeof(struct zynqmp_r5_core));
> >> + if (IS_ERR_OR_NULL(r5_rproc))
> >> + return -ENOMEM;
> >> +
> >> + r5_rproc->auto_boot = false;
> >> + r5_rproc->priv = r5_core;
> >> +
> >> + /* Add R5 remoteproc */
> >> + ret = devm_rproc_add(dev, r5_rproc);
> >> + if (ret) {
> >> + pr_err("failed to add r5 remoteproc\n");
> >> + return ret;
> >> + }
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> >> +{
> >> + int tcm_bank_count, tcm_node;
> >> + int i = 0, j;
> >> + struct zynqmp_r5_core *r5_core;
> >> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> >> + struct device *dev = cluster->dev;
> >> +
> >> + /* ToDo: Use predefined TCM address space values from driver until
> >> + * system-dt spec is not final fot TCM
> >> + */
> >> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> >> +
> >> + /* count per core tcm banks */
> >> + tcm_bank_count = tcm_bank_count / cluster->core_count;
> >> +
> >> + /* r5 core 0 will use all of TCM banks in lockstep mode.
> >> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> >> + * 128k. Assign TCM banks to each core accordingly
> >> + */
> >> + tcm_node = 0;
> >> + for (j = 0; j < cluster->core_count; j++) {
> >> + r5_core = &cluster->r5_cores[j];
> >> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> >> + tcm_bank_count, GFP_KERNEL);
> >> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> >> + return -ENOMEM;
> >> +
> >> + for (i = 0; i < tcm_bank_count; i++) {
> >> + /* Use pre-defined TCM reg values.
> >> + * Eventually this should be replaced by values
> >> + * parsed from dts.
> >> + */
> >> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> >> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> >> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> >> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> >> + tcm_node++;
> >> + }
> >> +
> >> + r5_core->tcm_bank_count = tcm_bank_count;
> >> + }
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> >> +{
> >> + int res_mem_count, i;
> >> + struct device *dev;
> >> + struct device_node *np, *rmem_np;
> >> + struct reserved_mem *rmem;
> >> +
> >> + dev = r5_core->dev;
> >> +
> >> + np = r5_core->np;
> >> + if (IS_ERR_OR_NULL(np)) {
> >> + pr_err("invalid device node of r5 core\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> >> + sizeof(phandle));
> >> + if (res_mem_count <= 0) {
> >> + dev_warn(dev, "failed to get memory-region property %d\n",
> >> + res_mem_count);
> >> + return -EINVAL;
> >> + }
> >> +
> >> + r5_core->res_mem = devm_kzalloc(dev,
> >> + res_mem_count * sizeof(struct reserved_mem),
> >> + GFP_KERNEL);
> >> + if (!r5_core->res_mem) {
> >> + dev_err(dev, "failed to allocate mem region memory\n");
> >> + return -ENOMEM;
> >> + }
> >> +
> >> + for (i = 0; i < res_mem_count; i++) {
> >> + rmem_np = of_parse_phandle(np, "memory-region", i);
> >> + if (!rmem_np)
> >> + return -EINVAL;
> >> +
> >> + rmem = of_reserved_mem_lookup(rmem_np);
> >> + if (!rmem) {
> >> + of_node_put(rmem_np);
> >> + return -EINVAL;
> >> + }
> >> +
> >> + memcpy(&r5_core->res_mem[i], rmem,
> >> + sizeof(struct reserved_mem));
> >> + of_node_put(rmem_np);
> >> + }
> >> +
> >> + r5_core->res_mem_count = res_mem_count;
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> >> +{
> >> + int ret, i;
> >> + struct zynqmp_r5_core *r5_core;
> >> + struct device *dev = cluster->dev;
> >> +
> >> + ret = zynqmp_r5_get_tcm_node(cluster);
> >> + if (ret < 0) {
> >> + dev_err(dev, "can't get tcm node, err %d\n", ret);
> >> + return ret;
> >> + }
> >> +
> >> + for (i = 0; i < cluster->core_count; i++) {
> >> + r5_core = &cluster->r5_cores[i];
> >> + if (!r5_core) {
> >> + pr_err("invalid r5 core\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + ret = zynqmp_r5_get_mem_region_node(r5_core);
> >> + if (ret)
> >> + dev_warn(dev, "memory-region prop failed %d\n", ret);
> >> +
> >> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> >> + 1, &r5_core->pm_domain_id);
> >> + if (ret) {
> >> + dev_err(dev, "failed to get power-domains property\n");
> >> + return ret;
> >> + }
> >> +
> >> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> >> + if (ret)
> >> + return ret;
> >> +
> >> + ret = zynqmp_r5_add_rproc_core(r5_core);
> >> + if (ret) {
> >> + dev_err(dev, "failed to init r5 core %d\n", i);
> >> + return ret;
> >> + }
> >> + }
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> >> +{
> >> + struct device *dev = cluster->dev;
> >> + struct device_node *dev_node = dev_of_node(dev);
> >> + struct device_node *child;
> >> + struct platform_device *child_pdev;
> >> + int core_count = 0, ret, i;
> >> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> >> + struct zynqmp_r5_core *r5_cores;
> >> +
> >> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> >> +
> >> + /* on success returns 0, if not defined then returns -EINVAL,
> >> + * In that case, default is LOCKSTEP mode
> >> + */
> >> + if (ret != -EINVAL && ret != 0) {
> >> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + if (cluster_mode == SINGLE_CPU_MODE) {
> >> + dev_err(dev, "driver does not support single cpu mode\n");
> >> + return -EINVAL;
> >> + } else if ((cluster_mode != SPLIT_MODE &&
> >> + cluster_mode != LOCKSTEP_MODE)) {
> >> + dev_err(dev, "Invalid cluster mode\n");
> >> + return -EINVAL;
> >> + }
> >> +
> >> + core_count = of_get_available_child_count(dev_node);
> >> + if (core_count <= 0) {
> >> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> >> + return -EINVAL;
> >> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> >> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> >> + return -EINVAL;
> >> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> >> + dev_warn(dev, "Only r5 core0 will be used\n");
> >> + core_count = 1;
> >> + }
> >> +
> >> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> >> + core_count, GFP_KERNEL);
> >> + if (IS_ERR_OR_NULL(r5_cores)) {
> >> + dev_err(dev, "can't allocate memory for cores\n");
> >> + return -ENOMEM;
> >> + }
> >> +
> >> + i = 0;
> >> + for_each_available_child_of_node(dev_node, child) {
> >> + child_pdev = of_find_device_by_node(child);
> >> + if (!child_pdev)
> >> + return -ENODEV;
> >> +
> >> + r5_cores[i].dev = &child_pdev->dev;
> >> + if (!r5_cores[i].dev) {
> >> + pr_err("can't get device for r5 core %d\n", i);
> >> + return -ENODEV;
> >> + }
> >> +
> >> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> >> + if (!r5_cores[i].np) {
> >> + pr_err("can't get device node for r5 core %d\n", i);
> >> + return -ENODEV;
> >> + }
> >> +
> >> + i++;
> >> + if (i == core_count)
> >> + break;
> >> + }
> >> +
> >> + cluster->mode = cluster_mode;
> >> + cluster->core_count = core_count;
> >> + cluster->r5_cores = r5_cores;
> >> +
> >> + ret = zynqmp_r5_core_init(cluster);
> >> + if (ret < 0) {
> >> + dev_err(dev, "failed to init r5 core err %d\n", ret);
> >> + return ret;
> >> + }
> >> +
> >> + zynqmp_r5_print_dt_node_info(cluster);
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +static void zynqmp_r5_cluster_exit(void *data)
> >> +{
> >> + struct platform_device *pdev = (struct platform_device *)data;
> >> +
> >> + platform_set_drvdata(pdev, NULL);
> >> +
> >> + pr_info("Exit r5f subsystem driver\n");
> >> +}
> >> +
> >> +/*
> >> + * zynqmp_r5_remoteproc_probe()
> >> + *
> >> + * @pdev: domain platform device for R5 cluster
> >> + *
> >> + * called when driver is probed, for each R5 core specified in DT,
> >> + * setup as needed to do remoteproc-related operations
> >> + *
> >> + * Return: 0 for success, negative value for failure.
> >> + */
> >> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> >> +{
> >> + int ret;
> >> + struct zynqmp_r5_cluster *cluster;
> >> + struct device *dev = &pdev->dev;
> >> +
> >> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> >> + if (IS_ERR_OR_NULL(cluster))
> >> + return -ENOMEM;
> >> +
> >> + cluster->dev = dev;
> >> +
> >> + ret = devm_of_platform_populate(dev);
> >> + if (ret) {
> >> + dev_err(dev, "failed to populate platform dev %d\n", ret);
> >> + return ret;
> >> + }
> >> +
> >> + /* wire in so each core can be cleaned up at driver remove */
> >> + platform_set_drvdata(pdev, cluster);
> >> +
> >> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> >> + if (ret)
> >> + return ret;
> >> +
> >> + ret = zynqmp_r5_cluster_init(cluster);
> >> + if (ret) {
> >> + dev_err(dev, "Invalid r5f subsystem device tree\n");
> >> + return ret;
> >> + }
> >> +
> >> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> >> + return 0;
> >> +}
> >> +
> >> +/* Match table for OF platform binding */
> >> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> >> + { .compatible = "xlnx,zynqmp-r5fss", },
> >> + { /* end of list */ },
> >> +};
> >> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> >> +
> >> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> >> + .probe = zynqmp_r5_remoteproc_probe,
> >> + .driver = {
> >> + .name = "zynqmp_r5_remoteproc",
> >> + .of_match_table = zynqmp_r5_remoteproc_match,
> >> + },
> >> +};
> >> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> >> +
> >> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> >> +MODULE_AUTHOR("Xilinx Inc.");
> >> +MODULE_LICENSE("GPL v2");
> >> --
> >> 2.25.1
> >>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-11-30 20:14 ` Mathieu Poirier
@ 2021-12-01 18:37 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-01 18:37 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On 12/1/21 1:44 AM, Mathieu Poirier wrote:
> On Tue, 30 Nov 2021 at 01:01, Tanmay Shah <tanmay.shah@xilinx.com> wrote:
>>
>> On 11/30/21 12:12 AM, Mathieu Poirier wrote:
>>> On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
>>>> This driver enables r5f dual core Real time Processing Unit subsystem
>>>> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
>>>> (cluster) can be configured in different modes e.g. split mode in which
>>>> two r5f cores work independent of each other and lock-step mode in which
>>>> both r5f cores execute same code clock-for-clock and notify if the
>>>> result is different.
>>>>
>>>> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
>>>> Platform Management Unit that handles the R5 configuration, memory access
>>>> and R5 lifecycle management. The interface to this manager is done in this
>>>> driver via zynqmp_pm_* function calls.
>>>>
>>>> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
>>>> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
>>>> ---
>>>> drivers/remoteproc/Kconfig | 12 +
>>>> drivers/remoteproc/Makefile | 1 +
>>>> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
>>>> 3 files changed, 972 insertions(+)
>>>> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>>> ...and this patch gives me complation warnings:
>>>
>>> CC drivers/remoteproc/xlnx_r5_remoteproc.o
>>> kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c: In function ‘add_tcm_carveout_lockstep_mode’:
>>> kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:412:28: warning: unused variable ‘cluster’ [-Wunused-variable]
>>> 412 | struct zynqmp_r5_cluster *cluster;
>>> | ^~~~~~~
>>> kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:411:26: warning: unused variable ‘parent_pdev’ [-Wunused-variable]
>>> 411 | struct platform_device *parent_pdev;
>>> | ^~~~~~~~~~~
>>>
>>> The above leads me to believe this patchset was not compiled before it was sent
>>> out.
>> Please don't assume that this patch-set was not compiled. This driver
>> was compiled and tested on Xilinx QEMU and zynqmp platform for its
>> functionality.
>>
>> This driver went through multiple internal reviews and I had to
>> re-architecture it multiple times.
> And yet two unused variables (which is pretty easy to spot) got by.
>
>> I simply missed to fix above warnings before submitting driver. I have
>> compiled driver with following command:
>>
>> //make ARCH=arm64 W=1 C=1 CROSS_COMPILE="aarch64-linux-gnu-" -j32 -Rr
>> O=$zynqmp_kernel_build/rproc-next
>>
>> I did my best to make sure driver stays warning free, however few
>> warnings were still missed. That is not intentional and by mistake. I
>> fully intend to comply with Linux Kernel community guideline and
>> checklist before submitting patches.
>>
> I am sure you do.
>
>> Apart from above warnings, two more warnings are there in v2 i.e. due to
>> typecast issue between (void __iomem *) and (void *). But, I had not
>> solution before. Also they were used before in different driver so, I
>> chose to use them anyway.
>>
> The sparse warnings related to the "__iomem *" can be ignored. I also
> tried to find a solution to those but realised it was more involved
> than expected.
>
>> I just found that using memremap set of functions will fix them. I will
>> use those functions in next patch set instead of ioremap_wc as explained
>> in this article: https://lwn.net/Articles/653585/
>>
> This is very interesting...
>
>>> Being new to this I can understand that checkpatch.pl was omitted (albeit amply
>>> documented) but obvious compilation warnings can't be excused. As such I
>>> am dropping this set and will not review another version until January.
>>
>> I understand having warnings in driver may lead to frustration and may
>> cause trust issues for the rest of the code and it can't be excused at all.
>>
> Very much so.
>
>> I am aware of guidelines of sending patches upstream
>> (https://www.kernel.org/doc/html/latest/process/submitting-patches.html)
>> and fully intend to comply with that and like I said, if I miss
>> something it is not by intention but human error.
>>
>> I constantly try to improve processes to upstream patches so we don't
>> face above type of issues.
>>
>> I did run checkpatch.pl as per best of my knowledge and fixed lot of
>> style related warnings reported by it. Also I enabled W=1 option in my
>> compilation command and fixed lots of warnings reported by compiler too.
>> As I explained earlier, I wasn't aware of unused-variable warnings, and
>> so I missed to fix them.
>>
> Ok
>
>> With this, I request to review driver from functionality point of view
>> as well along with style errors. So, I can address more number of
>> comments / concerns in less number of patches. If you want these
>> warnings to be fixed, I will send v3 and you can put more comments on
>> v3. However, I highly appreciate if we can continue reviews and not
>> postpone till January. Please let me know your thoughts.
>>
> I will give this patchset another chance later this week.
Thanks for this consideration.
>
>> Thanks,
>>
>> Tanmay
>>
>>
>>> Mathieu
>>
>>>> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
>>>> index f30d00a3aabe..27f66910d8d3 100644
>>>> --- a/drivers/remoteproc/Kconfig
>>>> +++ b/drivers/remoteproc/Kconfig
>>>> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
>>>> It's safe to say N here if you're not interested in utilizing
>>>> a slave processor.
>>>>
>>>> +config XLNX_R5_REMOTEPROC
>>>> + tristate "Xilinx R5 remoteproc support"
>>>> + depends on PM && ARCH_ZYNQMP
>>>> + depends on ZYNQMP_FIRMWARE
>>>> + select RPMSG_VIRTIO
>>>> + select ZYNQMP_IPI_MBOX
>>>> + help
>>>> + Say y or m here to support Xilinx R5 remote processors via the remote
>>>> + processor framework.
>>>> +
>>>> + It's safe to say N if not interested in using RPU r5f cores.
>>>> +
>>>> endif # REMOTEPROC
>>>>
>>>> endmenu
>>>> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
>>>> index bb26c9e4ef9c..334a8bed4c14 100644
>>>> --- a/drivers/remoteproc/Makefile
>>>> +++ b/drivers/remoteproc/Makefile
>>>> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
>>>> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
>>>> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
>>>> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
>>>> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
>>>> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
>>>> new file mode 100644
>>>> index 000000000000..c2167fd3869d
>>>> --- /dev/null
>>>> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
>>>> @@ -0,0 +1,959 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ZynqMP R5 Remote Processor driver
>>>> + *
>>>> + */
>>>> +
>>>> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
>>>> +#include <linux/firmware/xlnx-zynqmp.h>
>>>> +#include <linux/interrupt.h>
>>>> +#include <linux/kernel.h>
>>>> +#include <linux/list.h>
>>>> +#include <linux/mailbox_client.h>
>>>> +#include <linux/mailbox/zynqmp-ipi-message.h>
>>>> +#include <linux/module.h>
>>>> +#include <linux/of_address.h>
>>>> +#include <linux/of_platform.h>
>>>> +#include <linux/of_reserved_mem.h>
>>>> +#include <linux/platform_device.h>
>>>> +#include <linux/remoteproc.h>
>>>> +#include <linux/skbuff.h>
>>>> +#include <linux/sysfs.h>
>>>> +
>>>> +#include "remoteproc_internal.h"
>>>> +
>>>> +/* settings for RPU cluster mode */
>>>> +enum zynqmp_r5_cluster_mode {
>>>> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
>>>> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
>>>> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct mem_bank_data - Memory Bank description
>>>> + *
>>>> + * @addr: Start address of memory bank
>>>> + * @size: Size of Memory bank
>>>> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
>>>> + * @bank_name: name of the bank for remoteproc framework
>>>> + */
>>>> +struct mem_bank_data {
>>>> + phys_addr_t addr;
>>>> + size_t size;
>>>> + enum pm_node_id pm_domain_id;
>>>> + char *bank_name;
>>>> +};
>>>> +
>>>> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
>>>> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
>>>> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
>>>> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
>>>> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct zynqmp_r5_core - ZynqMP R5 core structure
>>>> + *
>>>> + * @dev: device of RPU instance
>>>> + * @np: device node of RPU instance
>>>> + * @tcm_bank_count: number TCM banks accessible to this RPU
>>>> + * @tcm_banks: array of each TCM bank data
>>>> + * @res_mem_count: number of Reserved Memory regions per core
>>>> + * @res_mem: array of reserved memory regions
>>>> + * @rproc: rproc handle
>>>> + * @pm_domain_id: RPU CPU power domain id
>>>> + */
>>>> +struct zynqmp_r5_core {
>>>> + struct device *dev;
>>>> + struct device_node *np;
>>>> + int tcm_bank_count;
>>>> + struct mem_bank_data *tcm_banks;
>>>> + int res_mem_count;
>>>> + struct reserved_mem *res_mem;
>>>> + struct rproc *rproc;
>>>> + enum pm_node_id pm_domain_id;
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
>>>> + *
>>>> + * @dev: r5f subsystem cluster device node
>>>> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
>>>> + * @core_count: number of r5 cores used for this cluster mode
>>>> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
>>>> + */
>>>> +struct zynqmp_r5_cluster {
>>>> + struct device *dev;
>>>> + enum zynqmp_r5_cluster_mode mode;
>>>> + int core_count;
>>>> + struct zynqmp_r5_core *r5_cores;
>>>> +};
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_set_mode - set RPU operation mode
>>>> + *
>>>> + * set RPU operation mode
>>>> + *
>>>> + * Return: 0 for success, negative value for failure
>>>> + */
>>>> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
>>>> + enum zynqmp_r5_cluster_mode rpu_mode)
>>>> +{
>>>> + enum rpu_tcm_comb tcm_mode;
>>>> + int ret, reg_val;
>>>> +
>>>> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
>>>> +
>>>> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
>>>> + if (ret < 0) {
>>>> + pr_err("failed to set RPU mode\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
>>>> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
>>>> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
>>>> + if (ret < 0)
>>>> + pr_err("failed to configure TCM\n");
>>>> +
>>>> + return ret;
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_rproc_start
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + *
>>>> + * Start R5 Core from designated boot address.
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>>>> +{
>>>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>>>> + enum rpu_boot_mem bootmem;
>>>> + int ret;
>>>> +
>>>> + if (!r5_core) {
>>>> + pr_err("can't get r5 core\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>>>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
>>>> +
>>>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>>>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>>>> +
>>>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>>>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>>>> + if (ret)
>>>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>>>> + return ret;
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_rproc_stop
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + *
>>>> + * Power down R5 Core.
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>>>> +{
>>>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>>>> + int ret;
>>>> +
>>>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>>>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>>>> + if (ret)
>>>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>>>> +
>>>> + return ret;
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_rproc_mem_map
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + * @mem: mem entry to map
>>>> + *
>>>> + * Callback to map va for memory-region's carveout.
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>>>> + struct rproc_mem_entry *mem)
>>>> +{
>>>> + void __iomem *va;
>>>> +
>>>> + va = ioremap_wc(mem->dma, mem->len);
>>>> + if (IS_ERR_OR_NULL(va))
>>>> + return -ENOMEM;
>>>> +
>>>> + mem->va = (void *)va;
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_rproc_mem_unmap
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + * @mem: mem entry to unmap
>>>> + *
>>>> + * Unmap memory-region carveout
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>>>> + struct rproc_mem_entry *mem)
>>>> +{
>>>> + iounmap((void __iomem *)mem->va);
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/*
>>>> + * add_mem_regions
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + *
>>>> + * Construct rproc mem carveouts from carveout provided in
>>>> + * memory-region property
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int add_mem_regions(struct rproc *rproc)
>>>> +{
>>>> + struct device *dev;
>>>> + struct rproc_mem_entry *mem;
>>>> + struct reserved_mem *rmem;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + int i;
>>>> +
>>>> + r5_core = rproc->priv;
>>>> + dev = r5_core->dev;
>>>> +
>>>> + /* Register associated reserved memory regions */
>>>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>>>> + rmem = &r5_core->res_mem[i];
>>>> + mem = rproc_mem_entry_init(dev, NULL,
>>>> + (dma_addr_t)rmem->base,
>>>> + rmem->size, rmem->base,
>>>> + zynqmp_r5_rproc_mem_map,
>>>> + zynqmp_r5_rproc_mem_unmap,
>>>> + rmem->name);
>>>> + if (IS_ERR_OR_NULL(mem))
>>>> + return -ENOMEM;
>>>> +
>>>> + rproc_add_carveout(rproc, mem);
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_rproc_mem_unmap
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + * @mem: mem entry to unmap
>>>> + *
>>>> + * Unmap TCM banks when powering down R5 core.
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>>>> +{
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + int i;
>>>> + enum pm_node_id pm_domain_id;
>>>> +
>>>> + r5_core = rproc->priv;
>>>> + if (!r5_core) {
>>>> + pr_err("r5 core is not available\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + iounmap((void __iomem *)mem->va);
>>>> +
>>>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>>>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>>>> + if (zynqmp_pm_release_node(pm_domain_id))
>>>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/*
>>>> + * tcm_mem_map
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + * @mem: mem entry to initialize the va and da fields of
>>>> + *
>>>> + * Given TCM bank entry, this callback will set device address for R5
>>>> + * running on TCM and also setup virtual address for TCM bank
>>>> + * remoteproc carveout.
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int tcm_mem_map(struct rproc *rproc,
>>>> + struct rproc_mem_entry *mem)
>>>> +{
>>>> + void __iomem *va;
>>>> +
>>>> + va = ioremap_wc(mem->dma, mem->len);
>>>> + if (IS_ERR_OR_NULL(va))
>>>> + return -ENOMEM;
>>>> +
>>>> + /* Update memory entry va */
>>>> + mem->va = (void *)va;
>>>> +
>>>> + /* clear TCMs */
>>>> + memset_io(va, 0, mem->len);
>>>> +
>>>> + /*
>>>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>>>> + * while on the Linux side they are at 0xffexxxxx.
>>>> + *
>>>> + * Zero out the high 12 bits of the address. This will give
>>>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>>>> + */
>>>> + mem->da &= 0x000fffff;
>>>> +
>>>> + /*
>>>> + * TCM Banks 1A and 1B still have to be translated.
>>>> + *
>>>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>>>> + * 0xffeb0000) and convert to the expected relative addresses
>>>> + * (0x0 and 0x20000).
>>>> + */
>>>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>>>> + mem->da -= 0x90000;
>>>> +
>>>> + /* if translated TCM bank address is not valid report error */
>>>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>>>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>>>> + return -EINVAL;
>>>> + }
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>>>> +{
>>>> + int i, num_banks, ret;
>>>> + struct rproc_mem_entry *mem;
>>>> + enum pm_node_id pm_domain_id;
>>>> + u32 bank_addr;
>>>> + size_t bank_size = 0;
>>>> + char *bank_name;
>>>> + struct device *dev;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> +
>>>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>>>> + if (!r5_core)
>>>> + return -EINVAL;
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + /* go through zynqmp banks for r5 node */
>>>> + num_banks = r5_core->tcm_bank_count;
>>>> + if (num_banks <= 0) {
>>>> + dev_err(dev, "need to specify TCM banks\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + for (i = 0; i < num_banks; i++) {
>>>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>>>> + bank_name = r5_core->tcm_banks[i].bank_name;
>>>> + bank_size = r5_core->tcm_banks[i].size;
>>>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>>>> +
>>>> + ret = zynqmp_pm_request_node(pm_domain_id,
>>>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>>>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>>>> + if (ret < 0) {
>>>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>>>> + bank_name, bank_addr, bank_size);
>>>> +
>>>> + /* add carveout */
>>>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>>>> + bank_size, bank_addr,
>>>> + tcm_mem_map, tcm_mem_unmap,
>>>> + bank_name);
>>>> + if (IS_ERR_OR_NULL(mem)) {
>>>> + /* Turn off all TCM banks turned on before */
>>>> + do {
>>>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>>>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>>>> + if (ret)
>>>> + dev_warn(dev,
>>>> + "fail to release node: %x, %x\n",
>>>> + (u32)pm_domain_id, ret);
>>>> + } while (i--);
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + rproc_add_carveout(rproc, mem);
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>>>> +{
>>>> + int i, num_banks, ret;
>>>> + struct rproc_mem_entry *mem;
>>>> + enum pm_node_id pm_domain_id;
>>>> + u32 bank_addr;
>>>> + size_t bank_size = 0;
>>>> + char *bank_name;
>>>> + struct device *dev;
>>>> + struct platform_device *parent_pdev;
>>>> + struct zynqmp_r5_cluster *cluster;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> +
>>>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>>>> + if (!r5_core)
>>>> + return -EINVAL;
>>>> +
>>>> + dev = r5_core->dev;
>>>> + if (!dev) {
>>>> + pr_err("r5 core device unavailable\n");
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + /* go through zynqmp banks for r5 node */
>>>> + num_banks = r5_core->tcm_bank_count;
>>>> + if (num_banks <= 0) {
>>>> + dev_err(dev, "need to specify TCM banks\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>>>> + bank_name = r5_core->tcm_banks[0].bank_name;
>>>> + for (i = 0; i < num_banks; i++) {
>>>> + bank_size += r5_core->tcm_banks[i].size;
>>>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>>>> +
>>>> + ret = zynqmp_pm_request_node(pm_domain_id,
>>>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>>>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>>>> + if (ret < 0) {
>>>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>>>> + return ret;
>>>> + }
>>>> + }
>>>> +
>>>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>>>> + bank_name, bank_addr, bank_size);
>>>> +
>>>> + /* add carveout */
>>>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>>>> + bank_size, bank_addr,
>>>> + tcm_mem_map, tcm_mem_unmap,
>>>> + bank_name);
>>>> + if (IS_ERR_OR_NULL(mem)) {
>>>> + for (i = 0; i < num_banks; i++) {
>>>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>>>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>>>> + if (ret)
>>>> + dev_warn(dev,
>>>> + "fail to release node: %x ret: %x\n",
>>>> + (u32)pm_domain_id, ret);
>>>> + }
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + rproc_add_carveout(rproc, mem);
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/*
>>>> + * add_tcm_banks()
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + *
>>>> + * Given R5 node in remoteproc instance
>>>> + * allocate remoteproc carveout for TCM memory
>>>> + * needed for firmware to be loaded
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int add_tcm_banks(struct rproc *rproc)
>>>> +{
>>>> + struct device *dev;
>>>> + struct platform_device *parent_pdev;
>>>> + struct zynqmp_r5_cluster *cluster;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> +
>>>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>>>> + if (!r5_core)
>>>> + return -EINVAL;
>>>> +
>>>> + dev = r5_core->dev;
>>>> + if (!dev) {
>>>> + pr_err("r5 core device unavailable\n");
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + parent_pdev = to_platform_device(dev->parent);
>>>> + if (!parent_pdev) {
>>>> + dev_err(dev, "parent platform dev unavailable\n");
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + cluster = platform_get_drvdata(parent_pdev);
>>>> + if (!cluster) {
>>>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + if (cluster->mode == SPLIT_MODE)
>>>> + return add_tcm_carveout_split_mode(rproc);
>>>> + else if (cluster->mode == LOCKSTEP_MODE)
>>>> + return add_tcm_carveout_lockstep_mode(rproc);
>>>> +
>>>> + dev_err(cluster->dev, "invalid cluster mode\n");
>>>> + return -EINVAL;
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_parse_fw()
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + * @fw: ptr to firmware to be loaded onto r5 core
>>>> + *
>>>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>>>> +{
>>>> + int ret;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + struct device *dev;
>>>> +
>>>> + r5_core = rproc->priv;
>>>> + if (!r5_core) {
>>>> + dev_err(&rproc->dev, "r5 core not available\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + ret = add_tcm_banks(rproc);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + ret = add_mem_regions(rproc);
>>>> + if (ret)
>>>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>>>> +
>>>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>>>> + if (ret == -EINVAL) {
>>>> + /*
>>>> + * resource table only required for IPC.
>>>> + * if not present, this is not necessarily an error;
>>>> + * for example, loading r5 hello world application
>>>> + * so simply inform user and keep going.
>>>> + */
>>>> + dev_info(&rproc->dev, "no resource table found.\n");
>>>> + ret = 0;
>>>> + }
>>>> + return ret;
>>>> +}
>>>> +
>>>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>>>> + .start = zynqmp_r5_rproc_start,
>>>> + .stop = zynqmp_r5_rproc_stop,
>>>> + .load = rproc_elf_load_segments,
>>>> + .parse_fw = zynqmp_r5_parse_fw,
>>>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>>>> + .sanity_check = rproc_elf_sanity_check,
>>>> + .get_boot_addr = rproc_elf_get_boot_addr,
>>>> +};
>>>> +
>>>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + int i, j, k;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> +
>>>> + dev_dbg(cluster->dev, "Printing dt node info\n");
>>>> +
>>>> + pr_debug("cluster mode = %d\n", cluster->mode);
>>>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>>>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>>>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>>>> +
>>>> + for (i = 0; i < cluster->core_count; i++) {
>>>> + r5_core = &cluster->r5_cores[i];
>>>> + if (!r5_core) {
>>>> + pr_err("can't get r5_core\n");
>>>> + continue;
>>>> + }
>>>> +
>>>> + pr_debug("r5 core %d nodes\n", i);
>>>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>>>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>>>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>>>> + k, r5_core->tcm_banks[k].addr,
>>>> + r5_core->tcm_banks[k].size,
>>>> + r5_core->tcm_banks[k].pm_domain_id,
>>>> + r5_core->tcm_banks[k].bank_name);
>>>> + }
>>>> +
>>>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>>>> +
>>>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>>>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>>>> + j, r5_core->res_mem[j].base,
>>>> + r5_core->res_mem[j].size,
>>>> + r5_core->res_mem[j].name);
>>>> + }
>>>> + }
>>>> +}
>>>> +
>>>> +/**
>>>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>>>> + * this is called for each individual R5 core to
>>>> + * set up mailbox, Xilinx platform manager unique ID,
>>>> + * add to rproc core
>>>> + *
>>>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>>>> + *
>>>> + * Return: 0 for success, negative value for failure.
>>>> + */
>>>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>>>> +{
>>>> + int ret;
>>>> + struct rproc *r5_rproc;
>>>> + struct device *dev;
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + /* Set up DMA mask */
>>>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + /* Allocate remoteproc instance */
>>>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>>>> + NULL, sizeof(struct zynqmp_r5_core));
>>>> + if (IS_ERR_OR_NULL(r5_rproc))
>>>> + return -ENOMEM;
>>>> +
>>>> + r5_rproc->auto_boot = false;
>>>> + r5_rproc->priv = r5_core;
>>>> +
>>>> + /* Add R5 remoteproc */
>>>> + ret = devm_rproc_add(dev, r5_rproc);
>>>> + if (ret) {
>>>> + pr_err("failed to add r5 remoteproc\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + int tcm_bank_count, tcm_node;
>>>> + int i = 0, j;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>>>> + struct device *dev = cluster->dev;
>>>> +
>>>> + /* ToDo: Use predefined TCM address space values from driver until
>>>> + * system-dt spec is not final fot TCM
>>>> + */
>>>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>>>> +
>>>> + /* count per core tcm banks */
>>>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>>>> +
>>>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>>>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>>>> + * 128k. Assign TCM banks to each core accordingly
>>>> + */
>>>> + tcm_node = 0;
>>>> + for (j = 0; j < cluster->core_count; j++) {
>>>> + r5_core = &cluster->r5_cores[j];
>>>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>>>> + tcm_bank_count, GFP_KERNEL);
>>>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>>>> + return -ENOMEM;
>>>> +
>>>> + for (i = 0; i < tcm_bank_count; i++) {
>>>> + /* Use pre-defined TCM reg values.
>>>> + * Eventually this should be replaced by values
>>>> + * parsed from dts.
>>>> + */
>>>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>>>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>>>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>>>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>>>> + tcm_node++;
>>>> + }
>>>> +
>>>> + r5_core->tcm_bank_count = tcm_bank_count;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>>>> +{
>>>> + int res_mem_count, i;
>>>> + struct device *dev;
>>>> + struct device_node *np, *rmem_np;
>>>> + struct reserved_mem *rmem;
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + np = r5_core->np;
>>>> + if (IS_ERR_OR_NULL(np)) {
>>>> + pr_err("invalid device node of r5 core\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>>>> + sizeof(phandle));
>>>> + if (res_mem_count <= 0) {
>>>> + dev_warn(dev, "failed to get memory-region property %d\n",
>>>> + res_mem_count);
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + r5_core->res_mem = devm_kzalloc(dev,
>>>> + res_mem_count * sizeof(struct reserved_mem),
>>>> + GFP_KERNEL);
>>>> + if (!r5_core->res_mem) {
>>>> + dev_err(dev, "failed to allocate mem region memory\n");
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + for (i = 0; i < res_mem_count; i++) {
>>>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>>>> + if (!rmem_np)
>>>> + return -EINVAL;
>>>> +
>>>> + rmem = of_reserved_mem_lookup(rmem_np);
>>>> + if (!rmem) {
>>>> + of_node_put(rmem_np);
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + memcpy(&r5_core->res_mem[i], rmem,
>>>> + sizeof(struct reserved_mem));
>>>> + of_node_put(rmem_np);
>>>> + }
>>>> +
>>>> + r5_core->res_mem_count = res_mem_count;
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + int ret, i;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + struct device *dev = cluster->dev;
>>>> +
>>>> + ret = zynqmp_r5_get_tcm_node(cluster);
>>>> + if (ret < 0) {
>>>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + for (i = 0; i < cluster->core_count; i++) {
>>>> + r5_core = &cluster->r5_cores[i];
>>>> + if (!r5_core) {
>>>> + pr_err("invalid r5 core\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>>>> + if (ret)
>>>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>>>> +
>>>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>>>> + 1, &r5_core->pm_domain_id);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to get power-domains property\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to init r5 core %d\n", i);
>>>> + return ret;
>>>> + }
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + struct device *dev = cluster->dev;
>>>> + struct device_node *dev_node = dev_of_node(dev);
>>>> + struct device_node *child;
>>>> + struct platform_device *child_pdev;
>>>> + int core_count = 0, ret, i;
>>>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>>>> + struct zynqmp_r5_core *r5_cores;
>>>> +
>>>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>>>> +
>>>> + /* on success returns 0, if not defined then returns -EINVAL,
>>>> + * In that case, default is LOCKSTEP mode
>>>> + */
>>>> + if (ret != -EINVAL && ret != 0) {
>>>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + if (cluster_mode == SINGLE_CPU_MODE) {
>>>> + dev_err(dev, "driver does not support single cpu mode\n");
>>>> + return -EINVAL;
>>>> + } else if ((cluster_mode != SPLIT_MODE &&
>>>> + cluster_mode != LOCKSTEP_MODE)) {
>>>> + dev_err(dev, "Invalid cluster mode\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + core_count = of_get_available_child_count(dev_node);
>>>> + if (core_count <= 0) {
>>>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>>>> + return -EINVAL;
>>>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>>>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>>>> + return -EINVAL;
>>>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>>>> + dev_warn(dev, "Only r5 core0 will be used\n");
>>>> + core_count = 1;
>>>> + }
>>>> +
>>>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>>>> + core_count, GFP_KERNEL);
>>>> + if (IS_ERR_OR_NULL(r5_cores)) {
>>>> + dev_err(dev, "can't allocate memory for cores\n");
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + i = 0;
>>>> + for_each_available_child_of_node(dev_node, child) {
>>>> + child_pdev = of_find_device_by_node(child);
>>>> + if (!child_pdev)
>>>> + return -ENODEV;
>>>> +
>>>> + r5_cores[i].dev = &child_pdev->dev;
>>>> + if (!r5_cores[i].dev) {
>>>> + pr_err("can't get device for r5 core %d\n", i);
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>>>> + if (!r5_cores[i].np) {
>>>> + pr_err("can't get device node for r5 core %d\n", i);
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + i++;
>>>> + if (i == core_count)
>>>> + break;
>>>> + }
>>>> +
>>>> + cluster->mode = cluster_mode;
>>>> + cluster->core_count = core_count;
>>>> + cluster->r5_cores = r5_cores;
>>>> +
>>>> + ret = zynqmp_r5_core_init(cluster);
>>>> + if (ret < 0) {
>>>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + zynqmp_r5_print_dt_node_info(cluster);
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static void zynqmp_r5_cluster_exit(void *data)
>>>> +{
>>>> + struct platform_device *pdev = (struct platform_device *)data;
>>>> +
>>>> + platform_set_drvdata(pdev, NULL);
>>>> +
>>>> + pr_info("Exit r5f subsystem driver\n");
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_remoteproc_probe()
>>>> + *
>>>> + * @pdev: domain platform device for R5 cluster
>>>> + *
>>>> + * called when driver is probed, for each R5 core specified in DT,
>>>> + * setup as needed to do remoteproc-related operations
>>>> + *
>>>> + * Return: 0 for success, negative value for failure.
>>>> + */
>>>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>>>> +{
>>>> + int ret;
>>>> + struct zynqmp_r5_cluster *cluster;
>>>> + struct device *dev = &pdev->dev;
>>>> +
>>>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>>>> + if (IS_ERR_OR_NULL(cluster))
>>>> + return -ENOMEM;
>>>> +
>>>> + cluster->dev = dev;
>>>> +
>>>> + ret = devm_of_platform_populate(dev);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + /* wire in so each core can be cleaned up at driver remove */
>>>> + platform_set_drvdata(pdev, cluster);
>>>> +
>>>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + ret = zynqmp_r5_cluster_init(cluster);
>>>> + if (ret) {
>>>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/* Match table for OF platform binding */
>>>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>>>> + { .compatible = "xlnx,zynqmp-r5fss", },
>>>> + { /* end of list */ },
>>>> +};
>>>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>>>> +
>>>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>>>> + .probe = zynqmp_r5_remoteproc_probe,
>>>> + .driver = {
>>>> + .name = "zynqmp_r5_remoteproc",
>>>> + .of_match_table = zynqmp_r5_remoteproc_match,
>>>> + },
>>>> +};
>>>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>>>> +
>>>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>>>> +MODULE_AUTHOR("Xilinx Inc.");
>>>> +MODULE_LICENSE("GPL v2");
>>>> --
>>>> 2.25.1
>>>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-01 18:37 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-01 18:37 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On 12/1/21 1:44 AM, Mathieu Poirier wrote:
> On Tue, 30 Nov 2021 at 01:01, Tanmay Shah <tanmay.shah@xilinx.com> wrote:
>>
>> On 11/30/21 12:12 AM, Mathieu Poirier wrote:
>>> On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
>>>> This driver enables r5f dual core Real time Processing Unit subsystem
>>>> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
>>>> (cluster) can be configured in different modes e.g. split mode in which
>>>> two r5f cores work independent of each other and lock-step mode in which
>>>> both r5f cores execute same code clock-for-clock and notify if the
>>>> result is different.
>>>>
>>>> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
>>>> Platform Management Unit that handles the R5 configuration, memory access
>>>> and R5 lifecycle management. The interface to this manager is done in this
>>>> driver via zynqmp_pm_* function calls.
>>>>
>>>> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
>>>> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
>>>> ---
>>>> drivers/remoteproc/Kconfig | 12 +
>>>> drivers/remoteproc/Makefile | 1 +
>>>> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
>>>> 3 files changed, 972 insertions(+)
>>>> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>>> ...and this patch gives me complation warnings:
>>>
>>> CC drivers/remoteproc/xlnx_r5_remoteproc.o
>>> kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c: In function ‘add_tcm_carveout_lockstep_mode’:
>>> kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:412:28: warning: unused variable ‘cluster’ [-Wunused-variable]
>>> 412 | struct zynqmp_r5_cluster *cluster;
>>> | ^~~~~~~
>>> kernel-review/drivers/remoteproc/xlnx_r5_remoteproc.c:411:26: warning: unused variable ‘parent_pdev’ [-Wunused-variable]
>>> 411 | struct platform_device *parent_pdev;
>>> | ^~~~~~~~~~~
>>>
>>> The above leads me to believe this patchset was not compiled before it was sent
>>> out.
>> Please don't assume that this patch-set was not compiled. This driver
>> was compiled and tested on Xilinx QEMU and zynqmp platform for its
>> functionality.
>>
>> This driver went through multiple internal reviews and I had to
>> re-architecture it multiple times.
> And yet two unused variables (which is pretty easy to spot) got by.
>
>> I simply missed to fix above warnings before submitting driver. I have
>> compiled driver with following command:
>>
>> //make ARCH=arm64 W=1 C=1 CROSS_COMPILE="aarch64-linux-gnu-" -j32 -Rr
>> O=$zynqmp_kernel_build/rproc-next
>>
>> I did my best to make sure driver stays warning free, however few
>> warnings were still missed. That is not intentional and by mistake. I
>> fully intend to comply with Linux Kernel community guideline and
>> checklist before submitting patches.
>>
> I am sure you do.
>
>> Apart from above warnings, two more warnings are there in v2 i.e. due to
>> typecast issue between (void __iomem *) and (void *). But, I had not
>> solution before. Also they were used before in different driver so, I
>> chose to use them anyway.
>>
> The sparse warnings related to the "__iomem *" can be ignored. I also
> tried to find a solution to those but realised it was more involved
> than expected.
>
>> I just found that using memremap set of functions will fix them. I will
>> use those functions in next patch set instead of ioremap_wc as explained
>> in this article: https://lwn.net/Articles/653585/
>>
> This is very interesting...
>
>>> Being new to this I can understand that checkpatch.pl was omitted (albeit amply
>>> documented) but obvious compilation warnings can't be excused. As such I
>>> am dropping this set and will not review another version until January.
>>
>> I understand having warnings in driver may lead to frustration and may
>> cause trust issues for the rest of the code and it can't be excused at all.
>>
> Very much so.
>
>> I am aware of guidelines of sending patches upstream
>> (https://www.kernel.org/doc/html/latest/process/submitting-patches.html)
>> and fully intend to comply with that and like I said, if I miss
>> something it is not by intention but human error.
>>
>> I constantly try to improve processes to upstream patches so we don't
>> face above type of issues.
>>
>> I did run checkpatch.pl as per best of my knowledge and fixed lot of
>> style related warnings reported by it. Also I enabled W=1 option in my
>> compilation command and fixed lots of warnings reported by compiler too.
>> As I explained earlier, I wasn't aware of unused-variable warnings, and
>> so I missed to fix them.
>>
> Ok
>
>> With this, I request to review driver from functionality point of view
>> as well along with style errors. So, I can address more number of
>> comments / concerns in less number of patches. If you want these
>> warnings to be fixed, I will send v3 and you can put more comments on
>> v3. However, I highly appreciate if we can continue reviews and not
>> postpone till January. Please let me know your thoughts.
>>
> I will give this patchset another chance later this week.
Thanks for this consideration.
>
>> Thanks,
>>
>> Tanmay
>>
>>
>>> Mathieu
>>
>>>> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
>>>> index f30d00a3aabe..27f66910d8d3 100644
>>>> --- a/drivers/remoteproc/Kconfig
>>>> +++ b/drivers/remoteproc/Kconfig
>>>> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
>>>> It's safe to say N here if you're not interested in utilizing
>>>> a slave processor.
>>>>
>>>> +config XLNX_R5_REMOTEPROC
>>>> + tristate "Xilinx R5 remoteproc support"
>>>> + depends on PM && ARCH_ZYNQMP
>>>> + depends on ZYNQMP_FIRMWARE
>>>> + select RPMSG_VIRTIO
>>>> + select ZYNQMP_IPI_MBOX
>>>> + help
>>>> + Say y or m here to support Xilinx R5 remote processors via the remote
>>>> + processor framework.
>>>> +
>>>> + It's safe to say N if not interested in using RPU r5f cores.
>>>> +
>>>> endif # REMOTEPROC
>>>>
>>>> endmenu
>>>> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
>>>> index bb26c9e4ef9c..334a8bed4c14 100644
>>>> --- a/drivers/remoteproc/Makefile
>>>> +++ b/drivers/remoteproc/Makefile
>>>> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
>>>> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
>>>> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
>>>> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
>>>> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
>>>> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
>>>> new file mode 100644
>>>> index 000000000000..c2167fd3869d
>>>> --- /dev/null
>>>> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
>>>> @@ -0,0 +1,959 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * ZynqMP R5 Remote Processor driver
>>>> + *
>>>> + */
>>>> +
>>>> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
>>>> +#include <linux/firmware/xlnx-zynqmp.h>
>>>> +#include <linux/interrupt.h>
>>>> +#include <linux/kernel.h>
>>>> +#include <linux/list.h>
>>>> +#include <linux/mailbox_client.h>
>>>> +#include <linux/mailbox/zynqmp-ipi-message.h>
>>>> +#include <linux/module.h>
>>>> +#include <linux/of_address.h>
>>>> +#include <linux/of_platform.h>
>>>> +#include <linux/of_reserved_mem.h>
>>>> +#include <linux/platform_device.h>
>>>> +#include <linux/remoteproc.h>
>>>> +#include <linux/skbuff.h>
>>>> +#include <linux/sysfs.h>
>>>> +
>>>> +#include "remoteproc_internal.h"
>>>> +
>>>> +/* settings for RPU cluster mode */
>>>> +enum zynqmp_r5_cluster_mode {
>>>> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
>>>> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
>>>> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct mem_bank_data - Memory Bank description
>>>> + *
>>>> + * @addr: Start address of memory bank
>>>> + * @size: Size of Memory bank
>>>> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
>>>> + * @bank_name: name of the bank for remoteproc framework
>>>> + */
>>>> +struct mem_bank_data {
>>>> + phys_addr_t addr;
>>>> + size_t size;
>>>> + enum pm_node_id pm_domain_id;
>>>> + char *bank_name;
>>>> +};
>>>> +
>>>> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
>>>> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
>>>> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
>>>> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
>>>> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct zynqmp_r5_core - ZynqMP R5 core structure
>>>> + *
>>>> + * @dev: device of RPU instance
>>>> + * @np: device node of RPU instance
>>>> + * @tcm_bank_count: number TCM banks accessible to this RPU
>>>> + * @tcm_banks: array of each TCM bank data
>>>> + * @res_mem_count: number of Reserved Memory regions per core
>>>> + * @res_mem: array of reserved memory regions
>>>> + * @rproc: rproc handle
>>>> + * @pm_domain_id: RPU CPU power domain id
>>>> + */
>>>> +struct zynqmp_r5_core {
>>>> + struct device *dev;
>>>> + struct device_node *np;
>>>> + int tcm_bank_count;
>>>> + struct mem_bank_data *tcm_banks;
>>>> + int res_mem_count;
>>>> + struct reserved_mem *res_mem;
>>>> + struct rproc *rproc;
>>>> + enum pm_node_id pm_domain_id;
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
>>>> + *
>>>> + * @dev: r5f subsystem cluster device node
>>>> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
>>>> + * @core_count: number of r5 cores used for this cluster mode
>>>> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
>>>> + */
>>>> +struct zynqmp_r5_cluster {
>>>> + struct device *dev;
>>>> + enum zynqmp_r5_cluster_mode mode;
>>>> + int core_count;
>>>> + struct zynqmp_r5_core *r5_cores;
>>>> +};
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_set_mode - set RPU operation mode
>>>> + *
>>>> + * set RPU operation mode
>>>> + *
>>>> + * Return: 0 for success, negative value for failure
>>>> + */
>>>> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
>>>> + enum zynqmp_r5_cluster_mode rpu_mode)
>>>> +{
>>>> + enum rpu_tcm_comb tcm_mode;
>>>> + int ret, reg_val;
>>>> +
>>>> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
>>>> +
>>>> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
>>>> + if (ret < 0) {
>>>> + pr_err("failed to set RPU mode\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
>>>> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
>>>> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
>>>> + if (ret < 0)
>>>> + pr_err("failed to configure TCM\n");
>>>> +
>>>> + return ret;
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_rproc_start
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + *
>>>> + * Start R5 Core from designated boot address.
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>>>> +{
>>>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>>>> + enum rpu_boot_mem bootmem;
>>>> + int ret;
>>>> +
>>>> + if (!r5_core) {
>>>> + pr_err("can't get r5 core\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>>>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
>>>> +
>>>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>>>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>>>> +
>>>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>>>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>>>> + if (ret)
>>>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>>>> + return ret;
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_rproc_stop
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + *
>>>> + * Power down R5 Core.
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>>>> +{
>>>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>>>> + int ret;
>>>> +
>>>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>>>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>>>> + if (ret)
>>>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>>>> +
>>>> + return ret;
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_rproc_mem_map
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + * @mem: mem entry to map
>>>> + *
>>>> + * Callback to map va for memory-region's carveout.
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>>>> + struct rproc_mem_entry *mem)
>>>> +{
>>>> + void __iomem *va;
>>>> +
>>>> + va = ioremap_wc(mem->dma, mem->len);
>>>> + if (IS_ERR_OR_NULL(va))
>>>> + return -ENOMEM;
>>>> +
>>>> + mem->va = (void *)va;
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_rproc_mem_unmap
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + * @mem: mem entry to unmap
>>>> + *
>>>> + * Unmap memory-region carveout
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>>>> + struct rproc_mem_entry *mem)
>>>> +{
>>>> + iounmap((void __iomem *)mem->va);
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/*
>>>> + * add_mem_regions
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + *
>>>> + * Construct rproc mem carveouts from carveout provided in
>>>> + * memory-region property
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int add_mem_regions(struct rproc *rproc)
>>>> +{
>>>> + struct device *dev;
>>>> + struct rproc_mem_entry *mem;
>>>> + struct reserved_mem *rmem;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + int i;
>>>> +
>>>> + r5_core = rproc->priv;
>>>> + dev = r5_core->dev;
>>>> +
>>>> + /* Register associated reserved memory regions */
>>>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>>>> + rmem = &r5_core->res_mem[i];
>>>> + mem = rproc_mem_entry_init(dev, NULL,
>>>> + (dma_addr_t)rmem->base,
>>>> + rmem->size, rmem->base,
>>>> + zynqmp_r5_rproc_mem_map,
>>>> + zynqmp_r5_rproc_mem_unmap,
>>>> + rmem->name);
>>>> + if (IS_ERR_OR_NULL(mem))
>>>> + return -ENOMEM;
>>>> +
>>>> + rproc_add_carveout(rproc, mem);
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_rproc_mem_unmap
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + * @mem: mem entry to unmap
>>>> + *
>>>> + * Unmap TCM banks when powering down R5 core.
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>>>> +{
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + int i;
>>>> + enum pm_node_id pm_domain_id;
>>>> +
>>>> + r5_core = rproc->priv;
>>>> + if (!r5_core) {
>>>> + pr_err("r5 core is not available\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + iounmap((void __iomem *)mem->va);
>>>> +
>>>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>>>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>>>> + if (zynqmp_pm_release_node(pm_domain_id))
>>>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/*
>>>> + * tcm_mem_map
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + * @mem: mem entry to initialize the va and da fields of
>>>> + *
>>>> + * Given TCM bank entry, this callback will set device address for R5
>>>> + * running on TCM and also setup virtual address for TCM bank
>>>> + * remoteproc carveout.
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int tcm_mem_map(struct rproc *rproc,
>>>> + struct rproc_mem_entry *mem)
>>>> +{
>>>> + void __iomem *va;
>>>> +
>>>> + va = ioremap_wc(mem->dma, mem->len);
>>>> + if (IS_ERR_OR_NULL(va))
>>>> + return -ENOMEM;
>>>> +
>>>> + /* Update memory entry va */
>>>> + mem->va = (void *)va;
>>>> +
>>>> + /* clear TCMs */
>>>> + memset_io(va, 0, mem->len);
>>>> +
>>>> + /*
>>>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>>>> + * while on the Linux side they are at 0xffexxxxx.
>>>> + *
>>>> + * Zero out the high 12 bits of the address. This will give
>>>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>>>> + */
>>>> + mem->da &= 0x000fffff;
>>>> +
>>>> + /*
>>>> + * TCM Banks 1A and 1B still have to be translated.
>>>> + *
>>>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>>>> + * 0xffeb0000) and convert to the expected relative addresses
>>>> + * (0x0 and 0x20000).
>>>> + */
>>>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>>>> + mem->da -= 0x90000;
>>>> +
>>>> + /* if translated TCM bank address is not valid report error */
>>>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>>>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>>>> + return -EINVAL;
>>>> + }
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>>>> +{
>>>> + int i, num_banks, ret;
>>>> + struct rproc_mem_entry *mem;
>>>> + enum pm_node_id pm_domain_id;
>>>> + u32 bank_addr;
>>>> + size_t bank_size = 0;
>>>> + char *bank_name;
>>>> + struct device *dev;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> +
>>>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>>>> + if (!r5_core)
>>>> + return -EINVAL;
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + /* go through zynqmp banks for r5 node */
>>>> + num_banks = r5_core->tcm_bank_count;
>>>> + if (num_banks <= 0) {
>>>> + dev_err(dev, "need to specify TCM banks\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + for (i = 0; i < num_banks; i++) {
>>>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>>>> + bank_name = r5_core->tcm_banks[i].bank_name;
>>>> + bank_size = r5_core->tcm_banks[i].size;
>>>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>>>> +
>>>> + ret = zynqmp_pm_request_node(pm_domain_id,
>>>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>>>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>>>> + if (ret < 0) {
>>>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>>>> + bank_name, bank_addr, bank_size);
>>>> +
>>>> + /* add carveout */
>>>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>>>> + bank_size, bank_addr,
>>>> + tcm_mem_map, tcm_mem_unmap,
>>>> + bank_name);
>>>> + if (IS_ERR_OR_NULL(mem)) {
>>>> + /* Turn off all TCM banks turned on before */
>>>> + do {
>>>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>>>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>>>> + if (ret)
>>>> + dev_warn(dev,
>>>> + "fail to release node: %x, %x\n",
>>>> + (u32)pm_domain_id, ret);
>>>> + } while (i--);
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + rproc_add_carveout(rproc, mem);
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>>>> +{
>>>> + int i, num_banks, ret;
>>>> + struct rproc_mem_entry *mem;
>>>> + enum pm_node_id pm_domain_id;
>>>> + u32 bank_addr;
>>>> + size_t bank_size = 0;
>>>> + char *bank_name;
>>>> + struct device *dev;
>>>> + struct platform_device *parent_pdev;
>>>> + struct zynqmp_r5_cluster *cluster;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> +
>>>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>>>> + if (!r5_core)
>>>> + return -EINVAL;
>>>> +
>>>> + dev = r5_core->dev;
>>>> + if (!dev) {
>>>> + pr_err("r5 core device unavailable\n");
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + /* go through zynqmp banks for r5 node */
>>>> + num_banks = r5_core->tcm_bank_count;
>>>> + if (num_banks <= 0) {
>>>> + dev_err(dev, "need to specify TCM banks\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>>>> + bank_name = r5_core->tcm_banks[0].bank_name;
>>>> + for (i = 0; i < num_banks; i++) {
>>>> + bank_size += r5_core->tcm_banks[i].size;
>>>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>>>> +
>>>> + ret = zynqmp_pm_request_node(pm_domain_id,
>>>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>>>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>>>> + if (ret < 0) {
>>>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>>>> + return ret;
>>>> + }
>>>> + }
>>>> +
>>>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>>>> + bank_name, bank_addr, bank_size);
>>>> +
>>>> + /* add carveout */
>>>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>>>> + bank_size, bank_addr,
>>>> + tcm_mem_map, tcm_mem_unmap,
>>>> + bank_name);
>>>> + if (IS_ERR_OR_NULL(mem)) {
>>>> + for (i = 0; i < num_banks; i++) {
>>>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>>>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>>>> + if (ret)
>>>> + dev_warn(dev,
>>>> + "fail to release node: %x ret: %x\n",
>>>> + (u32)pm_domain_id, ret);
>>>> + }
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + rproc_add_carveout(rproc, mem);
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/*
>>>> + * add_tcm_banks()
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + *
>>>> + * Given R5 node in remoteproc instance
>>>> + * allocate remoteproc carveout for TCM memory
>>>> + * needed for firmware to be loaded
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int add_tcm_banks(struct rproc *rproc)
>>>> +{
>>>> + struct device *dev;
>>>> + struct platform_device *parent_pdev;
>>>> + struct zynqmp_r5_cluster *cluster;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> +
>>>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>>>> + if (!r5_core)
>>>> + return -EINVAL;
>>>> +
>>>> + dev = r5_core->dev;
>>>> + if (!dev) {
>>>> + pr_err("r5 core device unavailable\n");
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + parent_pdev = to_platform_device(dev->parent);
>>>> + if (!parent_pdev) {
>>>> + dev_err(dev, "parent platform dev unavailable\n");
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + cluster = platform_get_drvdata(parent_pdev);
>>>> + if (!cluster) {
>>>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + if (cluster->mode == SPLIT_MODE)
>>>> + return add_tcm_carveout_split_mode(rproc);
>>>> + else if (cluster->mode == LOCKSTEP_MODE)
>>>> + return add_tcm_carveout_lockstep_mode(rproc);
>>>> +
>>>> + dev_err(cluster->dev, "invalid cluster mode\n");
>>>> + return -EINVAL;
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_parse_fw()
>>>> + * @rproc: single R5 core's corresponding rproc instance
>>>> + * @fw: ptr to firmware to be loaded onto r5 core
>>>> + *
>>>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>>>> + *
>>>> + * return 0 on success, otherwise non-zero value on failure
>>>> + */
>>>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>>>> +{
>>>> + int ret;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + struct device *dev;
>>>> +
>>>> + r5_core = rproc->priv;
>>>> + if (!r5_core) {
>>>> + dev_err(&rproc->dev, "r5 core not available\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + ret = add_tcm_banks(rproc);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + ret = add_mem_regions(rproc);
>>>> + if (ret)
>>>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>>>> +
>>>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>>>> + if (ret == -EINVAL) {
>>>> + /*
>>>> + * resource table only required for IPC.
>>>> + * if not present, this is not necessarily an error;
>>>> + * for example, loading r5 hello world application
>>>> + * so simply inform user and keep going.
>>>> + */
>>>> + dev_info(&rproc->dev, "no resource table found.\n");
>>>> + ret = 0;
>>>> + }
>>>> + return ret;
>>>> +}
>>>> +
>>>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>>>> + .start = zynqmp_r5_rproc_start,
>>>> + .stop = zynqmp_r5_rproc_stop,
>>>> + .load = rproc_elf_load_segments,
>>>> + .parse_fw = zynqmp_r5_parse_fw,
>>>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>>>> + .sanity_check = rproc_elf_sanity_check,
>>>> + .get_boot_addr = rproc_elf_get_boot_addr,
>>>> +};
>>>> +
>>>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + int i, j, k;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> +
>>>> + dev_dbg(cluster->dev, "Printing dt node info\n");
>>>> +
>>>> + pr_debug("cluster mode = %d\n", cluster->mode);
>>>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>>>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>>>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>>>> +
>>>> + for (i = 0; i < cluster->core_count; i++) {
>>>> + r5_core = &cluster->r5_cores[i];
>>>> + if (!r5_core) {
>>>> + pr_err("can't get r5_core\n");
>>>> + continue;
>>>> + }
>>>> +
>>>> + pr_debug("r5 core %d nodes\n", i);
>>>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>>>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>>>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>>>> + k, r5_core->tcm_banks[k].addr,
>>>> + r5_core->tcm_banks[k].size,
>>>> + r5_core->tcm_banks[k].pm_domain_id,
>>>> + r5_core->tcm_banks[k].bank_name);
>>>> + }
>>>> +
>>>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>>>> +
>>>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>>>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>>>> + j, r5_core->res_mem[j].base,
>>>> + r5_core->res_mem[j].size,
>>>> + r5_core->res_mem[j].name);
>>>> + }
>>>> + }
>>>> +}
>>>> +
>>>> +/**
>>>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>>>> + * this is called for each individual R5 core to
>>>> + * set up mailbox, Xilinx platform manager unique ID,
>>>> + * add to rproc core
>>>> + *
>>>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>>>> + *
>>>> + * Return: 0 for success, negative value for failure.
>>>> + */
>>>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>>>> +{
>>>> + int ret;
>>>> + struct rproc *r5_rproc;
>>>> + struct device *dev;
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + /* Set up DMA mask */
>>>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + /* Allocate remoteproc instance */
>>>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>>>> + NULL, sizeof(struct zynqmp_r5_core));
>>>> + if (IS_ERR_OR_NULL(r5_rproc))
>>>> + return -ENOMEM;
>>>> +
>>>> + r5_rproc->auto_boot = false;
>>>> + r5_rproc->priv = r5_core;
>>>> +
>>>> + /* Add R5 remoteproc */
>>>> + ret = devm_rproc_add(dev, r5_rproc);
>>>> + if (ret) {
>>>> + pr_err("failed to add r5 remoteproc\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + int tcm_bank_count, tcm_node;
>>>> + int i = 0, j;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>>>> + struct device *dev = cluster->dev;
>>>> +
>>>> + /* ToDo: Use predefined TCM address space values from driver until
>>>> + * system-dt spec is not final fot TCM
>>>> + */
>>>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>>>> +
>>>> + /* count per core tcm banks */
>>>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>>>> +
>>>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>>>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>>>> + * 128k. Assign TCM banks to each core accordingly
>>>> + */
>>>> + tcm_node = 0;
>>>> + for (j = 0; j < cluster->core_count; j++) {
>>>> + r5_core = &cluster->r5_cores[j];
>>>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>>>> + tcm_bank_count, GFP_KERNEL);
>>>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>>>> + return -ENOMEM;
>>>> +
>>>> + for (i = 0; i < tcm_bank_count; i++) {
>>>> + /* Use pre-defined TCM reg values.
>>>> + * Eventually this should be replaced by values
>>>> + * parsed from dts.
>>>> + */
>>>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>>>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>>>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>>>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>>>> + tcm_node++;
>>>> + }
>>>> +
>>>> + r5_core->tcm_bank_count = tcm_bank_count;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>>>> +{
>>>> + int res_mem_count, i;
>>>> + struct device *dev;
>>>> + struct device_node *np, *rmem_np;
>>>> + struct reserved_mem *rmem;
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + np = r5_core->np;
>>>> + if (IS_ERR_OR_NULL(np)) {
>>>> + pr_err("invalid device node of r5 core\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>>>> + sizeof(phandle));
>>>> + if (res_mem_count <= 0) {
>>>> + dev_warn(dev, "failed to get memory-region property %d\n",
>>>> + res_mem_count);
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + r5_core->res_mem = devm_kzalloc(dev,
>>>> + res_mem_count * sizeof(struct reserved_mem),
>>>> + GFP_KERNEL);
>>>> + if (!r5_core->res_mem) {
>>>> + dev_err(dev, "failed to allocate mem region memory\n");
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + for (i = 0; i < res_mem_count; i++) {
>>>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>>>> + if (!rmem_np)
>>>> + return -EINVAL;
>>>> +
>>>> + rmem = of_reserved_mem_lookup(rmem_np);
>>>> + if (!rmem) {
>>>> + of_node_put(rmem_np);
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + memcpy(&r5_core->res_mem[i], rmem,
>>>> + sizeof(struct reserved_mem));
>>>> + of_node_put(rmem_np);
>>>> + }
>>>> +
>>>> + r5_core->res_mem_count = res_mem_count;
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + int ret, i;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + struct device *dev = cluster->dev;
>>>> +
>>>> + ret = zynqmp_r5_get_tcm_node(cluster);
>>>> + if (ret < 0) {
>>>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + for (i = 0; i < cluster->core_count; i++) {
>>>> + r5_core = &cluster->r5_cores[i];
>>>> + if (!r5_core) {
>>>> + pr_err("invalid r5 core\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>>>> + if (ret)
>>>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>>>> +
>>>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>>>> + 1, &r5_core->pm_domain_id);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to get power-domains property\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to init r5 core %d\n", i);
>>>> + return ret;
>>>> + }
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + struct device *dev = cluster->dev;
>>>> + struct device_node *dev_node = dev_of_node(dev);
>>>> + struct device_node *child;
>>>> + struct platform_device *child_pdev;
>>>> + int core_count = 0, ret, i;
>>>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>>>> + struct zynqmp_r5_core *r5_cores;
>>>> +
>>>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>>>> +
>>>> + /* on success returns 0, if not defined then returns -EINVAL,
>>>> + * In that case, default is LOCKSTEP mode
>>>> + */
>>>> + if (ret != -EINVAL && ret != 0) {
>>>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + if (cluster_mode == SINGLE_CPU_MODE) {
>>>> + dev_err(dev, "driver does not support single cpu mode\n");
>>>> + return -EINVAL;
>>>> + } else if ((cluster_mode != SPLIT_MODE &&
>>>> + cluster_mode != LOCKSTEP_MODE)) {
>>>> + dev_err(dev, "Invalid cluster mode\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + core_count = of_get_available_child_count(dev_node);
>>>> + if (core_count <= 0) {
>>>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>>>> + return -EINVAL;
>>>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>>>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>>>> + return -EINVAL;
>>>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>>>> + dev_warn(dev, "Only r5 core0 will be used\n");
>>>> + core_count = 1;
>>>> + }
>>>> +
>>>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>>>> + core_count, GFP_KERNEL);
>>>> + if (IS_ERR_OR_NULL(r5_cores)) {
>>>> + dev_err(dev, "can't allocate memory for cores\n");
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + i = 0;
>>>> + for_each_available_child_of_node(dev_node, child) {
>>>> + child_pdev = of_find_device_by_node(child);
>>>> + if (!child_pdev)
>>>> + return -ENODEV;
>>>> +
>>>> + r5_cores[i].dev = &child_pdev->dev;
>>>> + if (!r5_cores[i].dev) {
>>>> + pr_err("can't get device for r5 core %d\n", i);
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>>>> + if (!r5_cores[i].np) {
>>>> + pr_err("can't get device node for r5 core %d\n", i);
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + i++;
>>>> + if (i == core_count)
>>>> + break;
>>>> + }
>>>> +
>>>> + cluster->mode = cluster_mode;
>>>> + cluster->core_count = core_count;
>>>> + cluster->r5_cores = r5_cores;
>>>> +
>>>> + ret = zynqmp_r5_core_init(cluster);
>>>> + if (ret < 0) {
>>>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + zynqmp_r5_print_dt_node_info(cluster);
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static void zynqmp_r5_cluster_exit(void *data)
>>>> +{
>>>> + struct platform_device *pdev = (struct platform_device *)data;
>>>> +
>>>> + platform_set_drvdata(pdev, NULL);
>>>> +
>>>> + pr_info("Exit r5f subsystem driver\n");
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_remoteproc_probe()
>>>> + *
>>>> + * @pdev: domain platform device for R5 cluster
>>>> + *
>>>> + * called when driver is probed, for each R5 core specified in DT,
>>>> + * setup as needed to do remoteproc-related operations
>>>> + *
>>>> + * Return: 0 for success, negative value for failure.
>>>> + */
>>>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>>>> +{
>>>> + int ret;
>>>> + struct zynqmp_r5_cluster *cluster;
>>>> + struct device *dev = &pdev->dev;
>>>> +
>>>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>>>> + if (IS_ERR_OR_NULL(cluster))
>>>> + return -ENOMEM;
>>>> +
>>>> + cluster->dev = dev;
>>>> +
>>>> + ret = devm_of_platform_populate(dev);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + /* wire in so each core can be cleaned up at driver remove */
>>>> + platform_set_drvdata(pdev, cluster);
>>>> +
>>>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + ret = zynqmp_r5_cluster_init(cluster);
>>>> + if (ret) {
>>>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/* Match table for OF platform binding */
>>>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>>>> + { .compatible = "xlnx,zynqmp-r5fss", },
>>>> + { /* end of list */ },
>>>> +};
>>>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>>>> +
>>>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>>>> + .probe = zynqmp_r5_remoteproc_probe,
>>>> + .driver = {
>>>> + .name = "zynqmp_r5_remoteproc",
>>>> + .of_match_table = zynqmp_r5_remoteproc_match,
>>>> + },
>>>> +};
>>>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>>>> +
>>>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>>>> +MODULE_AUTHOR("Xilinx Inc.");
>>>> +MODULE_LICENSE("GPL v2");
>>>> --
>>>> 2.25.1
>>>>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-11-23 6:20 ` Tanmay Shah
@ 2021-12-03 18:55 ` Mathieu Poirier
-1 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-12-03 18:55 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
Hi Tanmay,
On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
> This driver enables r5f dual core Real time Processing Unit subsystem
> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
> (cluster) can be configured in different modes e.g. split mode in which
> two r5f cores work independent of each other and lock-step mode in which
> both r5f cores execute same code clock-for-clock and notify if the
> result is different.
>
> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
>
> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
> ---
> drivers/remoteproc/Kconfig | 12 +
> drivers/remoteproc/Makefile | 1 +
> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
> 3 files changed, 972 insertions(+)
> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>
> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> index f30d00a3aabe..27f66910d8d3 100644
> --- a/drivers/remoteproc/Kconfig
> +++ b/drivers/remoteproc/Kconfig
> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
> It's safe to say N here if you're not interested in utilizing
> a slave processor.
>
> +config XLNX_R5_REMOTEPROC
> + tristate "Xilinx R5 remoteproc support"
> + depends on PM && ARCH_ZYNQMP
> + depends on ZYNQMP_FIRMWARE
> + select RPMSG_VIRTIO
> + select ZYNQMP_IPI_MBOX
> + help
> + Say y or m here to support Xilinx R5 remote processors via the remote
> + processor framework.
> +
> + It's safe to say N if not interested in using RPU r5f cores.
> +
> endif # REMOTEPROC
>
> endmenu
> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> index bb26c9e4ef9c..334a8bed4c14 100644
> --- a/drivers/remoteproc/Makefile
> +++ b/drivers/remoteproc/Makefile
> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> new file mode 100644
> index 000000000000..c2167fd3869d
> --- /dev/null
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -0,0 +1,959 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ZynqMP R5 Remote Processor driver
> + *
> + */
> +
> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
> +#include <linux/firmware/xlnx-zynqmp.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/mailbox_client.h>
> +#include <linux/mailbox/zynqmp-ipi-message.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/remoteproc.h>
> +#include <linux/skbuff.h>
> +#include <linux/sysfs.h>
> +
> +#include "remoteproc_internal.h"
> +
> +/* settings for RPU cluster mode */
> +enum zynqmp_r5_cluster_mode {
> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
Please use C style comments, i.e /*...*/
> +};
> +
> +/**
> + * struct mem_bank_data - Memory Bank description
> + *
> + * @addr: Start address of memory bank
> + * @size: Size of Memory bank
> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
> + * @bank_name: name of the bank for remoteproc framework
> + */
> +struct mem_bank_data {
> + phys_addr_t addr;
> + size_t size;
> + enum pm_node_id pm_domain_id;
> + char *bank_name;
> +};
> +
> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
> +};
Bizarre - more comments on this below...
> +
> +/**
> + * struct zynqmp_r5_core - ZynqMP R5 core structure
> + *
> + * @dev: device of RPU instance
> + * @np: device node of RPU instance
> + * @tcm_bank_count: number TCM banks accessible to this RPU
> + * @tcm_banks: array of each TCM bank data
> + * @res_mem_count: number of Reserved Memory regions per core
> + * @res_mem: array of reserved memory regions
> + * @rproc: rproc handle
> + * @pm_domain_id: RPU CPU power domain id
> + */
> +struct zynqmp_r5_core {
> + struct device *dev;
> + struct device_node *np;
> + int tcm_bank_count;
> + struct mem_bank_data *tcm_banks;
> + int res_mem_count;
> + struct reserved_mem *res_mem;
> + struct rproc *rproc;
> + enum pm_node_id pm_domain_id;
> +};
> +
> +/**
> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
> + *
> + * @dev: r5f subsystem cluster device node
> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
> + * @core_count: number of r5 cores used for this cluster mode
> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
> + */
> +struct zynqmp_r5_cluster {
> + struct device *dev;
> + enum zynqmp_r5_cluster_mode mode;
> + int core_count;
> + struct zynqmp_r5_core *r5_cores;
> +};
> +
> +/*
> + * zynqmp_r5_set_mode - set RPU operation mode
> + *
> + * set RPU operation mode
> + *
> + * Return: 0 for success, negative value for failure
> + */
> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
> + enum zynqmp_r5_cluster_mode rpu_mode)
> +{
> + enum rpu_tcm_comb tcm_mode;
> + int ret, reg_val;
> +
> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
> +
> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
> + if (ret < 0) {
> + pr_err("failed to set RPU mode\n");
> + return ret;
> + }
> +
> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
> + if (ret < 0)
> + pr_err("failed to configure TCM\n");
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_start
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Start R5 Core from designated boot address.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + enum rpu_boot_mem bootmem;
> + int ret;
> +
> + if (!r5_core) {
> + pr_err("can't get r5 core\n");
> + return -EINVAL;
> + }
> +
> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> +
> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> +
> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> + if (ret)
> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_stop
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Power down R5 Core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + int ret;
> +
> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret)
> + pr_err("failed to stop remoteproc RPU %d\n", ret);
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + mem->va = (void *)va;
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap memory-region carveout
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + iounmap((void __iomem *)mem->va);
> + return 0;
> +}
> +
> +/*
> + * add_mem_regions
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Construct rproc mem carveouts from carveout provided in
> + * memory-region property
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_mem_regions(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct rproc_mem_entry *mem;
> + struct reserved_mem *rmem;
> + struct zynqmp_r5_core *r5_core;
> + int i;
> +
> + r5_core = rproc->priv;
> + dev = r5_core->dev;
> +
> + /* Register associated reserved memory regions */
> + for (i = 0; i < r5_core->res_mem_count; i++) {
> + rmem = &r5_core->res_mem[i];
> + mem = rproc_mem_entry_init(dev, NULL,
> + (dma_addr_t)rmem->base,
> + rmem->size, rmem->base,
> + zynqmp_r5_rproc_mem_map,
> + zynqmp_r5_rproc_mem_unmap,
> + rmem->name);
> + if (IS_ERR_OR_NULL(mem))
> + return -ENOMEM;
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap TCM banks when powering down R5 core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> + struct zynqmp_r5_core *r5_core;
> + int i;
> + enum pm_node_id pm_domain_id;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + pr_err("r5 core is not available\n");
> + return -EINVAL;
> + }
> +
> + iounmap((void __iomem *)mem->va);
> +
> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + if (zynqmp_pm_release_node(pm_domain_id))
> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * tcm_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to initialize the va and da fields of
> + *
> + * Given TCM bank entry, this callback will set device address for R5
> + * running on TCM and also setup virtual address for TCM bank
> + * remoteproc carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + /* Update memory entry va */
> + mem->va = (void *)va;
> +
> + /* clear TCMs */
> + memset_io(va, 0, mem->len);
> +
> + /*
> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> + * while on the Linux side they are at 0xffexxxxx.
> + *
> + * Zero out the high 12 bits of the address. This will give
> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> + */
> + mem->da &= 0x000fffff;
> +
> + /*
> + * TCM Banks 1A and 1B still have to be translated.
> + *
> + * Below handle these two banks' absolute addresses (0xffe90000 and
> + * 0xffeb0000) and convert to the expected relative addresses
> + * (0x0 and 0x20000).
> + */
> + if (mem->da == 0x90000 || mem->da == 0xB0000)
> + mem->da -= 0x90000;
> +
> + /* if translated TCM bank address is not valid report error */
> + if (mem->da != 0x0 && mem->da != 0x20000) {
> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
> + return -EINVAL;
> + }
> + return 0;
> +}
> +
> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
> + char *bank_name;
> + struct device *dev;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + for (i = 0; i < num_banks; i++) {
> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
> + bank_name = r5_core->tcm_banks[i].bank_name;
> + bank_size = r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> +
> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + /* Turn off all TCM banks turned on before */
> + do {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x, %x\n",
> + (u32)pm_domain_id, ret);
> + } while (i--);
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
> + char *bank_name;
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
> + bank_name = r5_core->tcm_banks[0].bank_name;
> + for (i = 0; i < num_banks; i++) {
> + bank_size += r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> + }
> +
> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + for (i = 0; i < num_banks; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x ret: %x\n",
> + (u32)pm_domain_id, ret);
> + }
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> +
> + return 0;
> +}
> +
> +/*
> + * add_tcm_banks()
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Given R5 node in remoteproc instance
> + * allocate remoteproc carveout for TCM memory
> + * needed for firmware to be loaded
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_tcm_banks(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + parent_pdev = to_platform_device(dev->parent);
> + if (!parent_pdev) {
> + dev_err(dev, "parent platform dev unavailable\n");
> + return -ENODEV;
> + }
> +
> + cluster = platform_get_drvdata(parent_pdev);
> + if (!cluster) {
> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
> + return -EINVAL;
> + }
> +
> + if (cluster->mode == SPLIT_MODE)
> + return add_tcm_carveout_split_mode(rproc);
> + else if (cluster->mode == LOCKSTEP_MODE)
> + return add_tcm_carveout_lockstep_mode(rproc);
> +
> + dev_err(cluster->dev, "invalid cluster mode\n");
> + return -EINVAL;
> +}
> +
> +/*
> + * zynqmp_r5_parse_fw()
> + * @rproc: single R5 core's corresponding rproc instance
> + * @fw: ptr to firmware to be loaded onto r5 core
> + *
> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> + int ret;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + dev_err(&rproc->dev, "r5 core not available\n");
> + return -EINVAL;
> + }
> +
> + dev = r5_core->dev;
> +
> + ret = add_tcm_banks(rproc);
> + if (ret) {
> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
> + return ret;
> + }
> +
> + ret = add_mem_regions(rproc);
> + if (ret)
> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
> +
> + ret = rproc_elf_load_rsc_table(rproc, fw);
> + if (ret == -EINVAL) {
> + /*
> + * resource table only required for IPC.
> + * if not present, this is not necessarily an error;
> + * for example, loading r5 hello world application
> + * so simply inform user and keep going.
> + */
> + dev_info(&rproc->dev, "no resource table found.\n");
> + ret = 0;
> + }
> + return ret;
> +}
> +
> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> + .start = zynqmp_r5_rproc_start,
> + .stop = zynqmp_r5_rproc_stop,
> + .load = rproc_elf_load_segments,
> + .parse_fw = zynqmp_r5_parse_fw,
> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> + .sanity_check = rproc_elf_sanity_check,
> + .get_boot_addr = rproc_elf_get_boot_addr,
> +};
> +
> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
> +{
> + int i, j, k;
> + struct zynqmp_r5_core *r5_core;
> +
> + dev_dbg(cluster->dev, "Printing dt node info\n");
Why mixing dev_dbg() and pr_debug()? Please pick one and stick with it.
Even for debug level output this is very chatty - is all that information really
needed when it is already available in /proc/device-tree?
> +
> + pr_debug("cluster mode = %d\n", cluster->mode);
> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
> + pr_debug("r5f num cores = %d\n", cluster->core_count);
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("can't get r5_core\n");
> + continue;
> + }
> +
> + pr_debug("r5 core %d nodes\n", i);
> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
> + k, r5_core->tcm_banks[k].addr,
> + r5_core->tcm_banks[k].size,
> + r5_core->tcm_banks[k].pm_domain_id,
> + r5_core->tcm_banks[k].bank_name);
> + }
> +
> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
> +
> + for (j = 0; j < r5_core->res_mem_count; j++) {
> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
> + j, r5_core->res_mem[j].base,
> + r5_core->res_mem[j].size,
> + r5_core->res_mem[j].name);
> + }
> + }
> +}
> +
> +/**
> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
> + * this is called for each individual R5 core to
> + * set up mailbox, Xilinx platform manager unique ID,
> + * add to rproc core
> + *
> + * @r5_core: zynqmp_r5_core r5 core object to initialize
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
> +{
> + int ret;
> + struct rproc *r5_rproc;
> + struct device *dev;
> +
> + dev = r5_core->dev;
> +
> + /* Set up DMA mask */
> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> + if (ret)
> + return ret;
> +
> + /* Allocate remoteproc instance */
> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> + NULL, sizeof(struct zynqmp_r5_core));
> + if (IS_ERR_OR_NULL(r5_rproc))
> + return -ENOMEM;
> +
> + r5_rproc->auto_boot = false;
> + r5_rproc->priv = r5_core;
> +
> + /* Add R5 remoteproc */
> + ret = devm_rproc_add(dev, r5_rproc);
> + if (ret) {
> + pr_err("failed to add r5 remoteproc\n");
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> +{
> + int tcm_bank_count, tcm_node;
> + int i = 0, j;
> + struct zynqmp_r5_core *r5_core;
> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> + struct device *dev = cluster->dev;
> +
> + /* ToDo: Use predefined TCM address space values from driver until
> + * system-dt spec is not final fot TCM
> + */
Multilined comment should be as follow:
/*
* ToDo: Use predefined TCM address space values from driver until
* system-dt spec is not final fot TCM
*/
s/"final fot TCM"/"final for TCM"
Any reason this can't be done with "reg" properties like TI did for K3? It
would be nice to have TCMs included in the yaml file example.
> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> +
> + /* count per core tcm banks */
> + tcm_bank_count = tcm_bank_count / cluster->core_count;
> +
> + /* r5 core 0 will use all of TCM banks in lockstep mode.
> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> + * 128k. Assign TCM banks to each core accordingly
> + */
> + tcm_node = 0;
> + for (j = 0; j < cluster->core_count; j++) {
> + r5_core = &cluster->r5_cores[j];
> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> + tcm_bank_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> + return -ENOMEM;
> +
> + for (i = 0; i < tcm_bank_count; i++) {
> + /* Use pre-defined TCM reg values.
> + * Eventually this should be replaced by values
> + * parsed from dts.
> + */
> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> + tcm_node++;
> + }
> +
> + r5_core->tcm_bank_count = tcm_bank_count;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> +{
> + int res_mem_count, i;
> + struct device *dev;
> + struct device_node *np, *rmem_np;
> + struct reserved_mem *rmem;
> +
> + dev = r5_core->dev;
> +
> + np = r5_core->np;
> + if (IS_ERR_OR_NULL(np)) {
> + pr_err("invalid device node of r5 core\n");
> + return -EINVAL;
> + }
> +
> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> + sizeof(phandle));
> + if (res_mem_count <= 0) {
> + dev_warn(dev, "failed to get memory-region property %d\n",
> + res_mem_count);
> + return -EINVAL;
> + }
> +
> + r5_core->res_mem = devm_kzalloc(dev,
> + res_mem_count * sizeof(struct reserved_mem),
> + GFP_KERNEL);
> + if (!r5_core->res_mem) {
> + dev_err(dev, "failed to allocate mem region memory\n");
> + return -ENOMEM;
> + }
> +
> + for (i = 0; i < res_mem_count; i++) {
> + rmem_np = of_parse_phandle(np, "memory-region", i);
> + if (!rmem_np)
> + return -EINVAL;
> +
> + rmem = of_reserved_mem_lookup(rmem_np);
> + if (!rmem) {
> + of_node_put(rmem_np);
> + return -EINVAL;
> + }
> +
> + memcpy(&r5_core->res_mem[i], rmem,
> + sizeof(struct reserved_mem));
> + of_node_put(rmem_np);
> + }
> +
> + r5_core->res_mem_count = res_mem_count;
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> +{
> + int ret, i;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev = cluster->dev;
> +
> + ret = zynqmp_r5_get_tcm_node(cluster);
> + if (ret < 0) {
> + dev_err(dev, "can't get tcm node, err %d\n", ret);
> + return ret;
> + }
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("invalid r5 core\n");
> + return -EINVAL;
> + }
> +
> + ret = zynqmp_r5_get_mem_region_node(r5_core);
> + if (ret)
> + dev_warn(dev, "memory-region prop failed %d\n", ret);
> +
> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> + 1, &r5_core->pm_domain_id);
> + if (ret) {
> + dev_err(dev, "failed to get power-domains property\n");
> + return ret;
> + }
> +
> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_add_rproc_core(r5_core);
> + if (ret) {
> + dev_err(dev, "failed to init r5 core %d\n", i);
> + return ret;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> +{
> + struct device *dev = cluster->dev;
> + struct device_node *dev_node = dev_of_node(dev);
> + struct device_node *child;
> + struct platform_device *child_pdev;
> + int core_count = 0, ret, i;
> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> + struct zynqmp_r5_core *r5_cores;
> +
> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> +
> + /* on success returns 0, if not defined then returns -EINVAL,
> + * In that case, default is LOCKSTEP mode
> + */
> + if (ret != -EINVAL && ret != 0) {
> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> + return -EINVAL;
> + }
> +
> + if (cluster_mode == SINGLE_CPU_MODE) {
> + dev_err(dev, "driver does not support single cpu mode\n");
> + return -EINVAL;
> + } else if ((cluster_mode != SPLIT_MODE &&
> + cluster_mode != LOCKSTEP_MODE)) {
> + dev_err(dev, "Invalid cluster mode\n");
> + return -EINVAL;
> + }
> +
> + core_count = of_get_available_child_count(dev_node);
> + if (core_count <= 0) {
> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> + return -EINVAL;
> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> + return -EINVAL;
> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> + dev_warn(dev, "Only r5 core0 will be used\n");
> + core_count = 1;
> + }
> +
> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> + core_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_cores)) {
> + dev_err(dev, "can't allocate memory for cores\n");
> + return -ENOMEM;
> + }
> +
> + i = 0;
> + for_each_available_child_of_node(dev_node, child) {
> + child_pdev = of_find_device_by_node(child);
> + if (!child_pdev)
> + return -ENODEV;
> +
> + r5_cores[i].dev = &child_pdev->dev;
> + if (!r5_cores[i].dev) {
> + pr_err("can't get device for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> + if (!r5_cores[i].np) {
> + pr_err("can't get device node for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + i++;
> + if (i == core_count)
> + break;
> + }
> +
> + cluster->mode = cluster_mode;
> + cluster->core_count = core_count;
> + cluster->r5_cores = r5_cores;
> +
> + ret = zynqmp_r5_core_init(cluster);
> + if (ret < 0) {
> + dev_err(dev, "failed to init r5 core err %d\n", ret);
> + return ret;
> + }
> +
> + zynqmp_r5_print_dt_node_info(cluster);
> +
> + return 0;
> +}
> +
> +static void zynqmp_r5_cluster_exit(void *data)
> +{
> + struct platform_device *pdev = (struct platform_device *)data;
> +
> + platform_set_drvdata(pdev, NULL);
> +
> + pr_info("Exit r5f subsystem driver\n");
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_probe()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * called when driver is probed, for each R5 core specified in DT,
> + * setup as needed to do remoteproc-related operations
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> +{
> + int ret;
> + struct zynqmp_r5_cluster *cluster;
> + struct device *dev = &pdev->dev;
> +
> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> + if (IS_ERR_OR_NULL(cluster))
> + return -ENOMEM;
Function devm_kzalloc() does not return an code on error, just NULL. Please fix
throughout the driver.
> +
> + cluster->dev = dev;
> +
> + ret = devm_of_platform_populate(dev);
> + if (ret) {
> + dev_err(dev, "failed to populate platform dev %d\n", ret);
> + return ret;
> + }
> +
> + /* wire in so each core can be cleaned up at driver remove */
> + platform_set_drvdata(pdev, cluster);
> +
> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_cluster_init(cluster);
> + if (ret) {
> + dev_err(dev, "Invalid r5f subsystem device tree\n");
> + return ret;
> + }
> +
> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
Please remove this.
I am out of time for today and will continue on Monday.
Thanks,
Mathieu
> + return 0;
> +}
> +
> +/* Match table for OF platform binding */
> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> + { .compatible = "xlnx,zynqmp-r5fss", },
> + { /* end of list */ },
> +};
> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> +
> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> + .probe = zynqmp_r5_remoteproc_probe,
> + .driver = {
> + .name = "zynqmp_r5_remoteproc",
> + .of_match_table = zynqmp_r5_remoteproc_match,
> + },
> +};
> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> +
> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> +MODULE_AUTHOR("Xilinx Inc.");
> +MODULE_LICENSE("GPL v2");
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-03 18:55 ` Mathieu Poirier
0 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-12-03 18:55 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
Hi Tanmay,
On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
> This driver enables r5f dual core Real time Processing Unit subsystem
> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
> (cluster) can be configured in different modes e.g. split mode in which
> two r5f cores work independent of each other and lock-step mode in which
> both r5f cores execute same code clock-for-clock and notify if the
> result is different.
>
> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
>
> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
> ---
> drivers/remoteproc/Kconfig | 12 +
> drivers/remoteproc/Makefile | 1 +
> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
> 3 files changed, 972 insertions(+)
> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>
> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> index f30d00a3aabe..27f66910d8d3 100644
> --- a/drivers/remoteproc/Kconfig
> +++ b/drivers/remoteproc/Kconfig
> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
> It's safe to say N here if you're not interested in utilizing
> a slave processor.
>
> +config XLNX_R5_REMOTEPROC
> + tristate "Xilinx R5 remoteproc support"
> + depends on PM && ARCH_ZYNQMP
> + depends on ZYNQMP_FIRMWARE
> + select RPMSG_VIRTIO
> + select ZYNQMP_IPI_MBOX
> + help
> + Say y or m here to support Xilinx R5 remote processors via the remote
> + processor framework.
> +
> + It's safe to say N if not interested in using RPU r5f cores.
> +
> endif # REMOTEPROC
>
> endmenu
> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> index bb26c9e4ef9c..334a8bed4c14 100644
> --- a/drivers/remoteproc/Makefile
> +++ b/drivers/remoteproc/Makefile
> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> new file mode 100644
> index 000000000000..c2167fd3869d
> --- /dev/null
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -0,0 +1,959 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ZynqMP R5 Remote Processor driver
> + *
> + */
> +
> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
> +#include <linux/firmware/xlnx-zynqmp.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/mailbox_client.h>
> +#include <linux/mailbox/zynqmp-ipi-message.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/remoteproc.h>
> +#include <linux/skbuff.h>
> +#include <linux/sysfs.h>
> +
> +#include "remoteproc_internal.h"
> +
> +/* settings for RPU cluster mode */
> +enum zynqmp_r5_cluster_mode {
> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
Please use C style comments, i.e /*...*/
> +};
> +
> +/**
> + * struct mem_bank_data - Memory Bank description
> + *
> + * @addr: Start address of memory bank
> + * @size: Size of Memory bank
> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
> + * @bank_name: name of the bank for remoteproc framework
> + */
> +struct mem_bank_data {
> + phys_addr_t addr;
> + size_t size;
> + enum pm_node_id pm_domain_id;
> + char *bank_name;
> +};
> +
> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
> +};
Bizarre - more comments on this below...
> +
> +/**
> + * struct zynqmp_r5_core - ZynqMP R5 core structure
> + *
> + * @dev: device of RPU instance
> + * @np: device node of RPU instance
> + * @tcm_bank_count: number TCM banks accessible to this RPU
> + * @tcm_banks: array of each TCM bank data
> + * @res_mem_count: number of Reserved Memory regions per core
> + * @res_mem: array of reserved memory regions
> + * @rproc: rproc handle
> + * @pm_domain_id: RPU CPU power domain id
> + */
> +struct zynqmp_r5_core {
> + struct device *dev;
> + struct device_node *np;
> + int tcm_bank_count;
> + struct mem_bank_data *tcm_banks;
> + int res_mem_count;
> + struct reserved_mem *res_mem;
> + struct rproc *rproc;
> + enum pm_node_id pm_domain_id;
> +};
> +
> +/**
> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
> + *
> + * @dev: r5f subsystem cluster device node
> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
> + * @core_count: number of r5 cores used for this cluster mode
> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
> + */
> +struct zynqmp_r5_cluster {
> + struct device *dev;
> + enum zynqmp_r5_cluster_mode mode;
> + int core_count;
> + struct zynqmp_r5_core *r5_cores;
> +};
> +
> +/*
> + * zynqmp_r5_set_mode - set RPU operation mode
> + *
> + * set RPU operation mode
> + *
> + * Return: 0 for success, negative value for failure
> + */
> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
> + enum zynqmp_r5_cluster_mode rpu_mode)
> +{
> + enum rpu_tcm_comb tcm_mode;
> + int ret, reg_val;
> +
> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
> +
> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
> + if (ret < 0) {
> + pr_err("failed to set RPU mode\n");
> + return ret;
> + }
> +
> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
> + if (ret < 0)
> + pr_err("failed to configure TCM\n");
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_start
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Start R5 Core from designated boot address.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + enum rpu_boot_mem bootmem;
> + int ret;
> +
> + if (!r5_core) {
> + pr_err("can't get r5 core\n");
> + return -EINVAL;
> + }
> +
> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> +
> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> +
> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> + if (ret)
> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_stop
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Power down R5 Core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + int ret;
> +
> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret)
> + pr_err("failed to stop remoteproc RPU %d\n", ret);
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + mem->va = (void *)va;
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap memory-region carveout
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + iounmap((void __iomem *)mem->va);
> + return 0;
> +}
> +
> +/*
> + * add_mem_regions
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Construct rproc mem carveouts from carveout provided in
> + * memory-region property
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_mem_regions(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct rproc_mem_entry *mem;
> + struct reserved_mem *rmem;
> + struct zynqmp_r5_core *r5_core;
> + int i;
> +
> + r5_core = rproc->priv;
> + dev = r5_core->dev;
> +
> + /* Register associated reserved memory regions */
> + for (i = 0; i < r5_core->res_mem_count; i++) {
> + rmem = &r5_core->res_mem[i];
> + mem = rproc_mem_entry_init(dev, NULL,
> + (dma_addr_t)rmem->base,
> + rmem->size, rmem->base,
> + zynqmp_r5_rproc_mem_map,
> + zynqmp_r5_rproc_mem_unmap,
> + rmem->name);
> + if (IS_ERR_OR_NULL(mem))
> + return -ENOMEM;
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap TCM banks when powering down R5 core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> + struct zynqmp_r5_core *r5_core;
> + int i;
> + enum pm_node_id pm_domain_id;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + pr_err("r5 core is not available\n");
> + return -EINVAL;
> + }
> +
> + iounmap((void __iomem *)mem->va);
> +
> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + if (zynqmp_pm_release_node(pm_domain_id))
> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * tcm_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to initialize the va and da fields of
> + *
> + * Given TCM bank entry, this callback will set device address for R5
> + * running on TCM and also setup virtual address for TCM bank
> + * remoteproc carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + /* Update memory entry va */
> + mem->va = (void *)va;
> +
> + /* clear TCMs */
> + memset_io(va, 0, mem->len);
> +
> + /*
> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> + * while on the Linux side they are at 0xffexxxxx.
> + *
> + * Zero out the high 12 bits of the address. This will give
> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> + */
> + mem->da &= 0x000fffff;
> +
> + /*
> + * TCM Banks 1A and 1B still have to be translated.
> + *
> + * Below handle these two banks' absolute addresses (0xffe90000 and
> + * 0xffeb0000) and convert to the expected relative addresses
> + * (0x0 and 0x20000).
> + */
> + if (mem->da == 0x90000 || mem->da == 0xB0000)
> + mem->da -= 0x90000;
> +
> + /* if translated TCM bank address is not valid report error */
> + if (mem->da != 0x0 && mem->da != 0x20000) {
> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
> + return -EINVAL;
> + }
> + return 0;
> +}
> +
> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
> + char *bank_name;
> + struct device *dev;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + for (i = 0; i < num_banks; i++) {
> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
> + bank_name = r5_core->tcm_banks[i].bank_name;
> + bank_size = r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> +
> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + /* Turn off all TCM banks turned on before */
> + do {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x, %x\n",
> + (u32)pm_domain_id, ret);
> + } while (i--);
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
> + char *bank_name;
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
> + bank_name = r5_core->tcm_banks[0].bank_name;
> + for (i = 0; i < num_banks; i++) {
> + bank_size += r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> + }
> +
> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + for (i = 0; i < num_banks; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x ret: %x\n",
> + (u32)pm_domain_id, ret);
> + }
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> +
> + return 0;
> +}
> +
> +/*
> + * add_tcm_banks()
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Given R5 node in remoteproc instance
> + * allocate remoteproc carveout for TCM memory
> + * needed for firmware to be loaded
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_tcm_banks(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + parent_pdev = to_platform_device(dev->parent);
> + if (!parent_pdev) {
> + dev_err(dev, "parent platform dev unavailable\n");
> + return -ENODEV;
> + }
> +
> + cluster = platform_get_drvdata(parent_pdev);
> + if (!cluster) {
> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
> + return -EINVAL;
> + }
> +
> + if (cluster->mode == SPLIT_MODE)
> + return add_tcm_carveout_split_mode(rproc);
> + else if (cluster->mode == LOCKSTEP_MODE)
> + return add_tcm_carveout_lockstep_mode(rproc);
> +
> + dev_err(cluster->dev, "invalid cluster mode\n");
> + return -EINVAL;
> +}
> +
> +/*
> + * zynqmp_r5_parse_fw()
> + * @rproc: single R5 core's corresponding rproc instance
> + * @fw: ptr to firmware to be loaded onto r5 core
> + *
> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> + int ret;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + dev_err(&rproc->dev, "r5 core not available\n");
> + return -EINVAL;
> + }
> +
> + dev = r5_core->dev;
> +
> + ret = add_tcm_banks(rproc);
> + if (ret) {
> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
> + return ret;
> + }
> +
> + ret = add_mem_regions(rproc);
> + if (ret)
> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
> +
> + ret = rproc_elf_load_rsc_table(rproc, fw);
> + if (ret == -EINVAL) {
> + /*
> + * resource table only required for IPC.
> + * if not present, this is not necessarily an error;
> + * for example, loading r5 hello world application
> + * so simply inform user and keep going.
> + */
> + dev_info(&rproc->dev, "no resource table found.\n");
> + ret = 0;
> + }
> + return ret;
> +}
> +
> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> + .start = zynqmp_r5_rproc_start,
> + .stop = zynqmp_r5_rproc_stop,
> + .load = rproc_elf_load_segments,
> + .parse_fw = zynqmp_r5_parse_fw,
> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> + .sanity_check = rproc_elf_sanity_check,
> + .get_boot_addr = rproc_elf_get_boot_addr,
> +};
> +
> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
> +{
> + int i, j, k;
> + struct zynqmp_r5_core *r5_core;
> +
> + dev_dbg(cluster->dev, "Printing dt node info\n");
Why mixing dev_dbg() and pr_debug()? Please pick one and stick with it.
Even for debug level output this is very chatty - is all that information really
needed when it is already available in /proc/device-tree?
> +
> + pr_debug("cluster mode = %d\n", cluster->mode);
> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
> + pr_debug("r5f num cores = %d\n", cluster->core_count);
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("can't get r5_core\n");
> + continue;
> + }
> +
> + pr_debug("r5 core %d nodes\n", i);
> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
> + k, r5_core->tcm_banks[k].addr,
> + r5_core->tcm_banks[k].size,
> + r5_core->tcm_banks[k].pm_domain_id,
> + r5_core->tcm_banks[k].bank_name);
> + }
> +
> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
> +
> + for (j = 0; j < r5_core->res_mem_count; j++) {
> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
> + j, r5_core->res_mem[j].base,
> + r5_core->res_mem[j].size,
> + r5_core->res_mem[j].name);
> + }
> + }
> +}
> +
> +/**
> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
> + * this is called for each individual R5 core to
> + * set up mailbox, Xilinx platform manager unique ID,
> + * add to rproc core
> + *
> + * @r5_core: zynqmp_r5_core r5 core object to initialize
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
> +{
> + int ret;
> + struct rproc *r5_rproc;
> + struct device *dev;
> +
> + dev = r5_core->dev;
> +
> + /* Set up DMA mask */
> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> + if (ret)
> + return ret;
> +
> + /* Allocate remoteproc instance */
> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> + NULL, sizeof(struct zynqmp_r5_core));
> + if (IS_ERR_OR_NULL(r5_rproc))
> + return -ENOMEM;
> +
> + r5_rproc->auto_boot = false;
> + r5_rproc->priv = r5_core;
> +
> + /* Add R5 remoteproc */
> + ret = devm_rproc_add(dev, r5_rproc);
> + if (ret) {
> + pr_err("failed to add r5 remoteproc\n");
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> +{
> + int tcm_bank_count, tcm_node;
> + int i = 0, j;
> + struct zynqmp_r5_core *r5_core;
> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> + struct device *dev = cluster->dev;
> +
> + /* ToDo: Use predefined TCM address space values from driver until
> + * system-dt spec is not final fot TCM
> + */
Multilined comment should be as follow:
/*
* ToDo: Use predefined TCM address space values from driver until
* system-dt spec is not final fot TCM
*/
s/"final fot TCM"/"final for TCM"
Any reason this can't be done with "reg" properties like TI did for K3? It
would be nice to have TCMs included in the yaml file example.
> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> +
> + /* count per core tcm banks */
> + tcm_bank_count = tcm_bank_count / cluster->core_count;
> +
> + /* r5 core 0 will use all of TCM banks in lockstep mode.
> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> + * 128k. Assign TCM banks to each core accordingly
> + */
> + tcm_node = 0;
> + for (j = 0; j < cluster->core_count; j++) {
> + r5_core = &cluster->r5_cores[j];
> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> + tcm_bank_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> + return -ENOMEM;
> +
> + for (i = 0; i < tcm_bank_count; i++) {
> + /* Use pre-defined TCM reg values.
> + * Eventually this should be replaced by values
> + * parsed from dts.
> + */
> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> + tcm_node++;
> + }
> +
> + r5_core->tcm_bank_count = tcm_bank_count;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> +{
> + int res_mem_count, i;
> + struct device *dev;
> + struct device_node *np, *rmem_np;
> + struct reserved_mem *rmem;
> +
> + dev = r5_core->dev;
> +
> + np = r5_core->np;
> + if (IS_ERR_OR_NULL(np)) {
> + pr_err("invalid device node of r5 core\n");
> + return -EINVAL;
> + }
> +
> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> + sizeof(phandle));
> + if (res_mem_count <= 0) {
> + dev_warn(dev, "failed to get memory-region property %d\n",
> + res_mem_count);
> + return -EINVAL;
> + }
> +
> + r5_core->res_mem = devm_kzalloc(dev,
> + res_mem_count * sizeof(struct reserved_mem),
> + GFP_KERNEL);
> + if (!r5_core->res_mem) {
> + dev_err(dev, "failed to allocate mem region memory\n");
> + return -ENOMEM;
> + }
> +
> + for (i = 0; i < res_mem_count; i++) {
> + rmem_np = of_parse_phandle(np, "memory-region", i);
> + if (!rmem_np)
> + return -EINVAL;
> +
> + rmem = of_reserved_mem_lookup(rmem_np);
> + if (!rmem) {
> + of_node_put(rmem_np);
> + return -EINVAL;
> + }
> +
> + memcpy(&r5_core->res_mem[i], rmem,
> + sizeof(struct reserved_mem));
> + of_node_put(rmem_np);
> + }
> +
> + r5_core->res_mem_count = res_mem_count;
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> +{
> + int ret, i;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev = cluster->dev;
> +
> + ret = zynqmp_r5_get_tcm_node(cluster);
> + if (ret < 0) {
> + dev_err(dev, "can't get tcm node, err %d\n", ret);
> + return ret;
> + }
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("invalid r5 core\n");
> + return -EINVAL;
> + }
> +
> + ret = zynqmp_r5_get_mem_region_node(r5_core);
> + if (ret)
> + dev_warn(dev, "memory-region prop failed %d\n", ret);
> +
> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> + 1, &r5_core->pm_domain_id);
> + if (ret) {
> + dev_err(dev, "failed to get power-domains property\n");
> + return ret;
> + }
> +
> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_add_rproc_core(r5_core);
> + if (ret) {
> + dev_err(dev, "failed to init r5 core %d\n", i);
> + return ret;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> +{
> + struct device *dev = cluster->dev;
> + struct device_node *dev_node = dev_of_node(dev);
> + struct device_node *child;
> + struct platform_device *child_pdev;
> + int core_count = 0, ret, i;
> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> + struct zynqmp_r5_core *r5_cores;
> +
> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> +
> + /* on success returns 0, if not defined then returns -EINVAL,
> + * In that case, default is LOCKSTEP mode
> + */
> + if (ret != -EINVAL && ret != 0) {
> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> + return -EINVAL;
> + }
> +
> + if (cluster_mode == SINGLE_CPU_MODE) {
> + dev_err(dev, "driver does not support single cpu mode\n");
> + return -EINVAL;
> + } else if ((cluster_mode != SPLIT_MODE &&
> + cluster_mode != LOCKSTEP_MODE)) {
> + dev_err(dev, "Invalid cluster mode\n");
> + return -EINVAL;
> + }
> +
> + core_count = of_get_available_child_count(dev_node);
> + if (core_count <= 0) {
> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> + return -EINVAL;
> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> + return -EINVAL;
> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> + dev_warn(dev, "Only r5 core0 will be used\n");
> + core_count = 1;
> + }
> +
> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> + core_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_cores)) {
> + dev_err(dev, "can't allocate memory for cores\n");
> + return -ENOMEM;
> + }
> +
> + i = 0;
> + for_each_available_child_of_node(dev_node, child) {
> + child_pdev = of_find_device_by_node(child);
> + if (!child_pdev)
> + return -ENODEV;
> +
> + r5_cores[i].dev = &child_pdev->dev;
> + if (!r5_cores[i].dev) {
> + pr_err("can't get device for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> + if (!r5_cores[i].np) {
> + pr_err("can't get device node for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + i++;
> + if (i == core_count)
> + break;
> + }
> +
> + cluster->mode = cluster_mode;
> + cluster->core_count = core_count;
> + cluster->r5_cores = r5_cores;
> +
> + ret = zynqmp_r5_core_init(cluster);
> + if (ret < 0) {
> + dev_err(dev, "failed to init r5 core err %d\n", ret);
> + return ret;
> + }
> +
> + zynqmp_r5_print_dt_node_info(cluster);
> +
> + return 0;
> +}
> +
> +static void zynqmp_r5_cluster_exit(void *data)
> +{
> + struct platform_device *pdev = (struct platform_device *)data;
> +
> + platform_set_drvdata(pdev, NULL);
> +
> + pr_info("Exit r5f subsystem driver\n");
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_probe()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * called when driver is probed, for each R5 core specified in DT,
> + * setup as needed to do remoteproc-related operations
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> +{
> + int ret;
> + struct zynqmp_r5_cluster *cluster;
> + struct device *dev = &pdev->dev;
> +
> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> + if (IS_ERR_OR_NULL(cluster))
> + return -ENOMEM;
Function devm_kzalloc() does not return an code on error, just NULL. Please fix
throughout the driver.
> +
> + cluster->dev = dev;
> +
> + ret = devm_of_platform_populate(dev);
> + if (ret) {
> + dev_err(dev, "failed to populate platform dev %d\n", ret);
> + return ret;
> + }
> +
> + /* wire in so each core can be cleaned up at driver remove */
> + platform_set_drvdata(pdev, cluster);
> +
> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_cluster_init(cluster);
> + if (ret) {
> + dev_err(dev, "Invalid r5f subsystem device tree\n");
> + return ret;
> + }
> +
> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
Please remove this.
I am out of time for today and will continue on Monday.
Thanks,
Mathieu
> + return 0;
> +}
> +
> +/* Match table for OF platform binding */
> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> + { .compatible = "xlnx,zynqmp-r5fss", },
> + { /* end of list */ },
> +};
> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> +
> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> + .probe = zynqmp_r5_remoteproc_probe,
> + .driver = {
> + .name = "zynqmp_r5_remoteproc",
> + .of_match_table = zynqmp_r5_remoteproc_match,
> + },
> +};
> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> +
> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> +MODULE_AUTHOR("Xilinx Inc.");
> +MODULE_LICENSE("GPL v2");
> --
> 2.25.1
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-12-03 18:55 ` Mathieu Poirier
@ 2021-12-06 11:03 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-06 11:03 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
Hi Mathieu,
On 12/4/21 12:25 AM, Mathieu Poirier wrote:
> Hi Tanmay,
>
> On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
>> This driver enables r5f dual core Real time Processing Unit subsystem
>> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
>> (cluster) can be configured in different modes e.g. split mode in which
>> two r5f cores work independent of each other and lock-step mode in which
>> both r5f cores execute same code clock-for-clock and notify if the
>> result is different.
>>
>> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
>> Platform Management Unit that handles the R5 configuration, memory access
>> and R5 lifecycle management. The interface to this manager is done in this
>> driver via zynqmp_pm_* function calls.
>>
>> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
>> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
>> ---
>> drivers/remoteproc/Kconfig | 12 +
>> drivers/remoteproc/Makefile | 1 +
>> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
>> 3 files changed, 972 insertions(+)
>> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>>
>> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
>> index f30d00a3aabe..27f66910d8d3 100644
>> --- a/drivers/remoteproc/Kconfig
>> +++ b/drivers/remoteproc/Kconfig
>> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
>> It's safe to say N here if you're not interested in utilizing
>> a slave processor.
>>
>> +config XLNX_R5_REMOTEPROC
>> + tristate "Xilinx R5 remoteproc support"
>> + depends on PM && ARCH_ZYNQMP
>> + depends on ZYNQMP_FIRMWARE
>> + select RPMSG_VIRTIO
>> + select ZYNQMP_IPI_MBOX
>> + help
>> + Say y or m here to support Xilinx R5 remote processors via the remote
>> + processor framework.
>> +
>> + It's safe to say N if not interested in using RPU r5f cores.
>> +
>> endif # REMOTEPROC
>>
>> endmenu
>> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
>> index bb26c9e4ef9c..334a8bed4c14 100644
>> --- a/drivers/remoteproc/Makefile
>> +++ b/drivers/remoteproc/Makefile
>> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
>> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
>> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
>> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
>> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
>> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> new file mode 100644
>> index 000000000000..c2167fd3869d
>> --- /dev/null
>> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> @@ -0,0 +1,959 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ZynqMP R5 Remote Processor driver
>> + *
>> + */
>> +
>> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
>> +#include <linux/firmware/xlnx-zynqmp.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/kernel.h>
>> +#include <linux/list.h>
>> +#include <linux/mailbox_client.h>
>> +#include <linux/mailbox/zynqmp-ipi-message.h>
>> +#include <linux/module.h>
>> +#include <linux/of_address.h>
>> +#include <linux/of_platform.h>
>> +#include <linux/of_reserved_mem.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/remoteproc.h>
>> +#include <linux/skbuff.h>
>> +#include <linux/sysfs.h>
>> +
>> +#include "remoteproc_internal.h"
>> +
>> +/* settings for RPU cluster mode */
>> +enum zynqmp_r5_cluster_mode {
>> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
>> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
>> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
> Please use C style comments, i.e /*...*/
>
>> +};
>> +
>> +/**
>> + * struct mem_bank_data - Memory Bank description
>> + *
>> + * @addr: Start address of memory bank
>> + * @size: Size of Memory bank
>> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
>> + * @bank_name: name of the bank for remoteproc framework
>> + */
>> +struct mem_bank_data {
>> + phys_addr_t addr;
>> + size_t size;
>> + enum pm_node_id pm_domain_id;
>> + char *bank_name;
>> +};
>> +
>> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
>> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
>> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
>> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
>> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
>> +};
> Bizarre - more comments on this below...
>
>> +
>> +/**
>> + * struct zynqmp_r5_core - ZynqMP R5 core structure
>> + *
>> + * @dev: device of RPU instance
>> + * @np: device node of RPU instance
>> + * @tcm_bank_count: number TCM banks accessible to this RPU
>> + * @tcm_banks: array of each TCM bank data
>> + * @res_mem_count: number of Reserved Memory regions per core
>> + * @res_mem: array of reserved memory regions
>> + * @rproc: rproc handle
>> + * @pm_domain_id: RPU CPU power domain id
>> + */
>> +struct zynqmp_r5_core {
>> + struct device *dev;
>> + struct device_node *np;
>> + int tcm_bank_count;
>> + struct mem_bank_data *tcm_banks;
>> + int res_mem_count;
>> + struct reserved_mem *res_mem;
>> + struct rproc *rproc;
>> + enum pm_node_id pm_domain_id;
>> +};
>> +
>> +/**
>> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
>> + *
>> + * @dev: r5f subsystem cluster device node
>> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
>> + * @core_count: number of r5 cores used for this cluster mode
>> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
>> + */
>> +struct zynqmp_r5_cluster {
>> + struct device *dev;
>> + enum zynqmp_r5_cluster_mode mode;
>> + int core_count;
>> + struct zynqmp_r5_core *r5_cores;
>> +};
>> +
>> +/*
>> + * zynqmp_r5_set_mode - set RPU operation mode
>> + *
>> + * set RPU operation mode
>> + *
>> + * Return: 0 for success, negative value for failure
>> + */
>> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
>> + enum zynqmp_r5_cluster_mode rpu_mode)
>> +{
>> + enum rpu_tcm_comb tcm_mode;
>> + int ret, reg_val;
>> +
>> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
>> +
>> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
>> + if (ret < 0) {
>> + pr_err("failed to set RPU mode\n");
>> + return ret;
>> + }
>> +
>> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
>> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
>> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
>> + if (ret < 0)
>> + pr_err("failed to configure TCM\n");
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_start
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Start R5 Core from designated boot address.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + enum rpu_boot_mem bootmem;
>> + int ret;
>> +
>> + if (!r5_core) {
>> + pr_err("can't get r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
>> +
>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>> +
>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>> + if (ret)
>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_stop
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Power down R5 Core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + int ret;
>> +
>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret)
>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap memory-region carveout
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + iounmap((void __iomem *)mem->va);
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_mem_regions
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Construct rproc mem carveouts from carveout provided in
>> + * memory-region property
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_mem_regions(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct rproc_mem_entry *mem;
>> + struct reserved_mem *rmem;
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> +
>> + r5_core = rproc->priv;
>> + dev = r5_core->dev;
>> +
>> + /* Register associated reserved memory regions */
>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>> + rmem = &r5_core->res_mem[i];
>> + mem = rproc_mem_entry_init(dev, NULL,
>> + (dma_addr_t)rmem->base,
>> + rmem->size, rmem->base,
>> + zynqmp_r5_rproc_mem_map,
>> + zynqmp_r5_rproc_mem_unmap,
>> + rmem->name);
>> + if (IS_ERR_OR_NULL(mem))
>> + return -ENOMEM;
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap TCM banks when powering down R5 core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>> +{
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> + enum pm_node_id pm_domain_id;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + pr_err("r5 core is not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + iounmap((void __iomem *)mem->va);
>> +
>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + if (zynqmp_pm_release_node(pm_domain_id))
>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * tcm_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to initialize the va and da fields of
>> + *
>> + * Given TCM bank entry, this callback will set device address for R5
>> + * running on TCM and also setup virtual address for TCM bank
>> + * remoteproc carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + /* Update memory entry va */
>> + mem->va = (void *)va;
>> +
>> + /* clear TCMs */
>> + memset_io(va, 0, mem->len);
>> +
>> + /*
>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>> + * while on the Linux side they are at 0xffexxxxx.
>> + *
>> + * Zero out the high 12 bits of the address. This will give
>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>> + */
>> + mem->da &= 0x000fffff;
>> +
>> + /*
>> + * TCM Banks 1A and 1B still have to be translated.
>> + *
>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>> + * 0xffeb0000) and convert to the expected relative addresses
>> + * (0x0 and 0x20000).
>> + */
>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>> + mem->da -= 0x90000;
>> +
>> + /* if translated TCM bank address is not valid report error */
>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>> + return -EINVAL;
>> + }
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; i < num_banks; i++) {
>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>> + bank_name = r5_core->tcm_banks[i].bank_name;
>> + bank_size = r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + /* Turn off all TCM banks turned on before */
>> + do {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x, %x\n",
>> + (u32)pm_domain_id, ret);
>> + } while (i--);
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>> + bank_name = r5_core->tcm_banks[0].bank_name;
>> + for (i = 0; i < num_banks; i++) {
>> + bank_size += r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> + }
>> +
>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + for (i = 0; i < num_banks; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x ret: %x\n",
>> + (u32)pm_domain_id, ret);
>> + }
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_tcm_banks()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Given R5 node in remoteproc instance
>> + * allocate remoteproc carveout for TCM memory
>> + * needed for firmware to be loaded
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_parse_fw()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @fw: ptr to firmware to be loaded onto r5 core
>> + *
>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>> +{
>> + int ret;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + dev_err(&rproc->dev, "r5 core not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + dev = r5_core->dev;
>> +
>> + ret = add_tcm_banks(rproc);
>> + if (ret) {
>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = add_mem_regions(rproc);
>> + if (ret)
>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>> +
>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>> + if (ret == -EINVAL) {
>> + /*
>> + * resource table only required for IPC.
>> + * if not present, this is not necessarily an error;
>> + * for example, loading r5 hello world application
>> + * so simply inform user and keep going.
>> + */
>> + dev_info(&rproc->dev, "no resource table found.\n");
>> + ret = 0;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> +
>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int i, j, k;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + dev_dbg(cluster->dev, "Printing dt node info\n");
> Why mixing dev_dbg() and pr_debug()? Please pick one and stick with it.
>
> Even for debug level output this is very chatty - is all that information really
> needed when it is already available in /proc/device-tree?
>
>> +
>> + pr_debug("cluster mode = %d\n", cluster->mode);
>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("can't get r5_core\n");
>> + continue;
>> + }
>> +
>> + pr_debug("r5 core %d nodes\n", i);
>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>> + k, r5_core->tcm_banks[k].addr,
>> + r5_core->tcm_banks[k].size,
>> + r5_core->tcm_banks[k].pm_domain_id,
>> + r5_core->tcm_banks[k].bank_name);
>> + }
>> +
>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>> +
>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>> + j, r5_core->res_mem[j].base,
>> + r5_core->res_mem[j].size,
>> + r5_core->res_mem[j].name);
>> + }
>> + }
>> +}
>> +
>> +/**
>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>> + * this is called for each individual R5 core to
>> + * set up mailbox, Xilinx platform manager unique ID,
>> + * add to rproc core
>> + *
>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>> +{
>> + int ret;
>> + struct rproc *r5_rproc;
>> + struct device *dev;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* Set up DMA mask */
>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>> + if (ret)
>> + return ret;
>> +
>> + /* Allocate remoteproc instance */
>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>> + NULL, sizeof(struct zynqmp_r5_core));
>> + if (IS_ERR_OR_NULL(r5_rproc))
>> + return -ENOMEM;
>> +
>> + r5_rproc->auto_boot = false;
>> + r5_rproc->priv = r5_core;
>> +
>> + /* Add R5 remoteproc */
>> + ret = devm_rproc_add(dev, r5_rproc);
>> + if (ret) {
>> + pr_err("failed to add r5 remoteproc\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int tcm_bank_count, tcm_node;
>> + int i = 0, j;
>> + struct zynqmp_r5_core *r5_core;
>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>> + struct device *dev = cluster->dev;
>> +
>> + /* ToDo: Use predefined TCM address space values from driver until
>> + * system-dt spec is not final fot TCM
>> + */
> Multilined comment should be as follow:
>
> /*
> * ToDo: Use predefined TCM address space values from driver until
> * system-dt spec is not final fot TCM
> */
>
> s/"final fot TCM"/"final for TCM"
>
> Any reason this can't be done with "reg" properties like TI did for K3? It
> would be nice to have TCMs included in the yaml file example.
>
>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>> +
>> + /* count per core tcm banks */
>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>> +
>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>> + * 128k. Assign TCM banks to each core accordingly
>> + */
>> + tcm_node = 0;
>> + for (j = 0; j < cluster->core_count; j++) {
>> + r5_core = &cluster->r5_cores[j];
>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>> + tcm_bank_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>> + return -ENOMEM;
>> +
>> + for (i = 0; i < tcm_bank_count; i++) {
>> + /* Use pre-defined TCM reg values.
>> + * Eventually this should be replaced by values
>> + * parsed from dts.
>> + */
>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>> + tcm_node++;
>> + }
>> +
>> + r5_core->tcm_bank_count = tcm_bank_count;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>> +{
>> + int res_mem_count, i;
>> + struct device *dev;
>> + struct device_node *np, *rmem_np;
>> + struct reserved_mem *rmem;
>> +
>> + dev = r5_core->dev;
>> +
>> + np = r5_core->np;
>> + if (IS_ERR_OR_NULL(np)) {
>> + pr_err("invalid device node of r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>> + sizeof(phandle));
>> + if (res_mem_count <= 0) {
>> + dev_warn(dev, "failed to get memory-region property %d\n",
>> + res_mem_count);
>> + return -EINVAL;
>> + }
>> +
>> + r5_core->res_mem = devm_kzalloc(dev,
>> + res_mem_count * sizeof(struct reserved_mem),
>> + GFP_KERNEL);
>> + if (!r5_core->res_mem) {
>> + dev_err(dev, "failed to allocate mem region memory\n");
>> + return -ENOMEM;
>> + }
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int ret, i;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev = cluster->dev;
>> +
>> + ret = zynqmp_r5_get_tcm_node(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("invalid r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>> + if (ret)
>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>> +
>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>> + 1, &r5_core->pm_domain_id);
>> + if (ret) {
>> + dev_err(dev, "failed to get power-domains property\n");
>> + return ret;
>> + }
>> +
>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>> + if (ret) {
>> + dev_err(dev, "failed to init r5 core %d\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + struct device *dev = cluster->dev;
>> + struct device_node *dev_node = dev_of_node(dev);
>> + struct device_node *child;
>> + struct platform_device *child_pdev;
>> + int core_count = 0, ret, i;
>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>> + struct zynqmp_r5_core *r5_cores;
>> +
>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>> +
>> + /* on success returns 0, if not defined then returns -EINVAL,
>> + * In that case, default is LOCKSTEP mode
>> + */
>> + if (ret != -EINVAL && ret != 0) {
>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster_mode == SINGLE_CPU_MODE) {
>> + dev_err(dev, "driver does not support single cpu mode\n");
>> + return -EINVAL;
>> + } else if ((cluster_mode != SPLIT_MODE &&
>> + cluster_mode != LOCKSTEP_MODE)) {
>> + dev_err(dev, "Invalid cluster mode\n");
>> + return -EINVAL;
>> + }
>> +
>> + core_count = of_get_available_child_count(dev_node);
>> + if (core_count <= 0) {
>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>> + return -EINVAL;
>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>> + return -EINVAL;
>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>> + dev_warn(dev, "Only r5 core0 will be used\n");
>> + core_count = 1;
>> + }
>> +
>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>> + core_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_cores)) {
>> + dev_err(dev, "can't allocate memory for cores\n");
>> + return -ENOMEM;
>> + }
>> +
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
>> + return -ENODEV;
>> +
>> + r5_cores[i].dev = &child_pdev->dev;
>> + if (!r5_cores[i].dev) {
>> + pr_err("can't get device for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>> + if (!r5_cores[i].np) {
>> + pr_err("can't get device node for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + i++;
>> + if (i == core_count)
>> + break;
>> + }
>> +
>> + cluster->mode = cluster_mode;
>> + cluster->core_count = core_count;
>> + cluster->r5_cores = r5_cores;
>> +
>> + ret = zynqmp_r5_core_init(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + zynqmp_r5_print_dt_node_info(cluster);
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
>> +
>> + pr_info("Exit r5f subsystem driver\n");
>> +}
>> +
>> +/*
>> + * zynqmp_r5_remoteproc_probe()
>> + *
>> + * @pdev: domain platform device for R5 cluster
>> + *
>> + * called when driver is probed, for each R5 core specified in DT,
>> + * setup as needed to do remoteproc-related operations
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>> +{
>> + int ret;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct device *dev = &pdev->dev;
>> +
>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(cluster))
>> + return -ENOMEM;
> Function devm_kzalloc() does not return an code on error, just NULL. Please fix
> throughout the driver.
>
>> +
>> + cluster->dev = dev;
>> +
>> + ret = devm_of_platform_populate(dev);
>> + if (ret) {
>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /* wire in so each core can be cleaned up at driver remove */
>> + platform_set_drvdata(pdev, cluster);
>> +
>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_cluster_init(cluster);
>> + if (ret) {
>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>> + return ret;
>> + }
>> +
>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> Please remove this.
>
> I am out of time for today and will continue on Monday.
Thanks for reviews. Sure, I will respond once you are done with reviews.
>
> Thanks,
> Mathieu
>
>> + return 0;
>> +}
>> +
>> +/* Match table for OF platform binding */
>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>> + { .compatible = "xlnx,zynqmp-r5fss", },
>> + { /* end of list */ },
>> +};
>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>> +
>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>> + .probe = zynqmp_r5_remoteproc_probe,
>> + .driver = {
>> + .name = "zynqmp_r5_remoteproc",
>> + .of_match_table = zynqmp_r5_remoteproc_match,
>> + },
>> +};
>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>> +
>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>> +MODULE_AUTHOR("Xilinx Inc.");
>> +MODULE_LICENSE("GPL v2");
>> --
>> 2.25.1
>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-06 11:03 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-06 11:03 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
Hi Mathieu,
On 12/4/21 12:25 AM, Mathieu Poirier wrote:
> Hi Tanmay,
>
> On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
>> This driver enables r5f dual core Real time Processing Unit subsystem
>> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
>> (cluster) can be configured in different modes e.g. split mode in which
>> two r5f cores work independent of each other and lock-step mode in which
>> both r5f cores execute same code clock-for-clock and notify if the
>> result is different.
>>
>> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
>> Platform Management Unit that handles the R5 configuration, memory access
>> and R5 lifecycle management. The interface to this manager is done in this
>> driver via zynqmp_pm_* function calls.
>>
>> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
>> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
>> ---
>> drivers/remoteproc/Kconfig | 12 +
>> drivers/remoteproc/Makefile | 1 +
>> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
>> 3 files changed, 972 insertions(+)
>> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>>
>> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
>> index f30d00a3aabe..27f66910d8d3 100644
>> --- a/drivers/remoteproc/Kconfig
>> +++ b/drivers/remoteproc/Kconfig
>> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
>> It's safe to say N here if you're not interested in utilizing
>> a slave processor.
>>
>> +config XLNX_R5_REMOTEPROC
>> + tristate "Xilinx R5 remoteproc support"
>> + depends on PM && ARCH_ZYNQMP
>> + depends on ZYNQMP_FIRMWARE
>> + select RPMSG_VIRTIO
>> + select ZYNQMP_IPI_MBOX
>> + help
>> + Say y or m here to support Xilinx R5 remote processors via the remote
>> + processor framework.
>> +
>> + It's safe to say N if not interested in using RPU r5f cores.
>> +
>> endif # REMOTEPROC
>>
>> endmenu
>> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
>> index bb26c9e4ef9c..334a8bed4c14 100644
>> --- a/drivers/remoteproc/Makefile
>> +++ b/drivers/remoteproc/Makefile
>> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
>> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
>> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
>> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
>> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
>> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> new file mode 100644
>> index 000000000000..c2167fd3869d
>> --- /dev/null
>> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> @@ -0,0 +1,959 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ZynqMP R5 Remote Processor driver
>> + *
>> + */
>> +
>> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
>> +#include <linux/firmware/xlnx-zynqmp.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/kernel.h>
>> +#include <linux/list.h>
>> +#include <linux/mailbox_client.h>
>> +#include <linux/mailbox/zynqmp-ipi-message.h>
>> +#include <linux/module.h>
>> +#include <linux/of_address.h>
>> +#include <linux/of_platform.h>
>> +#include <linux/of_reserved_mem.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/remoteproc.h>
>> +#include <linux/skbuff.h>
>> +#include <linux/sysfs.h>
>> +
>> +#include "remoteproc_internal.h"
>> +
>> +/* settings for RPU cluster mode */
>> +enum zynqmp_r5_cluster_mode {
>> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
>> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
>> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
> Please use C style comments, i.e /*...*/
>
>> +};
>> +
>> +/**
>> + * struct mem_bank_data - Memory Bank description
>> + *
>> + * @addr: Start address of memory bank
>> + * @size: Size of Memory bank
>> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
>> + * @bank_name: name of the bank for remoteproc framework
>> + */
>> +struct mem_bank_data {
>> + phys_addr_t addr;
>> + size_t size;
>> + enum pm_node_id pm_domain_id;
>> + char *bank_name;
>> +};
>> +
>> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
>> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
>> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
>> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
>> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
>> +};
> Bizarre - more comments on this below...
>
>> +
>> +/**
>> + * struct zynqmp_r5_core - ZynqMP R5 core structure
>> + *
>> + * @dev: device of RPU instance
>> + * @np: device node of RPU instance
>> + * @tcm_bank_count: number TCM banks accessible to this RPU
>> + * @tcm_banks: array of each TCM bank data
>> + * @res_mem_count: number of Reserved Memory regions per core
>> + * @res_mem: array of reserved memory regions
>> + * @rproc: rproc handle
>> + * @pm_domain_id: RPU CPU power domain id
>> + */
>> +struct zynqmp_r5_core {
>> + struct device *dev;
>> + struct device_node *np;
>> + int tcm_bank_count;
>> + struct mem_bank_data *tcm_banks;
>> + int res_mem_count;
>> + struct reserved_mem *res_mem;
>> + struct rproc *rproc;
>> + enum pm_node_id pm_domain_id;
>> +};
>> +
>> +/**
>> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
>> + *
>> + * @dev: r5f subsystem cluster device node
>> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
>> + * @core_count: number of r5 cores used for this cluster mode
>> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
>> + */
>> +struct zynqmp_r5_cluster {
>> + struct device *dev;
>> + enum zynqmp_r5_cluster_mode mode;
>> + int core_count;
>> + struct zynqmp_r5_core *r5_cores;
>> +};
>> +
>> +/*
>> + * zynqmp_r5_set_mode - set RPU operation mode
>> + *
>> + * set RPU operation mode
>> + *
>> + * Return: 0 for success, negative value for failure
>> + */
>> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
>> + enum zynqmp_r5_cluster_mode rpu_mode)
>> +{
>> + enum rpu_tcm_comb tcm_mode;
>> + int ret, reg_val;
>> +
>> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
>> +
>> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
>> + if (ret < 0) {
>> + pr_err("failed to set RPU mode\n");
>> + return ret;
>> + }
>> +
>> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
>> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
>> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
>> + if (ret < 0)
>> + pr_err("failed to configure TCM\n");
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_start
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Start R5 Core from designated boot address.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + enum rpu_boot_mem bootmem;
>> + int ret;
>> +
>> + if (!r5_core) {
>> + pr_err("can't get r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
>> +
>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>> +
>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>> + if (ret)
>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_stop
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Power down R5 Core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + int ret;
>> +
>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret)
>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap memory-region carveout
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + iounmap((void __iomem *)mem->va);
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_mem_regions
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Construct rproc mem carveouts from carveout provided in
>> + * memory-region property
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_mem_regions(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct rproc_mem_entry *mem;
>> + struct reserved_mem *rmem;
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> +
>> + r5_core = rproc->priv;
>> + dev = r5_core->dev;
>> +
>> + /* Register associated reserved memory regions */
>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>> + rmem = &r5_core->res_mem[i];
>> + mem = rproc_mem_entry_init(dev, NULL,
>> + (dma_addr_t)rmem->base,
>> + rmem->size, rmem->base,
>> + zynqmp_r5_rproc_mem_map,
>> + zynqmp_r5_rproc_mem_unmap,
>> + rmem->name);
>> + if (IS_ERR_OR_NULL(mem))
>> + return -ENOMEM;
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap TCM banks when powering down R5 core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>> +{
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> + enum pm_node_id pm_domain_id;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + pr_err("r5 core is not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + iounmap((void __iomem *)mem->va);
>> +
>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + if (zynqmp_pm_release_node(pm_domain_id))
>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * tcm_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to initialize the va and da fields of
>> + *
>> + * Given TCM bank entry, this callback will set device address for R5
>> + * running on TCM and also setup virtual address for TCM bank
>> + * remoteproc carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + /* Update memory entry va */
>> + mem->va = (void *)va;
>> +
>> + /* clear TCMs */
>> + memset_io(va, 0, mem->len);
>> +
>> + /*
>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>> + * while on the Linux side they are at 0xffexxxxx.
>> + *
>> + * Zero out the high 12 bits of the address. This will give
>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>> + */
>> + mem->da &= 0x000fffff;
>> +
>> + /*
>> + * TCM Banks 1A and 1B still have to be translated.
>> + *
>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>> + * 0xffeb0000) and convert to the expected relative addresses
>> + * (0x0 and 0x20000).
>> + */
>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>> + mem->da -= 0x90000;
>> +
>> + /* if translated TCM bank address is not valid report error */
>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>> + return -EINVAL;
>> + }
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; i < num_banks; i++) {
>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>> + bank_name = r5_core->tcm_banks[i].bank_name;
>> + bank_size = r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + /* Turn off all TCM banks turned on before */
>> + do {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x, %x\n",
>> + (u32)pm_domain_id, ret);
>> + } while (i--);
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>> + bank_name = r5_core->tcm_banks[0].bank_name;
>> + for (i = 0; i < num_banks; i++) {
>> + bank_size += r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> + }
>> +
>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + for (i = 0; i < num_banks; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x ret: %x\n",
>> + (u32)pm_domain_id, ret);
>> + }
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_tcm_banks()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Given R5 node in remoteproc instance
>> + * allocate remoteproc carveout for TCM memory
>> + * needed for firmware to be loaded
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_parse_fw()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @fw: ptr to firmware to be loaded onto r5 core
>> + *
>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>> +{
>> + int ret;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + dev_err(&rproc->dev, "r5 core not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + dev = r5_core->dev;
>> +
>> + ret = add_tcm_banks(rproc);
>> + if (ret) {
>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = add_mem_regions(rproc);
>> + if (ret)
>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>> +
>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>> + if (ret == -EINVAL) {
>> + /*
>> + * resource table only required for IPC.
>> + * if not present, this is not necessarily an error;
>> + * for example, loading r5 hello world application
>> + * so simply inform user and keep going.
>> + */
>> + dev_info(&rproc->dev, "no resource table found.\n");
>> + ret = 0;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> +
>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int i, j, k;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + dev_dbg(cluster->dev, "Printing dt node info\n");
> Why mixing dev_dbg() and pr_debug()? Please pick one and stick with it.
>
> Even for debug level output this is very chatty - is all that information really
> needed when it is already available in /proc/device-tree?
>
>> +
>> + pr_debug("cluster mode = %d\n", cluster->mode);
>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("can't get r5_core\n");
>> + continue;
>> + }
>> +
>> + pr_debug("r5 core %d nodes\n", i);
>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>> + k, r5_core->tcm_banks[k].addr,
>> + r5_core->tcm_banks[k].size,
>> + r5_core->tcm_banks[k].pm_domain_id,
>> + r5_core->tcm_banks[k].bank_name);
>> + }
>> +
>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>> +
>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>> + j, r5_core->res_mem[j].base,
>> + r5_core->res_mem[j].size,
>> + r5_core->res_mem[j].name);
>> + }
>> + }
>> +}
>> +
>> +/**
>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>> + * this is called for each individual R5 core to
>> + * set up mailbox, Xilinx platform manager unique ID,
>> + * add to rproc core
>> + *
>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>> +{
>> + int ret;
>> + struct rproc *r5_rproc;
>> + struct device *dev;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* Set up DMA mask */
>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>> + if (ret)
>> + return ret;
>> +
>> + /* Allocate remoteproc instance */
>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>> + NULL, sizeof(struct zynqmp_r5_core));
>> + if (IS_ERR_OR_NULL(r5_rproc))
>> + return -ENOMEM;
>> +
>> + r5_rproc->auto_boot = false;
>> + r5_rproc->priv = r5_core;
>> +
>> + /* Add R5 remoteproc */
>> + ret = devm_rproc_add(dev, r5_rproc);
>> + if (ret) {
>> + pr_err("failed to add r5 remoteproc\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int tcm_bank_count, tcm_node;
>> + int i = 0, j;
>> + struct zynqmp_r5_core *r5_core;
>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>> + struct device *dev = cluster->dev;
>> +
>> + /* ToDo: Use predefined TCM address space values from driver until
>> + * system-dt spec is not final fot TCM
>> + */
> Multilined comment should be as follow:
>
> /*
> * ToDo: Use predefined TCM address space values from driver until
> * system-dt spec is not final fot TCM
> */
>
> s/"final fot TCM"/"final for TCM"
>
> Any reason this can't be done with "reg" properties like TI did for K3? It
> would be nice to have TCMs included in the yaml file example.
>
>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>> +
>> + /* count per core tcm banks */
>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>> +
>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>> + * 128k. Assign TCM banks to each core accordingly
>> + */
>> + tcm_node = 0;
>> + for (j = 0; j < cluster->core_count; j++) {
>> + r5_core = &cluster->r5_cores[j];
>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>> + tcm_bank_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>> + return -ENOMEM;
>> +
>> + for (i = 0; i < tcm_bank_count; i++) {
>> + /* Use pre-defined TCM reg values.
>> + * Eventually this should be replaced by values
>> + * parsed from dts.
>> + */
>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>> + tcm_node++;
>> + }
>> +
>> + r5_core->tcm_bank_count = tcm_bank_count;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>> +{
>> + int res_mem_count, i;
>> + struct device *dev;
>> + struct device_node *np, *rmem_np;
>> + struct reserved_mem *rmem;
>> +
>> + dev = r5_core->dev;
>> +
>> + np = r5_core->np;
>> + if (IS_ERR_OR_NULL(np)) {
>> + pr_err("invalid device node of r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>> + sizeof(phandle));
>> + if (res_mem_count <= 0) {
>> + dev_warn(dev, "failed to get memory-region property %d\n",
>> + res_mem_count);
>> + return -EINVAL;
>> + }
>> +
>> + r5_core->res_mem = devm_kzalloc(dev,
>> + res_mem_count * sizeof(struct reserved_mem),
>> + GFP_KERNEL);
>> + if (!r5_core->res_mem) {
>> + dev_err(dev, "failed to allocate mem region memory\n");
>> + return -ENOMEM;
>> + }
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int ret, i;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev = cluster->dev;
>> +
>> + ret = zynqmp_r5_get_tcm_node(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("invalid r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>> + if (ret)
>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>> +
>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>> + 1, &r5_core->pm_domain_id);
>> + if (ret) {
>> + dev_err(dev, "failed to get power-domains property\n");
>> + return ret;
>> + }
>> +
>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>> + if (ret) {
>> + dev_err(dev, "failed to init r5 core %d\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + struct device *dev = cluster->dev;
>> + struct device_node *dev_node = dev_of_node(dev);
>> + struct device_node *child;
>> + struct platform_device *child_pdev;
>> + int core_count = 0, ret, i;
>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>> + struct zynqmp_r5_core *r5_cores;
>> +
>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>> +
>> + /* on success returns 0, if not defined then returns -EINVAL,
>> + * In that case, default is LOCKSTEP mode
>> + */
>> + if (ret != -EINVAL && ret != 0) {
>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster_mode == SINGLE_CPU_MODE) {
>> + dev_err(dev, "driver does not support single cpu mode\n");
>> + return -EINVAL;
>> + } else if ((cluster_mode != SPLIT_MODE &&
>> + cluster_mode != LOCKSTEP_MODE)) {
>> + dev_err(dev, "Invalid cluster mode\n");
>> + return -EINVAL;
>> + }
>> +
>> + core_count = of_get_available_child_count(dev_node);
>> + if (core_count <= 0) {
>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>> + return -EINVAL;
>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>> + return -EINVAL;
>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>> + dev_warn(dev, "Only r5 core0 will be used\n");
>> + core_count = 1;
>> + }
>> +
>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>> + core_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_cores)) {
>> + dev_err(dev, "can't allocate memory for cores\n");
>> + return -ENOMEM;
>> + }
>> +
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
>> + return -ENODEV;
>> +
>> + r5_cores[i].dev = &child_pdev->dev;
>> + if (!r5_cores[i].dev) {
>> + pr_err("can't get device for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>> + if (!r5_cores[i].np) {
>> + pr_err("can't get device node for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + i++;
>> + if (i == core_count)
>> + break;
>> + }
>> +
>> + cluster->mode = cluster_mode;
>> + cluster->core_count = core_count;
>> + cluster->r5_cores = r5_cores;
>> +
>> + ret = zynqmp_r5_core_init(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + zynqmp_r5_print_dt_node_info(cluster);
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
>> +
>> + pr_info("Exit r5f subsystem driver\n");
>> +}
>> +
>> +/*
>> + * zynqmp_r5_remoteproc_probe()
>> + *
>> + * @pdev: domain platform device for R5 cluster
>> + *
>> + * called when driver is probed, for each R5 core specified in DT,
>> + * setup as needed to do remoteproc-related operations
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>> +{
>> + int ret;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct device *dev = &pdev->dev;
>> +
>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(cluster))
>> + return -ENOMEM;
> Function devm_kzalloc() does not return an code on error, just NULL. Please fix
> throughout the driver.
>
>> +
>> + cluster->dev = dev;
>> +
>> + ret = devm_of_platform_populate(dev);
>> + if (ret) {
>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /* wire in so each core can be cleaned up at driver remove */
>> + platform_set_drvdata(pdev, cluster);
>> +
>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_cluster_init(cluster);
>> + if (ret) {
>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>> + return ret;
>> + }
>> +
>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> Please remove this.
>
> I am out of time for today and will continue on Monday.
Thanks for reviews. Sure, I will respond once you are done with reviews.
>
> Thanks,
> Mathieu
>
>> + return 0;
>> +}
>> +
>> +/* Match table for OF platform binding */
>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>> + { .compatible = "xlnx,zynqmp-r5fss", },
>> + { /* end of list */ },
>> +};
>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>> +
>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>> + .probe = zynqmp_r5_remoteproc_probe,
>> + .driver = {
>> + .name = "zynqmp_r5_remoteproc",
>> + .of_match_table = zynqmp_r5_remoteproc_match,
>> + },
>> +};
>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>> +
>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>> +MODULE_AUTHOR("Xilinx Inc.");
>> +MODULE_LICENSE("GPL v2");
>> --
>> 2.25.1
>>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-11-23 6:20 ` Tanmay Shah
@ 2021-12-06 19:04 ` Mathieu Poirier
-1 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-12-06 19:04 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
> This driver enables r5f dual core Real time Processing Unit subsystem
> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
> (cluster) can be configured in different modes e.g. split mode in which
> two r5f cores work independent of each other and lock-step mode in which
> both r5f cores execute same code clock-for-clock and notify if the
> result is different.
>
> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
>
> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
> ---
> drivers/remoteproc/Kconfig | 12 +
> drivers/remoteproc/Makefile | 1 +
> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
> 3 files changed, 972 insertions(+)
> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>
> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> index f30d00a3aabe..27f66910d8d3 100644
> --- a/drivers/remoteproc/Kconfig
> +++ b/drivers/remoteproc/Kconfig
> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
> It's safe to say N here if you're not interested in utilizing
> a slave processor.
>
> +config XLNX_R5_REMOTEPROC
> + tristate "Xilinx R5 remoteproc support"
> + depends on PM && ARCH_ZYNQMP
> + depends on ZYNQMP_FIRMWARE
> + select RPMSG_VIRTIO
> + select ZYNQMP_IPI_MBOX
> + help
> + Say y or m here to support Xilinx R5 remote processors via the remote
> + processor framework.
> +
> + It's safe to say N if not interested in using RPU r5f cores.
> +
> endif # REMOTEPROC
>
> endmenu
> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> index bb26c9e4ef9c..334a8bed4c14 100644
> --- a/drivers/remoteproc/Makefile
> +++ b/drivers/remoteproc/Makefile
> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> new file mode 100644
> index 000000000000..c2167fd3869d
> --- /dev/null
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -0,0 +1,959 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ZynqMP R5 Remote Processor driver
> + *
> + */
> +
> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
> +#include <linux/firmware/xlnx-zynqmp.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/mailbox_client.h>
> +#include <linux/mailbox/zynqmp-ipi-message.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/remoteproc.h>
> +#include <linux/skbuff.h>
> +#include <linux/sysfs.h>
> +
> +#include "remoteproc_internal.h"
> +
> +/* settings for RPU cluster mode */
> +enum zynqmp_r5_cluster_mode {
> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
> +};
> +
> +/**
> + * struct mem_bank_data - Memory Bank description
> + *
> + * @addr: Start address of memory bank
> + * @size: Size of Memory bank
> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
> + * @bank_name: name of the bank for remoteproc framework
> + */
> +struct mem_bank_data {
> + phys_addr_t addr;
> + size_t size;
> + enum pm_node_id pm_domain_id;
> + char *bank_name;
> +};
> +
> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
> +};
> +
> +/**
> + * struct zynqmp_r5_core - ZynqMP R5 core structure
> + *
> + * @dev: device of RPU instance
> + * @np: device node of RPU instance
> + * @tcm_bank_count: number TCM banks accessible to this RPU
> + * @tcm_banks: array of each TCM bank data
> + * @res_mem_count: number of Reserved Memory regions per core
> + * @res_mem: array of reserved memory regions
> + * @rproc: rproc handle
> + * @pm_domain_id: RPU CPU power domain id
> + */
> +struct zynqmp_r5_core {
> + struct device *dev;
> + struct device_node *np;
> + int tcm_bank_count;
> + struct mem_bank_data *tcm_banks;
> + int res_mem_count;
> + struct reserved_mem *res_mem;
> + struct rproc *rproc;
> + enum pm_node_id pm_domain_id;
> +};
> +
> +/**
> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
> + *
> + * @dev: r5f subsystem cluster device node
> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
> + * @core_count: number of r5 cores used for this cluster mode
> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
> + */
> +struct zynqmp_r5_cluster {
> + struct device *dev;
> + enum zynqmp_r5_cluster_mode mode;
> + int core_count;
> + struct zynqmp_r5_core *r5_cores;
> +};
> +
> +/*
> + * zynqmp_r5_set_mode - set RPU operation mode
> + *
> + * set RPU operation mode
> + *
> + * Return: 0 for success, negative value for failure
> + */
> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
> + enum zynqmp_r5_cluster_mode rpu_mode)
> +{
> + enum rpu_tcm_comb tcm_mode;
> + int ret, reg_val;
> +
> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
Here split and single-cpu mode are treated the same... Is this an omission or
really how the HW works? A good dose of documentation is definitely needed.
> +
> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
> + if (ret < 0) {
> + pr_err("failed to set RPU mode\n");
> + return ret;
> + }
> +
> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
> + if (ret < 0)
> + pr_err("failed to configure TCM\n");
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_start
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Start R5 Core from designated boot address.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + enum rpu_boot_mem bootmem;
> + int ret;
> +
> + if (!r5_core) {
> + pr_err("can't get r5 core\n");
> + return -EINVAL;
> + }
> +
> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> +
> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> +
> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> + if (ret)
> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_stop
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Power down R5 Core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + int ret;
> +
> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret)
> + pr_err("failed to stop remoteproc RPU %d\n", ret);
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + mem->va = (void *)va;
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap memory-region carveout
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + iounmap((void __iomem *)mem->va);
> + return 0;
> +}
> +
> +/*
> + * add_mem_regions
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Construct rproc mem carveouts from carveout provided in
> + * memory-region property
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_mem_regions(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct rproc_mem_entry *mem;
> + struct reserved_mem *rmem;
> + struct zynqmp_r5_core *r5_core;
> + int i;
> +
> + r5_core = rproc->priv;
> + dev = r5_core->dev;
> +
> + /* Register associated reserved memory regions */
> + for (i = 0; i < r5_core->res_mem_count; i++) {
> + rmem = &r5_core->res_mem[i];
> + mem = rproc_mem_entry_init(dev, NULL,
> + (dma_addr_t)rmem->base,
> + rmem->size, rmem->base,
> + zynqmp_r5_rproc_mem_map,
> + zynqmp_r5_rproc_mem_unmap,
> + rmem->name);
> + if (IS_ERR_OR_NULL(mem))
> + return -ENOMEM;
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap TCM banks when powering down R5 core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> + struct zynqmp_r5_core *r5_core;
> + int i;
> + enum pm_node_id pm_domain_id;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + pr_err("r5 core is not available\n");
> + return -EINVAL;
> + }
> +
> + iounmap((void __iomem *)mem->va);
> +
> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + if (zynqmp_pm_release_node(pm_domain_id))
> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * tcm_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to initialize the va and da fields of
> + *
> + * Given TCM bank entry, this callback will set device address for R5
> + * running on TCM and also setup virtual address for TCM bank
> + * remoteproc carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + /* Update memory entry va */
> + mem->va = (void *)va;
> +
> + /* clear TCMs */
> + memset_io(va, 0, mem->len);
> +
> + /*
> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> + * while on the Linux side they are at 0xffexxxxx.
> + *
> + * Zero out the high 12 bits of the address. This will give
> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> + */
> + mem->da &= 0x000fffff;
> +
> + /*
> + * TCM Banks 1A and 1B still have to be translated.
> + *
> + * Below handle these two banks' absolute addresses (0xffe90000 and
> + * 0xffeb0000) and convert to the expected relative addresses
> + * (0x0 and 0x20000).
> + */
> + if (mem->da == 0x90000 || mem->da == 0xB0000)
> + mem->da -= 0x90000;
> +
> + /* if translated TCM bank address is not valid report error */
> + if (mem->da != 0x0 && mem->da != 0x20000) {
> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
> + return -EINVAL;
> + }
> + return 0;
> +}
> +
> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
> + char *bank_name;
> + struct device *dev;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + for (i = 0; i < num_banks; i++) {
> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
> + bank_name = r5_core->tcm_banks[i].bank_name;
> + bank_size = r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> +
> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + /* Turn off all TCM banks turned on before */
> + do {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x, %x\n",
> + (u32)pm_domain_id, ret);
> + } while (i--);
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
> + char *bank_name;
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
> + bank_name = r5_core->tcm_banks[0].bank_name;
> + for (i = 0; i < num_banks; i++) {
> + bank_size += r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> + }
> +
> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + for (i = 0; i < num_banks; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x ret: %x\n",
> + (u32)pm_domain_id, ret);
> + }
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> +
> + return 0;
> +}
> +
> +/*
> + * add_tcm_banks()
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Given R5 node in remoteproc instance
> + * allocate remoteproc carveout for TCM memory
> + * needed for firmware to be loaded
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_tcm_banks(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + parent_pdev = to_platform_device(dev->parent);
> + if (!parent_pdev) {
> + dev_err(dev, "parent platform dev unavailable\n");
> + return -ENODEV;
> + }
> +
> + cluster = platform_get_drvdata(parent_pdev);
> + if (!cluster) {
> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
> + return -EINVAL;
> + }
> +
> + if (cluster->mode == SPLIT_MODE)
> + return add_tcm_carveout_split_mode(rproc);
> + else if (cluster->mode == LOCKSTEP_MODE)
> + return add_tcm_carveout_lockstep_mode(rproc);
> +
> + dev_err(cluster->dev, "invalid cluster mode\n");
> + return -EINVAL;
> +}
> +
> +/*
> + * zynqmp_r5_parse_fw()
> + * @rproc: single R5 core's corresponding rproc instance
> + * @fw: ptr to firmware to be loaded onto r5 core
> + *
> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> + int ret;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + dev_err(&rproc->dev, "r5 core not available\n");
> + return -EINVAL;
> + }
> +
> + dev = r5_core->dev;
> +
> + ret = add_tcm_banks(rproc);
> + if (ret) {
> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
> + return ret;
> + }
> +
> + ret = add_mem_regions(rproc);
> + if (ret)
> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
> +
> + ret = rproc_elf_load_rsc_table(rproc, fw);
> + if (ret == -EINVAL) {
> + /*
> + * resource table only required for IPC.
> + * if not present, this is not necessarily an error;
> + * for example, loading r5 hello world application
> + * so simply inform user and keep going.
> + */
> + dev_info(&rproc->dev, "no resource table found.\n");
> + ret = 0;
> + }
> + return ret;
> +}
> +
> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> + .start = zynqmp_r5_rproc_start,
> + .stop = zynqmp_r5_rproc_stop,
> + .load = rproc_elf_load_segments,
> + .parse_fw = zynqmp_r5_parse_fw,
> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> + .sanity_check = rproc_elf_sanity_check,
> + .get_boot_addr = rproc_elf_get_boot_addr,
> +};
> +
> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
> +{
> + int i, j, k;
> + struct zynqmp_r5_core *r5_core;
> +
> + dev_dbg(cluster->dev, "Printing dt node info\n");
> +
> + pr_debug("cluster mode = %d\n", cluster->mode);
> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
> + pr_debug("r5f num cores = %d\n", cluster->core_count);
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("can't get r5_core\n");
> + continue;
> + }
> +
> + pr_debug("r5 core %d nodes\n", i);
> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
> + k, r5_core->tcm_banks[k].addr,
> + r5_core->tcm_banks[k].size,
> + r5_core->tcm_banks[k].pm_domain_id,
> + r5_core->tcm_banks[k].bank_name);
> + }
> +
> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
> +
> + for (j = 0; j < r5_core->res_mem_count; j++) {
> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
> + j, r5_core->res_mem[j].base,
> + r5_core->res_mem[j].size,
> + r5_core->res_mem[j].name);
> + }
> + }
> +}
> +
> +/**
> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
> + * this is called for each individual R5 core to
> + * set up mailbox, Xilinx platform manager unique ID,
> + * add to rproc core
> + *
Please get rid of the indentations. That way it conforms to the rest of the
style in this file.
> + * @r5_core: zynqmp_r5_core r5 core object to initialize
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
> +{
> + int ret;
> + struct rproc *r5_rproc;
> + struct device *dev;
> +
> + dev = r5_core->dev;
> +
> + /* Set up DMA mask */
> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> + if (ret)
> + return ret;
> +
> + /* Allocate remoteproc instance */
> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> + NULL, sizeof(struct zynqmp_r5_core));
> + if (IS_ERR_OR_NULL(r5_rproc))
> + return -ENOMEM;
devm_rproc_alloc() doesn't return an error code.
> +
> + r5_rproc->auto_boot = false;
> + r5_rproc->priv = r5_core;
Here the original value of rproc->priv allocated by rproc_alloc() is replaced by
@r5_core, resulting in a memory leak.
> +
> + /* Add R5 remoteproc */
> + ret = devm_rproc_add(dev, r5_rproc);
> + if (ret) {
> + pr_err("failed to add r5 remoteproc\n");
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> +{
> + int tcm_bank_count, tcm_node;
> + int i = 0, j;
> + struct zynqmp_r5_core *r5_core;
> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> + struct device *dev = cluster->dev;
> +
> + /* ToDo: Use predefined TCM address space values from driver until
> + * system-dt spec is not final fot TCM
> + */
> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> +
> + /* count per core tcm banks */
> + tcm_bank_count = tcm_bank_count / cluster->core_count;
> +
> + /* r5 core 0 will use all of TCM banks in lockstep mode.
> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> + * 128k. Assign TCM banks to each core accordingly
> + */
> + tcm_node = 0;
> + for (j = 0; j < cluster->core_count; j++) {
> + r5_core = &cluster->r5_cores[j];
> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> + tcm_bank_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> + return -ENOMEM;
> +
> + for (i = 0; i < tcm_bank_count; i++) {
> + /* Use pre-defined TCM reg values.
> + * Eventually this should be replaced by values
> + * parsed from dts.
> + */
> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> + tcm_node++;
> + }
> +
> + r5_core->tcm_bank_count = tcm_bank_count;
> + }
Please use 'i' for the outer loop and 'j' for the inner one. Otherwise people
are looking for a subtlety that isn't here.
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> +{
> + int res_mem_count, i;
> + struct device *dev;
> + struct device_node *np, *rmem_np;
> + struct reserved_mem *rmem;
> +
> + dev = r5_core->dev;
> +
> + np = r5_core->np;
> + if (IS_ERR_OR_NULL(np)) {
> + pr_err("invalid device node of r5 core\n");
> + return -EINVAL;
> + }
In normal cases this kind of defensive programming makes sense but here
r5_core->np has just been set in zynqmp_r5_cluster_init() where there is
plenty of gards against errors. Please remove.
> +
> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> + sizeof(phandle));
> + if (res_mem_count <= 0) {
> + dev_warn(dev, "failed to get memory-region property %d\n",
> + res_mem_count);
> + return -EINVAL;
> + }
> +
> + r5_core->res_mem = devm_kzalloc(dev,
> + res_mem_count * sizeof(struct reserved_mem),
> + GFP_KERNEL);
Use devm_kcalloc() when more than one element is required. Here and throughout
the driver.
> + if (!r5_core->res_mem) {
> + dev_err(dev, "failed to allocate mem region memory\n");
> + return -ENOMEM;
> + }
> +
> + for (i = 0; i < res_mem_count; i++) {
> + rmem_np = of_parse_phandle(np, "memory-region", i);
> + if (!rmem_np)
> + return -EINVAL;
> +
> + rmem = of_reserved_mem_lookup(rmem_np);
> + if (!rmem) {
> + of_node_put(rmem_np);
> + return -EINVAL;
> + }
> +
> + memcpy(&r5_core->res_mem[i], rmem,
> + sizeof(struct reserved_mem));
Why is the whole reserved_mem is duplicated when only ->base, ->size and ->name
are used? And even those I'm not sure about really needing them...
> + of_node_put(rmem_np);
> + }
> +
> + r5_core->res_mem_count = res_mem_count;
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> +{
> + int ret, i;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev = cluster->dev;
> +
> + ret = zynqmp_r5_get_tcm_node(cluster);
> + if (ret < 0) {
> + dev_err(dev, "can't get tcm node, err %d\n", ret);
> + return ret;
> + }
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("invalid r5 core\n");
> + return -EINVAL;
> + }
Same here - please remove.
> +
> + ret = zynqmp_r5_get_mem_region_node(r5_core);
> + if (ret)
> + dev_warn(dev, "memory-region prop failed %d\n", ret);
> +
> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> + 1, &r5_core->pm_domain_id);
> + if (ret) {
> + dev_err(dev, "failed to get power-domains property\n");
> + return ret;
> + }
> +
> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_add_rproc_core(r5_core);
> + if (ret) {
> + dev_err(dev, "failed to init r5 core %d\n", i);
> + return ret;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> +{
> + struct device *dev = cluster->dev;
> + struct device_node *dev_node = dev_of_node(dev);
> + struct device_node *child;
> + struct platform_device *child_pdev;
> + int core_count = 0, ret, i;
> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> + struct zynqmp_r5_core *r5_cores;
> +
> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> +
> + /* on success returns 0, if not defined then returns -EINVAL,
> + * In that case, default is LOCKSTEP mode
> + */
> + if (ret != -EINVAL && ret != 0) {
> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> + return -EINVAL;
> + }
> +
> + if (cluster_mode == SINGLE_CPU_MODE) {
> + dev_err(dev, "driver does not support single cpu mode\n");
> + return -EINVAL;
> + } else if ((cluster_mode != SPLIT_MODE &&
> + cluster_mode != LOCKSTEP_MODE)) {
> + dev_err(dev, "Invalid cluster mode\n");
> + return -EINVAL;
> + }
> +
> + core_count = of_get_available_child_count(dev_node);
> + if (core_count <= 0) {
> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> + return -EINVAL;
> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> + return -EINVAL;
> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> + dev_warn(dev, "Only r5 core0 will be used\n");
> + core_count = 1;
> + }
> +
> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> + core_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_cores)) {
> + dev_err(dev, "can't allocate memory for cores\n");
> + return -ENOMEM;
> + }
> +
> + i = 0;
> + for_each_available_child_of_node(dev_node, child) {
> + child_pdev = of_find_device_by_node(child);
> + if (!child_pdev)
> + return -ENODEV;
> +
> + r5_cores[i].dev = &child_pdev->dev;
> + if (!r5_cores[i].dev) {
> + pr_err("can't get device for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> + if (!r5_cores[i].np) {
> + pr_err("can't get device node for r5 core %d\n", i);
> + return -ENODEV;
> + }
The error path for both if() conditions above needs to release the
child_pdev->dev, as instructed by the documentation for
of_find_device_by_node(). The same needs to be done when there is no errors.
More comments to come tomorrow.
Thanks,
Mathieu
> +
> + i++;
> + if (i == core_count)
> + break;
> + }
> +
> + cluster->mode = cluster_mode;
> + cluster->core_count = core_count;
> + cluster->r5_cores = r5_cores;
> +
> + ret = zynqmp_r5_core_init(cluster);
> + if (ret < 0) {
> + dev_err(dev, "failed to init r5 core err %d\n", ret);
> + return ret;
> + }
> +
> + zynqmp_r5_print_dt_node_info(cluster);
> +
> + return 0;
> +}
> +
> +static void zynqmp_r5_cluster_exit(void *data)
> +{
> + struct platform_device *pdev = (struct platform_device *)data;
> +
> + platform_set_drvdata(pdev, NULL);
> +
> + pr_info("Exit r5f subsystem driver\n");
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_probe()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * called when driver is probed, for each R5 core specified in DT,
> + * setup as needed to do remoteproc-related operations
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> +{
> + int ret;
> + struct zynqmp_r5_cluster *cluster;
> + struct device *dev = &pdev->dev;
> +
> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> + if (IS_ERR_OR_NULL(cluster))
> + return -ENOMEM;
> +
> + cluster->dev = dev;
> +
> + ret = devm_of_platform_populate(dev);
> + if (ret) {
> + dev_err(dev, "failed to populate platform dev %d\n", ret);
> + return ret;
> + }
> +
> + /* wire in so each core can be cleaned up at driver remove */
> + platform_set_drvdata(pdev, cluster);
> +
> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_cluster_init(cluster);
> + if (ret) {
> + dev_err(dev, "Invalid r5f subsystem device tree\n");
> + return ret;
> + }
> +
> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> + return 0;
> +}
> +
> +/* Match table for OF platform binding */
> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> + { .compatible = "xlnx,zynqmp-r5fss", },
> + { /* end of list */ },
> +};
> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> +
> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> + .probe = zynqmp_r5_remoteproc_probe,
> + .driver = {
> + .name = "zynqmp_r5_remoteproc",
> + .of_match_table = zynqmp_r5_remoteproc_match,
> + },
> +};
> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> +
> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> +MODULE_AUTHOR("Xilinx Inc.");
> +MODULE_LICENSE("GPL v2");
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-06 19:04 ` Mathieu Poirier
0 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-12-06 19:04 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
> This driver enables r5f dual core Real time Processing Unit subsystem
> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
> (cluster) can be configured in different modes e.g. split mode in which
> two r5f cores work independent of each other and lock-step mode in which
> both r5f cores execute same code clock-for-clock and notify if the
> result is different.
>
> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
>
> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
> ---
> drivers/remoteproc/Kconfig | 12 +
> drivers/remoteproc/Makefile | 1 +
> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
> 3 files changed, 972 insertions(+)
> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>
> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> index f30d00a3aabe..27f66910d8d3 100644
> --- a/drivers/remoteproc/Kconfig
> +++ b/drivers/remoteproc/Kconfig
> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
> It's safe to say N here if you're not interested in utilizing
> a slave processor.
>
> +config XLNX_R5_REMOTEPROC
> + tristate "Xilinx R5 remoteproc support"
> + depends on PM && ARCH_ZYNQMP
> + depends on ZYNQMP_FIRMWARE
> + select RPMSG_VIRTIO
> + select ZYNQMP_IPI_MBOX
> + help
> + Say y or m here to support Xilinx R5 remote processors via the remote
> + processor framework.
> +
> + It's safe to say N if not interested in using RPU r5f cores.
> +
> endif # REMOTEPROC
>
> endmenu
> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> index bb26c9e4ef9c..334a8bed4c14 100644
> --- a/drivers/remoteproc/Makefile
> +++ b/drivers/remoteproc/Makefile
> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> new file mode 100644
> index 000000000000..c2167fd3869d
> --- /dev/null
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -0,0 +1,959 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ZynqMP R5 Remote Processor driver
> + *
> + */
> +
> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
> +#include <linux/firmware/xlnx-zynqmp.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/mailbox_client.h>
> +#include <linux/mailbox/zynqmp-ipi-message.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/remoteproc.h>
> +#include <linux/skbuff.h>
> +#include <linux/sysfs.h>
> +
> +#include "remoteproc_internal.h"
> +
> +/* settings for RPU cluster mode */
> +enum zynqmp_r5_cluster_mode {
> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
> +};
> +
> +/**
> + * struct mem_bank_data - Memory Bank description
> + *
> + * @addr: Start address of memory bank
> + * @size: Size of Memory bank
> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
> + * @bank_name: name of the bank for remoteproc framework
> + */
> +struct mem_bank_data {
> + phys_addr_t addr;
> + size_t size;
> + enum pm_node_id pm_domain_id;
> + char *bank_name;
> +};
> +
> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
> +};
> +
> +/**
> + * struct zynqmp_r5_core - ZynqMP R5 core structure
> + *
> + * @dev: device of RPU instance
> + * @np: device node of RPU instance
> + * @tcm_bank_count: number TCM banks accessible to this RPU
> + * @tcm_banks: array of each TCM bank data
> + * @res_mem_count: number of Reserved Memory regions per core
> + * @res_mem: array of reserved memory regions
> + * @rproc: rproc handle
> + * @pm_domain_id: RPU CPU power domain id
> + */
> +struct zynqmp_r5_core {
> + struct device *dev;
> + struct device_node *np;
> + int tcm_bank_count;
> + struct mem_bank_data *tcm_banks;
> + int res_mem_count;
> + struct reserved_mem *res_mem;
> + struct rproc *rproc;
> + enum pm_node_id pm_domain_id;
> +};
> +
> +/**
> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
> + *
> + * @dev: r5f subsystem cluster device node
> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
> + * @core_count: number of r5 cores used for this cluster mode
> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
> + */
> +struct zynqmp_r5_cluster {
> + struct device *dev;
> + enum zynqmp_r5_cluster_mode mode;
> + int core_count;
> + struct zynqmp_r5_core *r5_cores;
> +};
> +
> +/*
> + * zynqmp_r5_set_mode - set RPU operation mode
> + *
> + * set RPU operation mode
> + *
> + * Return: 0 for success, negative value for failure
> + */
> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
> + enum zynqmp_r5_cluster_mode rpu_mode)
> +{
> + enum rpu_tcm_comb tcm_mode;
> + int ret, reg_val;
> +
> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
Here split and single-cpu mode are treated the same... Is this an omission or
really how the HW works? A good dose of documentation is definitely needed.
> +
> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
> + if (ret < 0) {
> + pr_err("failed to set RPU mode\n");
> + return ret;
> + }
> +
> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
> + if (ret < 0)
> + pr_err("failed to configure TCM\n");
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_start
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Start R5 Core from designated boot address.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + enum rpu_boot_mem bootmem;
> + int ret;
> +
> + if (!r5_core) {
> + pr_err("can't get r5 core\n");
> + return -EINVAL;
> + }
> +
> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> +
> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> +
> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> + if (ret)
> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_stop
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Power down R5 Core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + int ret;
> +
> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret)
> + pr_err("failed to stop remoteproc RPU %d\n", ret);
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + mem->va = (void *)va;
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap memory-region carveout
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + iounmap((void __iomem *)mem->va);
> + return 0;
> +}
> +
> +/*
> + * add_mem_regions
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Construct rproc mem carveouts from carveout provided in
> + * memory-region property
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_mem_regions(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct rproc_mem_entry *mem;
> + struct reserved_mem *rmem;
> + struct zynqmp_r5_core *r5_core;
> + int i;
> +
> + r5_core = rproc->priv;
> + dev = r5_core->dev;
> +
> + /* Register associated reserved memory regions */
> + for (i = 0; i < r5_core->res_mem_count; i++) {
> + rmem = &r5_core->res_mem[i];
> + mem = rproc_mem_entry_init(dev, NULL,
> + (dma_addr_t)rmem->base,
> + rmem->size, rmem->base,
> + zynqmp_r5_rproc_mem_map,
> + zynqmp_r5_rproc_mem_unmap,
> + rmem->name);
> + if (IS_ERR_OR_NULL(mem))
> + return -ENOMEM;
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap TCM banks when powering down R5 core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> + struct zynqmp_r5_core *r5_core;
> + int i;
> + enum pm_node_id pm_domain_id;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + pr_err("r5 core is not available\n");
> + return -EINVAL;
> + }
> +
> + iounmap((void __iomem *)mem->va);
> +
> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + if (zynqmp_pm_release_node(pm_domain_id))
> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * tcm_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to initialize the va and da fields of
> + *
> + * Given TCM bank entry, this callback will set device address for R5
> + * running on TCM and also setup virtual address for TCM bank
> + * remoteproc carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + /* Update memory entry va */
> + mem->va = (void *)va;
> +
> + /* clear TCMs */
> + memset_io(va, 0, mem->len);
> +
> + /*
> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> + * while on the Linux side they are at 0xffexxxxx.
> + *
> + * Zero out the high 12 bits of the address. This will give
> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> + */
> + mem->da &= 0x000fffff;
> +
> + /*
> + * TCM Banks 1A and 1B still have to be translated.
> + *
> + * Below handle these two banks' absolute addresses (0xffe90000 and
> + * 0xffeb0000) and convert to the expected relative addresses
> + * (0x0 and 0x20000).
> + */
> + if (mem->da == 0x90000 || mem->da == 0xB0000)
> + mem->da -= 0x90000;
> +
> + /* if translated TCM bank address is not valid report error */
> + if (mem->da != 0x0 && mem->da != 0x20000) {
> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
> + return -EINVAL;
> + }
> + return 0;
> +}
> +
> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
> + char *bank_name;
> + struct device *dev;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + for (i = 0; i < num_banks; i++) {
> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
> + bank_name = r5_core->tcm_banks[i].bank_name;
> + bank_size = r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> +
> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + /* Turn off all TCM banks turned on before */
> + do {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x, %x\n",
> + (u32)pm_domain_id, ret);
> + } while (i--);
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
> + char *bank_name;
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
> + bank_name = r5_core->tcm_banks[0].bank_name;
> + for (i = 0; i < num_banks; i++) {
> + bank_size += r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> + }
> +
> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + for (i = 0; i < num_banks; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x ret: %x\n",
> + (u32)pm_domain_id, ret);
> + }
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> +
> + return 0;
> +}
> +
> +/*
> + * add_tcm_banks()
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Given R5 node in remoteproc instance
> + * allocate remoteproc carveout for TCM memory
> + * needed for firmware to be loaded
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_tcm_banks(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + parent_pdev = to_platform_device(dev->parent);
> + if (!parent_pdev) {
> + dev_err(dev, "parent platform dev unavailable\n");
> + return -ENODEV;
> + }
> +
> + cluster = platform_get_drvdata(parent_pdev);
> + if (!cluster) {
> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
> + return -EINVAL;
> + }
> +
> + if (cluster->mode == SPLIT_MODE)
> + return add_tcm_carveout_split_mode(rproc);
> + else if (cluster->mode == LOCKSTEP_MODE)
> + return add_tcm_carveout_lockstep_mode(rproc);
> +
> + dev_err(cluster->dev, "invalid cluster mode\n");
> + return -EINVAL;
> +}
> +
> +/*
> + * zynqmp_r5_parse_fw()
> + * @rproc: single R5 core's corresponding rproc instance
> + * @fw: ptr to firmware to be loaded onto r5 core
> + *
> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> + int ret;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + dev_err(&rproc->dev, "r5 core not available\n");
> + return -EINVAL;
> + }
> +
> + dev = r5_core->dev;
> +
> + ret = add_tcm_banks(rproc);
> + if (ret) {
> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
> + return ret;
> + }
> +
> + ret = add_mem_regions(rproc);
> + if (ret)
> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
> +
> + ret = rproc_elf_load_rsc_table(rproc, fw);
> + if (ret == -EINVAL) {
> + /*
> + * resource table only required for IPC.
> + * if not present, this is not necessarily an error;
> + * for example, loading r5 hello world application
> + * so simply inform user and keep going.
> + */
> + dev_info(&rproc->dev, "no resource table found.\n");
> + ret = 0;
> + }
> + return ret;
> +}
> +
> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> + .start = zynqmp_r5_rproc_start,
> + .stop = zynqmp_r5_rproc_stop,
> + .load = rproc_elf_load_segments,
> + .parse_fw = zynqmp_r5_parse_fw,
> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> + .sanity_check = rproc_elf_sanity_check,
> + .get_boot_addr = rproc_elf_get_boot_addr,
> +};
> +
> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
> +{
> + int i, j, k;
> + struct zynqmp_r5_core *r5_core;
> +
> + dev_dbg(cluster->dev, "Printing dt node info\n");
> +
> + pr_debug("cluster mode = %d\n", cluster->mode);
> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
> + pr_debug("r5f num cores = %d\n", cluster->core_count);
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("can't get r5_core\n");
> + continue;
> + }
> +
> + pr_debug("r5 core %d nodes\n", i);
> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
> + k, r5_core->tcm_banks[k].addr,
> + r5_core->tcm_banks[k].size,
> + r5_core->tcm_banks[k].pm_domain_id,
> + r5_core->tcm_banks[k].bank_name);
> + }
> +
> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
> +
> + for (j = 0; j < r5_core->res_mem_count; j++) {
> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
> + j, r5_core->res_mem[j].base,
> + r5_core->res_mem[j].size,
> + r5_core->res_mem[j].name);
> + }
> + }
> +}
> +
> +/**
> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
> + * this is called for each individual R5 core to
> + * set up mailbox, Xilinx platform manager unique ID,
> + * add to rproc core
> + *
Please get rid of the indentations. That way it conforms to the rest of the
style in this file.
> + * @r5_core: zynqmp_r5_core r5 core object to initialize
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
> +{
> + int ret;
> + struct rproc *r5_rproc;
> + struct device *dev;
> +
> + dev = r5_core->dev;
> +
> + /* Set up DMA mask */
> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> + if (ret)
> + return ret;
> +
> + /* Allocate remoteproc instance */
> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> + NULL, sizeof(struct zynqmp_r5_core));
> + if (IS_ERR_OR_NULL(r5_rproc))
> + return -ENOMEM;
devm_rproc_alloc() doesn't return an error code.
> +
> + r5_rproc->auto_boot = false;
> + r5_rproc->priv = r5_core;
Here the original value of rproc->priv allocated by rproc_alloc() is replaced by
@r5_core, resulting in a memory leak.
> +
> + /* Add R5 remoteproc */
> + ret = devm_rproc_add(dev, r5_rproc);
> + if (ret) {
> + pr_err("failed to add r5 remoteproc\n");
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> +{
> + int tcm_bank_count, tcm_node;
> + int i = 0, j;
> + struct zynqmp_r5_core *r5_core;
> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> + struct device *dev = cluster->dev;
> +
> + /* ToDo: Use predefined TCM address space values from driver until
> + * system-dt spec is not final fot TCM
> + */
> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> +
> + /* count per core tcm banks */
> + tcm_bank_count = tcm_bank_count / cluster->core_count;
> +
> + /* r5 core 0 will use all of TCM banks in lockstep mode.
> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> + * 128k. Assign TCM banks to each core accordingly
> + */
> + tcm_node = 0;
> + for (j = 0; j < cluster->core_count; j++) {
> + r5_core = &cluster->r5_cores[j];
> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> + tcm_bank_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> + return -ENOMEM;
> +
> + for (i = 0; i < tcm_bank_count; i++) {
> + /* Use pre-defined TCM reg values.
> + * Eventually this should be replaced by values
> + * parsed from dts.
> + */
> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> + tcm_node++;
> + }
> +
> + r5_core->tcm_bank_count = tcm_bank_count;
> + }
Please use 'i' for the outer loop and 'j' for the inner one. Otherwise people
are looking for a subtlety that isn't here.
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> +{
> + int res_mem_count, i;
> + struct device *dev;
> + struct device_node *np, *rmem_np;
> + struct reserved_mem *rmem;
> +
> + dev = r5_core->dev;
> +
> + np = r5_core->np;
> + if (IS_ERR_OR_NULL(np)) {
> + pr_err("invalid device node of r5 core\n");
> + return -EINVAL;
> + }
In normal cases this kind of defensive programming makes sense but here
r5_core->np has just been set in zynqmp_r5_cluster_init() where there is
plenty of gards against errors. Please remove.
> +
> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> + sizeof(phandle));
> + if (res_mem_count <= 0) {
> + dev_warn(dev, "failed to get memory-region property %d\n",
> + res_mem_count);
> + return -EINVAL;
> + }
> +
> + r5_core->res_mem = devm_kzalloc(dev,
> + res_mem_count * sizeof(struct reserved_mem),
> + GFP_KERNEL);
Use devm_kcalloc() when more than one element is required. Here and throughout
the driver.
> + if (!r5_core->res_mem) {
> + dev_err(dev, "failed to allocate mem region memory\n");
> + return -ENOMEM;
> + }
> +
> + for (i = 0; i < res_mem_count; i++) {
> + rmem_np = of_parse_phandle(np, "memory-region", i);
> + if (!rmem_np)
> + return -EINVAL;
> +
> + rmem = of_reserved_mem_lookup(rmem_np);
> + if (!rmem) {
> + of_node_put(rmem_np);
> + return -EINVAL;
> + }
> +
> + memcpy(&r5_core->res_mem[i], rmem,
> + sizeof(struct reserved_mem));
Why is the whole reserved_mem is duplicated when only ->base, ->size and ->name
are used? And even those I'm not sure about really needing them...
> + of_node_put(rmem_np);
> + }
> +
> + r5_core->res_mem_count = res_mem_count;
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> +{
> + int ret, i;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev = cluster->dev;
> +
> + ret = zynqmp_r5_get_tcm_node(cluster);
> + if (ret < 0) {
> + dev_err(dev, "can't get tcm node, err %d\n", ret);
> + return ret;
> + }
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("invalid r5 core\n");
> + return -EINVAL;
> + }
Same here - please remove.
> +
> + ret = zynqmp_r5_get_mem_region_node(r5_core);
> + if (ret)
> + dev_warn(dev, "memory-region prop failed %d\n", ret);
> +
> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> + 1, &r5_core->pm_domain_id);
> + if (ret) {
> + dev_err(dev, "failed to get power-domains property\n");
> + return ret;
> + }
> +
> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_add_rproc_core(r5_core);
> + if (ret) {
> + dev_err(dev, "failed to init r5 core %d\n", i);
> + return ret;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> +{
> + struct device *dev = cluster->dev;
> + struct device_node *dev_node = dev_of_node(dev);
> + struct device_node *child;
> + struct platform_device *child_pdev;
> + int core_count = 0, ret, i;
> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> + struct zynqmp_r5_core *r5_cores;
> +
> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> +
> + /* on success returns 0, if not defined then returns -EINVAL,
> + * In that case, default is LOCKSTEP mode
> + */
> + if (ret != -EINVAL && ret != 0) {
> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> + return -EINVAL;
> + }
> +
> + if (cluster_mode == SINGLE_CPU_MODE) {
> + dev_err(dev, "driver does not support single cpu mode\n");
> + return -EINVAL;
> + } else if ((cluster_mode != SPLIT_MODE &&
> + cluster_mode != LOCKSTEP_MODE)) {
> + dev_err(dev, "Invalid cluster mode\n");
> + return -EINVAL;
> + }
> +
> + core_count = of_get_available_child_count(dev_node);
> + if (core_count <= 0) {
> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> + return -EINVAL;
> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> + return -EINVAL;
> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> + dev_warn(dev, "Only r5 core0 will be used\n");
> + core_count = 1;
> + }
> +
> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> + core_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_cores)) {
> + dev_err(dev, "can't allocate memory for cores\n");
> + return -ENOMEM;
> + }
> +
> + i = 0;
> + for_each_available_child_of_node(dev_node, child) {
> + child_pdev = of_find_device_by_node(child);
> + if (!child_pdev)
> + return -ENODEV;
> +
> + r5_cores[i].dev = &child_pdev->dev;
> + if (!r5_cores[i].dev) {
> + pr_err("can't get device for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> + if (!r5_cores[i].np) {
> + pr_err("can't get device node for r5 core %d\n", i);
> + return -ENODEV;
> + }
The error path for both if() conditions above needs to release the
child_pdev->dev, as instructed by the documentation for
of_find_device_by_node(). The same needs to be done when there is no errors.
More comments to come tomorrow.
Thanks,
Mathieu
> +
> + i++;
> + if (i == core_count)
> + break;
> + }
> +
> + cluster->mode = cluster_mode;
> + cluster->core_count = core_count;
> + cluster->r5_cores = r5_cores;
> +
> + ret = zynqmp_r5_core_init(cluster);
> + if (ret < 0) {
> + dev_err(dev, "failed to init r5 core err %d\n", ret);
> + return ret;
> + }
> +
> + zynqmp_r5_print_dt_node_info(cluster);
> +
> + return 0;
> +}
> +
> +static void zynqmp_r5_cluster_exit(void *data)
> +{
> + struct platform_device *pdev = (struct platform_device *)data;
> +
> + platform_set_drvdata(pdev, NULL);
> +
> + pr_info("Exit r5f subsystem driver\n");
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_probe()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * called when driver is probed, for each R5 core specified in DT,
> + * setup as needed to do remoteproc-related operations
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> +{
> + int ret;
> + struct zynqmp_r5_cluster *cluster;
> + struct device *dev = &pdev->dev;
> +
> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> + if (IS_ERR_OR_NULL(cluster))
> + return -ENOMEM;
> +
> + cluster->dev = dev;
> +
> + ret = devm_of_platform_populate(dev);
> + if (ret) {
> + dev_err(dev, "failed to populate platform dev %d\n", ret);
> + return ret;
> + }
> +
> + /* wire in so each core can be cleaned up at driver remove */
> + platform_set_drvdata(pdev, cluster);
> +
> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_cluster_init(cluster);
> + if (ret) {
> + dev_err(dev, "Invalid r5f subsystem device tree\n");
> + return ret;
> + }
> +
> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> + return 0;
> +}
> +
> +/* Match table for OF platform binding */
> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> + { .compatible = "xlnx,zynqmp-r5fss", },
> + { /* end of list */ },
> +};
> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> +
> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> + .probe = zynqmp_r5_remoteproc_probe,
> + .driver = {
> + .name = "zynqmp_r5_remoteproc",
> + .of_match_table = zynqmp_r5_remoteproc_match,
> + },
> +};
> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> +
> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> +MODULE_AUTHOR("Xilinx Inc.");
> +MODULE_LICENSE("GPL v2");
> --
> 2.25.1
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-11-23 6:20 ` Tanmay Shah
@ 2021-12-07 18:31 ` Mathieu Poirier
-1 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-12-07 18:31 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
> This driver enables r5f dual core Real time Processing Unit subsystem
> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
> (cluster) can be configured in different modes e.g. split mode in which
> two r5f cores work independent of each other and lock-step mode in which
> both r5f cores execute same code clock-for-clock and notify if the
> result is different.
>
> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
>
> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
> ---
> drivers/remoteproc/Kconfig | 12 +
> drivers/remoteproc/Makefile | 1 +
> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
> 3 files changed, 972 insertions(+)
> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>
> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> index f30d00a3aabe..27f66910d8d3 100644
> --- a/drivers/remoteproc/Kconfig
> +++ b/drivers/remoteproc/Kconfig
> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
> It's safe to say N here if you're not interested in utilizing
> a slave processor.
>
> +config XLNX_R5_REMOTEPROC
> + tristate "Xilinx R5 remoteproc support"
> + depends on PM && ARCH_ZYNQMP
> + depends on ZYNQMP_FIRMWARE
> + select RPMSG_VIRTIO
> + select ZYNQMP_IPI_MBOX
> + help
> + Say y or m here to support Xilinx R5 remote processors via the remote
> + processor framework.
> +
> + It's safe to say N if not interested in using RPU r5f cores.
> +
> endif # REMOTEPROC
>
> endmenu
> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> index bb26c9e4ef9c..334a8bed4c14 100644
> --- a/drivers/remoteproc/Makefile
> +++ b/drivers/remoteproc/Makefile
> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> new file mode 100644
> index 000000000000..c2167fd3869d
> --- /dev/null
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -0,0 +1,959 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ZynqMP R5 Remote Processor driver
> + *
> + */
> +
> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
> +#include <linux/firmware/xlnx-zynqmp.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/mailbox_client.h>
> +#include <linux/mailbox/zynqmp-ipi-message.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/remoteproc.h>
> +#include <linux/skbuff.h>
> +#include <linux/sysfs.h>
> +
> +#include "remoteproc_internal.h"
> +
> +/* settings for RPU cluster mode */
> +enum zynqmp_r5_cluster_mode {
> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
> +};
> +
> +/**
> + * struct mem_bank_data - Memory Bank description
> + *
> + * @addr: Start address of memory bank
> + * @size: Size of Memory bank
> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
> + * @bank_name: name of the bank for remoteproc framework
> + */
> +struct mem_bank_data {
> + phys_addr_t addr;
> + size_t size;
> + enum pm_node_id pm_domain_id;
> + char *bank_name;
> +};
> +
> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
> +};
> +
> +/**
> + * struct zynqmp_r5_core - ZynqMP R5 core structure
> + *
> + * @dev: device of RPU instance
> + * @np: device node of RPU instance
> + * @tcm_bank_count: number TCM banks accessible to this RPU
> + * @tcm_banks: array of each TCM bank data
> + * @res_mem_count: number of Reserved Memory regions per core
> + * @res_mem: array of reserved memory regions
> + * @rproc: rproc handle
> + * @pm_domain_id: RPU CPU power domain id
> + */
> +struct zynqmp_r5_core {
> + struct device *dev;
> + struct device_node *np;
> + int tcm_bank_count;
> + struct mem_bank_data *tcm_banks;
> + int res_mem_count;
> + struct reserved_mem *res_mem;
> + struct rproc *rproc;
> + enum pm_node_id pm_domain_id;
> +};
> +
> +/**
> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
> + *
> + * @dev: r5f subsystem cluster device node
> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
> + * @core_count: number of r5 cores used for this cluster mode
> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
> + */
> +struct zynqmp_r5_cluster {
> + struct device *dev;
> + enum zynqmp_r5_cluster_mode mode;
> + int core_count;
> + struct zynqmp_r5_core *r5_cores;
> +};
> +
> +/*
> + * zynqmp_r5_set_mode - set RPU operation mode
> + *
> + * set RPU operation mode
> + *
> + * Return: 0 for success, negative value for failure
> + */
> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
> + enum zynqmp_r5_cluster_mode rpu_mode)
> +{
> + enum rpu_tcm_comb tcm_mode;
> + int ret, reg_val;
> +
> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
> +
> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
> + if (ret < 0) {
> + pr_err("failed to set RPU mode\n");
> + return ret;
> + }
> +
> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
> + if (ret < 0)
> + pr_err("failed to configure TCM\n");
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_start
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Start R5 Core from designated boot address.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + enum rpu_boot_mem bootmem;
> + int ret;
> +
> + if (!r5_core) {
> + pr_err("can't get r5 core\n");
> + return -EINVAL;
> + }
Please remove.
> +
> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
Some comments as to what is happening here would be appreciated.
> +
> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> +
> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> + if (ret)
> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_stop
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Power down R5 Core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + int ret;
> +
> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret)
> + pr_err("failed to stop remoteproc RPU %d\n", ret);
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + mem->va = (void *)va;
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap memory-region carveout
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + iounmap((void __iomem *)mem->va);
> + return 0;
> +}
> +
> +/*
> + * add_mem_regions
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Construct rproc mem carveouts from carveout provided in
> + * memory-region property
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_mem_regions(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct rproc_mem_entry *mem;
> + struct reserved_mem *rmem;
> + struct zynqmp_r5_core *r5_core;
> + int i;
> +
> + r5_core = rproc->priv;
> + dev = r5_core->dev;
> +
> + /* Register associated reserved memory regions */
> + for (i = 0; i < r5_core->res_mem_count; i++) {
> + rmem = &r5_core->res_mem[i];
Indentation problem
> + mem = rproc_mem_entry_init(dev, NULL,
> + (dma_addr_t)rmem->base,
> + rmem->size, rmem->base,
> + zynqmp_r5_rproc_mem_map,
> + zynqmp_r5_rproc_mem_unmap,
> + rmem->name);
> + if (IS_ERR_OR_NULL(mem))
> + return -ENOMEM;
> +
> + rproc_add_carveout(rproc, mem);
> + }
Not sure why this wasn't done in zynqmp_r5_get_mem_region_node()... That would
prevent the need to keep a copy of the reserved memories in the zynqmp_r5_core
structure.
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap TCM banks when powering down R5 core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> + struct zynqmp_r5_core *r5_core;
> + int i;
> + enum pm_node_id pm_domain_id;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + pr_err("r5 core is not available\n");
> + return -EINVAL;
> + }
> +
> + iounmap((void __iomem *)mem->va);
> +
> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + if (zynqmp_pm_release_node(pm_domain_id))
> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * tcm_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to initialize the va and da fields of
> + *
> + * Given TCM bank entry, this callback will set device address for R5
> + * running on TCM and also setup virtual address for TCM bank
> + * remoteproc carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
Indentation problem
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + /* Update memory entry va */
> + mem->va = (void *)va;
> +
> + /* clear TCMs */
> + memset_io(va, 0, mem->len);
> +
> + /*
> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> + * while on the Linux side they are at 0xffexxxxx.
> + *
> + * Zero out the high 12 bits of the address. This will give
> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> + */
> + mem->da &= 0x000fffff;
> +
> + /*
> + * TCM Banks 1A and 1B still have to be translated.
> + *
> + * Below handle these two banks' absolute addresses (0xffe90000 and
> + * 0xffeb0000) and convert to the expected relative addresses
> + * (0x0 and 0x20000).
> + */
> + if (mem->da == 0x90000 || mem->da == 0xB0000)
> + mem->da -= 0x90000;
> +
> + /* if translated TCM bank address is not valid report error */
> + if (mem->da != 0x0 && mem->da != 0x20000) {
> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
> + return -EINVAL;
> + }
> + return 0;
> +}
> +
> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
phys_addr_t bank_addr;
> + size_t bank_size = 0;
No initialisation needed.
> + char *bank_name;
> + struct device *dev;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
Remove.
> +
> + dev = r5_core->dev;
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + for (i = 0; i < num_banks; i++) {
> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
> + bank_name = r5_core->tcm_banks[i].bank_name;
> + bank_size = r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> +
> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
I'm pretty sure that if you run the sparse checker on this you will get a lot of
angry warnings.
> + if (IS_ERR_OR_NULL(mem)) {
Function rproc_mem_entry_init() does not return an error code.
> + /* Turn off all TCM banks turned on before */
> + do {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
Here @pm_domain_id is casted as a u32 but it isn't above when calling
zynqmp_pm_request_node(), and that despite both functions taking a "const u32
node". Again, I'm sure the sparse checker will complain here.
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x, %x\n",
> + (u32)pm_domain_id, ret);
> + } while (i--);
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
Same as above.
> + char *bank_name;
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
Same as above.
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
> + bank_name = r5_core->tcm_banks[0].bank_name;
> + for (i = 0; i < num_banks; i++) {
> + bank_size += r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> + }
I know what is happening above but in 6 months it will all be gone from
memory. Please add enough documentation for someone not familiar with the
platform to understand why things are done that way. Here and throughout the
driver.
> +
> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + for (i = 0; i < num_banks; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x ret: %x\n",
> + (u32)pm_domain_id, ret);
> + }
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> +
> + return 0;
> +}
> +
> +/*
> + * add_tcm_banks()
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Given R5 node in remoteproc instance
> + * allocate remoteproc carveout for TCM memory
> + * needed for firmware to be loaded
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_tcm_banks(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
Remove.
> +
> + parent_pdev = to_platform_device(dev->parent);
> + if (!parent_pdev) {
> + dev_err(dev, "parent platform dev unavailable\n");
> + return -ENODEV;
> + }
> +
> + cluster = platform_get_drvdata(parent_pdev);
> + if (!cluster) {
> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
> + return -EINVAL;
> + }
Why not simply calling dev_get_drvdata(dev->parent) instead of going through the
platform API?
> +
> + if (cluster->mode == SPLIT_MODE)
> + return add_tcm_carveout_split_mode(rproc);
> + else if (cluster->mode == LOCKSTEP_MODE)
> + return add_tcm_carveout_lockstep_mode(rproc);
> +
> + dev_err(cluster->dev, "invalid cluster mode\n");
> + return -EINVAL;
> +}
> +
> +/*
> + * zynqmp_r5_parse_fw()
> + * @rproc: single R5 core's corresponding rproc instance
> + * @fw: ptr to firmware to be loaded onto r5 core
> + *
> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> + int ret;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + dev_err(&rproc->dev, "r5 core not available\n");
> + return -EINVAL;
> + }
Same as above and throughout.
I am done reviewing this set - the overall structure is sound but there is a
fair amount of work to be done. There is no need to rush another revision as I
won't have time to review it before the holidays. On that front, change
_only_ the things I have commented on. We can address other issues as we move
forward. Last but not least, please run the sparse checker on your work.
Thanks,
Mathieu
> +
> + dev = r5_core->dev;
> +
> + ret = add_tcm_banks(rproc);
> + if (ret) {
> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
> + return ret;
> + }
> +
> + ret = add_mem_regions(rproc);
> + if (ret)
> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
> +
> + ret = rproc_elf_load_rsc_table(rproc, fw);
> + if (ret == -EINVAL) {
> + /*
> + * resource table only required for IPC.
> + * if not present, this is not necessarily an error;
> + * for example, loading r5 hello world application
> + * so simply inform user and keep going.
> + */
> + dev_info(&rproc->dev, "no resource table found.\n");
> + ret = 0;
> + }
> + return ret;
> +}
> +
> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> + .start = zynqmp_r5_rproc_start,
> + .stop = zynqmp_r5_rproc_stop,
> + .load = rproc_elf_load_segments,
> + .parse_fw = zynqmp_r5_parse_fw,
> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> + .sanity_check = rproc_elf_sanity_check,
> + .get_boot_addr = rproc_elf_get_boot_addr,
> +};
> +
> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
> +{
> + int i, j, k;
> + struct zynqmp_r5_core *r5_core;
> +
> + dev_dbg(cluster->dev, "Printing dt node info\n");
> +
> + pr_debug("cluster mode = %d\n", cluster->mode);
> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
> + pr_debug("r5f num cores = %d\n", cluster->core_count);
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("can't get r5_core\n");
> + continue;
> + }
> +
> + pr_debug("r5 core %d nodes\n", i);
> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
> + k, r5_core->tcm_banks[k].addr,
> + r5_core->tcm_banks[k].size,
> + r5_core->tcm_banks[k].pm_domain_id,
> + r5_core->tcm_banks[k].bank_name);
> + }
> +
> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
> +
> + for (j = 0; j < r5_core->res_mem_count; j++) {
> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
> + j, r5_core->res_mem[j].base,
> + r5_core->res_mem[j].size,
> + r5_core->res_mem[j].name);
> + }
> + }
> +}
> +
> +/**
> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
> + * this is called for each individual R5 core to
> + * set up mailbox, Xilinx platform manager unique ID,
> + * add to rproc core
> + *
> + * @r5_core: zynqmp_r5_core r5 core object to initialize
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
> +{
> + int ret;
> + struct rproc *r5_rproc;
> + struct device *dev;
> +
> + dev = r5_core->dev;
> +
> + /* Set up DMA mask */
> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> + if (ret)
> + return ret;
> +
> + /* Allocate remoteproc instance */
> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> + NULL, sizeof(struct zynqmp_r5_core));
> + if (IS_ERR_OR_NULL(r5_rproc))
> + return -ENOMEM;
> +
> + r5_rproc->auto_boot = false;
> + r5_rproc->priv = r5_core;
> +
> + /* Add R5 remoteproc */
> + ret = devm_rproc_add(dev, r5_rproc);
> + if (ret) {
> + pr_err("failed to add r5 remoteproc\n");
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> +{
> + int tcm_bank_count, tcm_node;
> + int i = 0, j;
> + struct zynqmp_r5_core *r5_core;
> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> + struct device *dev = cluster->dev;
> +
> + /* ToDo: Use predefined TCM address space values from driver until
> + * system-dt spec is not final fot TCM
> + */
> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> +
> + /* count per core tcm banks */
> + tcm_bank_count = tcm_bank_count / cluster->core_count;
> +
> + /* r5 core 0 will use all of TCM banks in lockstep mode.
> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> + * 128k. Assign TCM banks to each core accordingly
> + */
> + tcm_node = 0;
> + for (j = 0; j < cluster->core_count; j++) {
> + r5_core = &cluster->r5_cores[j];
> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> + tcm_bank_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> + return -ENOMEM;
> +
> + for (i = 0; i < tcm_bank_count; i++) {
> + /* Use pre-defined TCM reg values.
> + * Eventually this should be replaced by values
> + * parsed from dts.
> + */
> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> + tcm_node++;
> + }
> +
> + r5_core->tcm_bank_count = tcm_bank_count;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> +{
> + int res_mem_count, i;
> + struct device *dev;
> + struct device_node *np, *rmem_np;
> + struct reserved_mem *rmem;
> +
> + dev = r5_core->dev;
> +
> + np = r5_core->np;
> + if (IS_ERR_OR_NULL(np)) {
> + pr_err("invalid device node of r5 core\n");
> + return -EINVAL;
> + }
> +
> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> + sizeof(phandle));
> + if (res_mem_count <= 0) {
> + dev_warn(dev, "failed to get memory-region property %d\n",
> + res_mem_count);
> + return -EINVAL;
> + }
> +
> + r5_core->res_mem = devm_kzalloc(dev,
> + res_mem_count * sizeof(struct reserved_mem),
> + GFP_KERNEL);
> + if (!r5_core->res_mem) {
> + dev_err(dev, "failed to allocate mem region memory\n");
> + return -ENOMEM;
> + }
> +
> + for (i = 0; i < res_mem_count; i++) {
> + rmem_np = of_parse_phandle(np, "memory-region", i);
> + if (!rmem_np)
> + return -EINVAL;
> +
> + rmem = of_reserved_mem_lookup(rmem_np);
> + if (!rmem) {
> + of_node_put(rmem_np);
> + return -EINVAL;
> + }
> +
> + memcpy(&r5_core->res_mem[i], rmem,
> + sizeof(struct reserved_mem));
> + of_node_put(rmem_np);
> + }
> +
> + r5_core->res_mem_count = res_mem_count;
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> +{
> + int ret, i;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev = cluster->dev;
> +
> + ret = zynqmp_r5_get_tcm_node(cluster);
> + if (ret < 0) {
> + dev_err(dev, "can't get tcm node, err %d\n", ret);
> + return ret;
> + }
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("invalid r5 core\n");
> + return -EINVAL;
> + }
> +
> + ret = zynqmp_r5_get_mem_region_node(r5_core);
> + if (ret)
> + dev_warn(dev, "memory-region prop failed %d\n", ret);
> +
> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> + 1, &r5_core->pm_domain_id);
> + if (ret) {
> + dev_err(dev, "failed to get power-domains property\n");
> + return ret;
> + }
> +
> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_add_rproc_core(r5_core);
> + if (ret) {
> + dev_err(dev, "failed to init r5 core %d\n", i);
> + return ret;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> +{
> + struct device *dev = cluster->dev;
> + struct device_node *dev_node = dev_of_node(dev);
> + struct device_node *child;
> + struct platform_device *child_pdev;
> + int core_count = 0, ret, i;
> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> + struct zynqmp_r5_core *r5_cores;
> +
> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> +
> + /* on success returns 0, if not defined then returns -EINVAL,
> + * In that case, default is LOCKSTEP mode
> + */
> + if (ret != -EINVAL && ret != 0) {
> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> + return -EINVAL;
> + }
> +
> + if (cluster_mode == SINGLE_CPU_MODE) {
> + dev_err(dev, "driver does not support single cpu mode\n");
> + return -EINVAL;
> + } else if ((cluster_mode != SPLIT_MODE &&
> + cluster_mode != LOCKSTEP_MODE)) {
> + dev_err(dev, "Invalid cluster mode\n");
> + return -EINVAL;
> + }
> +
> + core_count = of_get_available_child_count(dev_node);
> + if (core_count <= 0) {
> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> + return -EINVAL;
> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> + return -EINVAL;
> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> + dev_warn(dev, "Only r5 core0 will be used\n");
> + core_count = 1;
> + }
> +
> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> + core_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_cores)) {
> + dev_err(dev, "can't allocate memory for cores\n");
> + return -ENOMEM;
> + }
> +
> + i = 0;
> + for_each_available_child_of_node(dev_node, child) {
> + child_pdev = of_find_device_by_node(child);
> + if (!child_pdev)
> + return -ENODEV;
> +
> + r5_cores[i].dev = &child_pdev->dev;
> + if (!r5_cores[i].dev) {
> + pr_err("can't get device for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> + if (!r5_cores[i].np) {
> + pr_err("can't get device node for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + i++;
> + if (i == core_count)
> + break;
> + }
> +
> + cluster->mode = cluster_mode;
> + cluster->core_count = core_count;
> + cluster->r5_cores = r5_cores;
> +
> + ret = zynqmp_r5_core_init(cluster);
> + if (ret < 0) {
> + dev_err(dev, "failed to init r5 core err %d\n", ret);
> + return ret;
> + }
> +
> + zynqmp_r5_print_dt_node_info(cluster);
> +
> + return 0;
> +}
> +
> +static void zynqmp_r5_cluster_exit(void *data)
> +{
> + struct platform_device *pdev = (struct platform_device *)data;
> +
> + platform_set_drvdata(pdev, NULL);
> +
> + pr_info("Exit r5f subsystem driver\n");
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_probe()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * called when driver is probed, for each R5 core specified in DT,
> + * setup as needed to do remoteproc-related operations
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> +{
> + int ret;
> + struct zynqmp_r5_cluster *cluster;
> + struct device *dev = &pdev->dev;
> +
> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> + if (IS_ERR_OR_NULL(cluster))
> + return -ENOMEM;
> +
> + cluster->dev = dev;
> +
> + ret = devm_of_platform_populate(dev);
> + if (ret) {
> + dev_err(dev, "failed to populate platform dev %d\n", ret);
> + return ret;
> + }
> +
> + /* wire in so each core can be cleaned up at driver remove */
> + platform_set_drvdata(pdev, cluster);
> +
> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_cluster_init(cluster);
> + if (ret) {
> + dev_err(dev, "Invalid r5f subsystem device tree\n");
> + return ret;
> + }
> +
> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> + return 0;
> +}
> +
> +/* Match table for OF platform binding */
> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> + { .compatible = "xlnx,zynqmp-r5fss", },
> + { /* end of list */ },
> +};
> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> +
> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> + .probe = zynqmp_r5_remoteproc_probe,
> + .driver = {
> + .name = "zynqmp_r5_remoteproc",
> + .of_match_table = zynqmp_r5_remoteproc_match,
> + },
> +};
> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> +
> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> +MODULE_AUTHOR("Xilinx Inc.");
> +MODULE_LICENSE("GPL v2");
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-07 18:31 ` Mathieu Poirier
0 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-12-07 18:31 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
> This driver enables r5f dual core Real time Processing Unit subsystem
> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
> (cluster) can be configured in different modes e.g. split mode in which
> two r5f cores work independent of each other and lock-step mode in which
> both r5f cores execute same code clock-for-clock and notify if the
> result is different.
>
> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
>
> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
> ---
> drivers/remoteproc/Kconfig | 12 +
> drivers/remoteproc/Makefile | 1 +
> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
> 3 files changed, 972 insertions(+)
> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>
> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> index f30d00a3aabe..27f66910d8d3 100644
> --- a/drivers/remoteproc/Kconfig
> +++ b/drivers/remoteproc/Kconfig
> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
> It's safe to say N here if you're not interested in utilizing
> a slave processor.
>
> +config XLNX_R5_REMOTEPROC
> + tristate "Xilinx R5 remoteproc support"
> + depends on PM && ARCH_ZYNQMP
> + depends on ZYNQMP_FIRMWARE
> + select RPMSG_VIRTIO
> + select ZYNQMP_IPI_MBOX
> + help
> + Say y or m here to support Xilinx R5 remote processors via the remote
> + processor framework.
> +
> + It's safe to say N if not interested in using RPU r5f cores.
> +
> endif # REMOTEPROC
>
> endmenu
> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> index bb26c9e4ef9c..334a8bed4c14 100644
> --- a/drivers/remoteproc/Makefile
> +++ b/drivers/remoteproc/Makefile
> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> new file mode 100644
> index 000000000000..c2167fd3869d
> --- /dev/null
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -0,0 +1,959 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * ZynqMP R5 Remote Processor driver
> + *
> + */
> +
> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
> +#include <linux/firmware/xlnx-zynqmp.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/mailbox_client.h>
> +#include <linux/mailbox/zynqmp-ipi-message.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/remoteproc.h>
> +#include <linux/skbuff.h>
> +#include <linux/sysfs.h>
> +
> +#include "remoteproc_internal.h"
> +
> +/* settings for RPU cluster mode */
> +enum zynqmp_r5_cluster_mode {
> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
> +};
> +
> +/**
> + * struct mem_bank_data - Memory Bank description
> + *
> + * @addr: Start address of memory bank
> + * @size: Size of Memory bank
> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
> + * @bank_name: name of the bank for remoteproc framework
> + */
> +struct mem_bank_data {
> + phys_addr_t addr;
> + size_t size;
> + enum pm_node_id pm_domain_id;
> + char *bank_name;
> +};
> +
> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
> +};
> +
> +/**
> + * struct zynqmp_r5_core - ZynqMP R5 core structure
> + *
> + * @dev: device of RPU instance
> + * @np: device node of RPU instance
> + * @tcm_bank_count: number TCM banks accessible to this RPU
> + * @tcm_banks: array of each TCM bank data
> + * @res_mem_count: number of Reserved Memory regions per core
> + * @res_mem: array of reserved memory regions
> + * @rproc: rproc handle
> + * @pm_domain_id: RPU CPU power domain id
> + */
> +struct zynqmp_r5_core {
> + struct device *dev;
> + struct device_node *np;
> + int tcm_bank_count;
> + struct mem_bank_data *tcm_banks;
> + int res_mem_count;
> + struct reserved_mem *res_mem;
> + struct rproc *rproc;
> + enum pm_node_id pm_domain_id;
> +};
> +
> +/**
> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
> + *
> + * @dev: r5f subsystem cluster device node
> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
> + * @core_count: number of r5 cores used for this cluster mode
> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
> + */
> +struct zynqmp_r5_cluster {
> + struct device *dev;
> + enum zynqmp_r5_cluster_mode mode;
> + int core_count;
> + struct zynqmp_r5_core *r5_cores;
> +};
> +
> +/*
> + * zynqmp_r5_set_mode - set RPU operation mode
> + *
> + * set RPU operation mode
> + *
> + * Return: 0 for success, negative value for failure
> + */
> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
> + enum zynqmp_r5_cluster_mode rpu_mode)
> +{
> + enum rpu_tcm_comb tcm_mode;
> + int ret, reg_val;
> +
> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
> +
> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
> + if (ret < 0) {
> + pr_err("failed to set RPU mode\n");
> + return ret;
> + }
> +
> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
> + if (ret < 0)
> + pr_err("failed to configure TCM\n");
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_start
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Start R5 Core from designated boot address.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + enum rpu_boot_mem bootmem;
> + int ret;
> +
> + if (!r5_core) {
> + pr_err("can't get r5 core\n");
> + return -EINVAL;
> + }
Please remove.
> +
> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
Some comments as to what is happening here would be appreciated.
> +
> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> +
> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> + if (ret)
> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_stop
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Power down R5 Core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> +{
> + struct zynqmp_r5_core *r5_core = rproc->priv;
> + int ret;
> +
> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret)
> + pr_err("failed to stop remoteproc RPU %d\n", ret);
> +
> + return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + mem->va = (void *)va;
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap memory-region carveout
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + iounmap((void __iomem *)mem->va);
> + return 0;
> +}
> +
> +/*
> + * add_mem_regions
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Construct rproc mem carveouts from carveout provided in
> + * memory-region property
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_mem_regions(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct rproc_mem_entry *mem;
> + struct reserved_mem *rmem;
> + struct zynqmp_r5_core *r5_core;
> + int i;
> +
> + r5_core = rproc->priv;
> + dev = r5_core->dev;
> +
> + /* Register associated reserved memory regions */
> + for (i = 0; i < r5_core->res_mem_count; i++) {
> + rmem = &r5_core->res_mem[i];
Indentation problem
> + mem = rproc_mem_entry_init(dev, NULL,
> + (dma_addr_t)rmem->base,
> + rmem->size, rmem->base,
> + zynqmp_r5_rproc_mem_map,
> + zynqmp_r5_rproc_mem_unmap,
> + rmem->name);
> + if (IS_ERR_OR_NULL(mem))
> + return -ENOMEM;
> +
> + rproc_add_carveout(rproc, mem);
> + }
Not sure why this wasn't done in zynqmp_r5_get_mem_region_node()... That would
prevent the need to keep a copy of the reserved memories in the zynqmp_r5_core
structure.
> +
> + return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_unmap
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap TCM banks when powering down R5 core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> + struct zynqmp_r5_core *r5_core;
> + int i;
> + enum pm_node_id pm_domain_id;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + pr_err("r5 core is not available\n");
> + return -EINVAL;
> + }
> +
> + iounmap((void __iomem *)mem->va);
> +
> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + if (zynqmp_pm_release_node(pm_domain_id))
> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * tcm_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to initialize the va and da fields of
> + *
> + * Given TCM bank entry, this callback will set device address for R5
> + * running on TCM and also setup virtual address for TCM bank
> + * remoteproc carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
Indentation problem
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + /* Update memory entry va */
> + mem->va = (void *)va;
> +
> + /* clear TCMs */
> + memset_io(va, 0, mem->len);
> +
> + /*
> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> + * while on the Linux side they are at 0xffexxxxx.
> + *
> + * Zero out the high 12 bits of the address. This will give
> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> + */
> + mem->da &= 0x000fffff;
> +
> + /*
> + * TCM Banks 1A and 1B still have to be translated.
> + *
> + * Below handle these two banks' absolute addresses (0xffe90000 and
> + * 0xffeb0000) and convert to the expected relative addresses
> + * (0x0 and 0x20000).
> + */
> + if (mem->da == 0x90000 || mem->da == 0xB0000)
> + mem->da -= 0x90000;
> +
> + /* if translated TCM bank address is not valid report error */
> + if (mem->da != 0x0 && mem->da != 0x20000) {
> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
> + return -EINVAL;
> + }
> + return 0;
> +}
> +
> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
phys_addr_t bank_addr;
> + size_t bank_size = 0;
No initialisation needed.
> + char *bank_name;
> + struct device *dev;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
Remove.
> +
> + dev = r5_core->dev;
> +
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + for (i = 0; i < num_banks; i++) {
> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
> + bank_name = r5_core->tcm_banks[i].bank_name;
> + bank_size = r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> +
> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
I'm pretty sure that if you run the sparse checker on this you will get a lot of
angry warnings.
> + if (IS_ERR_OR_NULL(mem)) {
Function rproc_mem_entry_init() does not return an error code.
> + /* Turn off all TCM banks turned on before */
> + do {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
Here @pm_domain_id is casted as a u32 but it isn't above when calling
zynqmp_pm_request_node(), and that despite both functions taking a "const u32
node". Again, I'm sure the sparse checker will complain here.
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x, %x\n",
> + (u32)pm_domain_id, ret);
> + } while (i--);
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> + }
> +
> + return 0;
> +}
> +
> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> +{
> + int i, num_banks, ret;
> + struct rproc_mem_entry *mem;
> + enum pm_node_id pm_domain_id;
> + u32 bank_addr;
> + size_t bank_size = 0;
Same as above.
> + char *bank_name;
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
Same as above.
> + /* go through zynqmp banks for r5 node */
> + num_banks = r5_core->tcm_bank_count;
> + if (num_banks <= 0) {
> + dev_err(dev, "need to specify TCM banks\n");
> + return -EINVAL;
> + }
> +
> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
> + bank_name = r5_core->tcm_banks[0].bank_name;
> + for (i = 0; i < num_banks; i++) {
> + bank_size += r5_core->tcm_banks[i].size;
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> +
> + ret = zynqmp_pm_request_node(pm_domain_id,
> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> + if (ret < 0) {
> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
> + return ret;
> + }
> + }
I know what is happening above but in 6 months it will all be gone from
memory. Please add enough documentation for someone not familiar with the
platform to understand why things are done that way. Here and throughout the
driver.
> +
> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
> + bank_name, bank_addr, bank_size);
> +
> + /* add carveout */
> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> + bank_size, bank_addr,
> + tcm_mem_map, tcm_mem_unmap,
> + bank_name);
> + if (IS_ERR_OR_NULL(mem)) {
> + for (i = 0; i < num_banks; i++) {
> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> + if (ret)
> + dev_warn(dev,
> + "fail to release node: %x ret: %x\n",
> + (u32)pm_domain_id, ret);
> + }
> + return -ENOMEM;
> + }
> +
> + rproc_add_carveout(rproc, mem);
> +
> + return 0;
> +}
> +
> +/*
> + * add_tcm_banks()
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Given R5 node in remoteproc instance
> + * allocate remoteproc carveout for TCM memory
> + * needed for firmware to be loaded
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int add_tcm_banks(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
Remove.
> +
> + parent_pdev = to_platform_device(dev->parent);
> + if (!parent_pdev) {
> + dev_err(dev, "parent platform dev unavailable\n");
> + return -ENODEV;
> + }
> +
> + cluster = platform_get_drvdata(parent_pdev);
> + if (!cluster) {
> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
> + return -EINVAL;
> + }
Why not simply calling dev_get_drvdata(dev->parent) instead of going through the
platform API?
> +
> + if (cluster->mode == SPLIT_MODE)
> + return add_tcm_carveout_split_mode(rproc);
> + else if (cluster->mode == LOCKSTEP_MODE)
> + return add_tcm_carveout_lockstep_mode(rproc);
> +
> + dev_err(cluster->dev, "invalid cluster mode\n");
> + return -EINVAL;
> +}
> +
> +/*
> + * zynqmp_r5_parse_fw()
> + * @rproc: single R5 core's corresponding rproc instance
> + * @fw: ptr to firmware to be loaded onto r5 core
> + *
> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> + int ret;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev;
> +
> + r5_core = rproc->priv;
> + if (!r5_core) {
> + dev_err(&rproc->dev, "r5 core not available\n");
> + return -EINVAL;
> + }
Same as above and throughout.
I am done reviewing this set - the overall structure is sound but there is a
fair amount of work to be done. There is no need to rush another revision as I
won't have time to review it before the holidays. On that front, change
_only_ the things I have commented on. We can address other issues as we move
forward. Last but not least, please run the sparse checker on your work.
Thanks,
Mathieu
> +
> + dev = r5_core->dev;
> +
> + ret = add_tcm_banks(rproc);
> + if (ret) {
> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
> + return ret;
> + }
> +
> + ret = add_mem_regions(rproc);
> + if (ret)
> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
> +
> + ret = rproc_elf_load_rsc_table(rproc, fw);
> + if (ret == -EINVAL) {
> + /*
> + * resource table only required for IPC.
> + * if not present, this is not necessarily an error;
> + * for example, loading r5 hello world application
> + * so simply inform user and keep going.
> + */
> + dev_info(&rproc->dev, "no resource table found.\n");
> + ret = 0;
> + }
> + return ret;
> +}
> +
> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> + .start = zynqmp_r5_rproc_start,
> + .stop = zynqmp_r5_rproc_stop,
> + .load = rproc_elf_load_segments,
> + .parse_fw = zynqmp_r5_parse_fw,
> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> + .sanity_check = rproc_elf_sanity_check,
> + .get_boot_addr = rproc_elf_get_boot_addr,
> +};
> +
> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
> +{
> + int i, j, k;
> + struct zynqmp_r5_core *r5_core;
> +
> + dev_dbg(cluster->dev, "Printing dt node info\n");
> +
> + pr_debug("cluster mode = %d\n", cluster->mode);
> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
> + pr_debug("r5f num cores = %d\n", cluster->core_count);
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("can't get r5_core\n");
> + continue;
> + }
> +
> + pr_debug("r5 core %d nodes\n", i);
> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
> + k, r5_core->tcm_banks[k].addr,
> + r5_core->tcm_banks[k].size,
> + r5_core->tcm_banks[k].pm_domain_id,
> + r5_core->tcm_banks[k].bank_name);
> + }
> +
> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
> +
> + for (j = 0; j < r5_core->res_mem_count; j++) {
> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
> + j, r5_core->res_mem[j].base,
> + r5_core->res_mem[j].size,
> + r5_core->res_mem[j].name);
> + }
> + }
> +}
> +
> +/**
> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
> + * this is called for each individual R5 core to
> + * set up mailbox, Xilinx platform manager unique ID,
> + * add to rproc core
> + *
> + * @r5_core: zynqmp_r5_core r5 core object to initialize
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
> +{
> + int ret;
> + struct rproc *r5_rproc;
> + struct device *dev;
> +
> + dev = r5_core->dev;
> +
> + /* Set up DMA mask */
> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> + if (ret)
> + return ret;
> +
> + /* Allocate remoteproc instance */
> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> + NULL, sizeof(struct zynqmp_r5_core));
> + if (IS_ERR_OR_NULL(r5_rproc))
> + return -ENOMEM;
> +
> + r5_rproc->auto_boot = false;
> + r5_rproc->priv = r5_core;
> +
> + /* Add R5 remoteproc */
> + ret = devm_rproc_add(dev, r5_rproc);
> + if (ret) {
> + pr_err("failed to add r5 remoteproc\n");
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> +{
> + int tcm_bank_count, tcm_node;
> + int i = 0, j;
> + struct zynqmp_r5_core *r5_core;
> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> + struct device *dev = cluster->dev;
> +
> + /* ToDo: Use predefined TCM address space values from driver until
> + * system-dt spec is not final fot TCM
> + */
> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> +
> + /* count per core tcm banks */
> + tcm_bank_count = tcm_bank_count / cluster->core_count;
> +
> + /* r5 core 0 will use all of TCM banks in lockstep mode.
> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> + * 128k. Assign TCM banks to each core accordingly
> + */
> + tcm_node = 0;
> + for (j = 0; j < cluster->core_count; j++) {
> + r5_core = &cluster->r5_cores[j];
> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> + tcm_bank_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> + return -ENOMEM;
> +
> + for (i = 0; i < tcm_bank_count; i++) {
> + /* Use pre-defined TCM reg values.
> + * Eventually this should be replaced by values
> + * parsed from dts.
> + */
> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> + tcm_node++;
> + }
> +
> + r5_core->tcm_bank_count = tcm_bank_count;
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> +{
> + int res_mem_count, i;
> + struct device *dev;
> + struct device_node *np, *rmem_np;
> + struct reserved_mem *rmem;
> +
> + dev = r5_core->dev;
> +
> + np = r5_core->np;
> + if (IS_ERR_OR_NULL(np)) {
> + pr_err("invalid device node of r5 core\n");
> + return -EINVAL;
> + }
> +
> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> + sizeof(phandle));
> + if (res_mem_count <= 0) {
> + dev_warn(dev, "failed to get memory-region property %d\n",
> + res_mem_count);
> + return -EINVAL;
> + }
> +
> + r5_core->res_mem = devm_kzalloc(dev,
> + res_mem_count * sizeof(struct reserved_mem),
> + GFP_KERNEL);
> + if (!r5_core->res_mem) {
> + dev_err(dev, "failed to allocate mem region memory\n");
> + return -ENOMEM;
> + }
> +
> + for (i = 0; i < res_mem_count; i++) {
> + rmem_np = of_parse_phandle(np, "memory-region", i);
> + if (!rmem_np)
> + return -EINVAL;
> +
> + rmem = of_reserved_mem_lookup(rmem_np);
> + if (!rmem) {
> + of_node_put(rmem_np);
> + return -EINVAL;
> + }
> +
> + memcpy(&r5_core->res_mem[i], rmem,
> + sizeof(struct reserved_mem));
> + of_node_put(rmem_np);
> + }
> +
> + r5_core->res_mem_count = res_mem_count;
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> +{
> + int ret, i;
> + struct zynqmp_r5_core *r5_core;
> + struct device *dev = cluster->dev;
> +
> + ret = zynqmp_r5_get_tcm_node(cluster);
> + if (ret < 0) {
> + dev_err(dev, "can't get tcm node, err %d\n", ret);
> + return ret;
> + }
> +
> + for (i = 0; i < cluster->core_count; i++) {
> + r5_core = &cluster->r5_cores[i];
> + if (!r5_core) {
> + pr_err("invalid r5 core\n");
> + return -EINVAL;
> + }
> +
> + ret = zynqmp_r5_get_mem_region_node(r5_core);
> + if (ret)
> + dev_warn(dev, "memory-region prop failed %d\n", ret);
> +
> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> + 1, &r5_core->pm_domain_id);
> + if (ret) {
> + dev_err(dev, "failed to get power-domains property\n");
> + return ret;
> + }
> +
> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_add_rproc_core(r5_core);
> + if (ret) {
> + dev_err(dev, "failed to init r5 core %d\n", i);
> + return ret;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> +{
> + struct device *dev = cluster->dev;
> + struct device_node *dev_node = dev_of_node(dev);
> + struct device_node *child;
> + struct platform_device *child_pdev;
> + int core_count = 0, ret, i;
> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> + struct zynqmp_r5_core *r5_cores;
> +
> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> +
> + /* on success returns 0, if not defined then returns -EINVAL,
> + * In that case, default is LOCKSTEP mode
> + */
> + if (ret != -EINVAL && ret != 0) {
> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> + return -EINVAL;
> + }
> +
> + if (cluster_mode == SINGLE_CPU_MODE) {
> + dev_err(dev, "driver does not support single cpu mode\n");
> + return -EINVAL;
> + } else if ((cluster_mode != SPLIT_MODE &&
> + cluster_mode != LOCKSTEP_MODE)) {
> + dev_err(dev, "Invalid cluster mode\n");
> + return -EINVAL;
> + }
> +
> + core_count = of_get_available_child_count(dev_node);
> + if (core_count <= 0) {
> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> + return -EINVAL;
> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> + return -EINVAL;
> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> + dev_warn(dev, "Only r5 core0 will be used\n");
> + core_count = 1;
> + }
> +
> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> + core_count, GFP_KERNEL);
> + if (IS_ERR_OR_NULL(r5_cores)) {
> + dev_err(dev, "can't allocate memory for cores\n");
> + return -ENOMEM;
> + }
> +
> + i = 0;
> + for_each_available_child_of_node(dev_node, child) {
> + child_pdev = of_find_device_by_node(child);
> + if (!child_pdev)
> + return -ENODEV;
> +
> + r5_cores[i].dev = &child_pdev->dev;
> + if (!r5_cores[i].dev) {
> + pr_err("can't get device for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> + if (!r5_cores[i].np) {
> + pr_err("can't get device node for r5 core %d\n", i);
> + return -ENODEV;
> + }
> +
> + i++;
> + if (i == core_count)
> + break;
> + }
> +
> + cluster->mode = cluster_mode;
> + cluster->core_count = core_count;
> + cluster->r5_cores = r5_cores;
> +
> + ret = zynqmp_r5_core_init(cluster);
> + if (ret < 0) {
> + dev_err(dev, "failed to init r5 core err %d\n", ret);
> + return ret;
> + }
> +
> + zynqmp_r5_print_dt_node_info(cluster);
> +
> + return 0;
> +}
> +
> +static void zynqmp_r5_cluster_exit(void *data)
> +{
> + struct platform_device *pdev = (struct platform_device *)data;
> +
> + platform_set_drvdata(pdev, NULL);
> +
> + pr_info("Exit r5f subsystem driver\n");
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_probe()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * called when driver is probed, for each R5 core specified in DT,
> + * setup as needed to do remoteproc-related operations
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> +{
> + int ret;
> + struct zynqmp_r5_cluster *cluster;
> + struct device *dev = &pdev->dev;
> +
> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> + if (IS_ERR_OR_NULL(cluster))
> + return -ENOMEM;
> +
> + cluster->dev = dev;
> +
> + ret = devm_of_platform_populate(dev);
> + if (ret) {
> + dev_err(dev, "failed to populate platform dev %d\n", ret);
> + return ret;
> + }
> +
> + /* wire in so each core can be cleaned up at driver remove */
> + platform_set_drvdata(pdev, cluster);
> +
> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> + if (ret)
> + return ret;
> +
> + ret = zynqmp_r5_cluster_init(cluster);
> + if (ret) {
> + dev_err(dev, "Invalid r5f subsystem device tree\n");
> + return ret;
> + }
> +
> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> + return 0;
> +}
> +
> +/* Match table for OF platform binding */
> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> + { .compatible = "xlnx,zynqmp-r5fss", },
> + { /* end of list */ },
> +};
> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> +
> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> + .probe = zynqmp_r5_remoteproc_probe,
> + .driver = {
> + .name = "zynqmp_r5_remoteproc",
> + .of_match_table = zynqmp_r5_remoteproc_match,
> + },
> +};
> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> +
> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> +MODULE_AUTHOR("Xilinx Inc.");
> +MODULE_LICENSE("GPL v2");
> --
> 2.25.1
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-12-03 18:55 ` Mathieu Poirier
@ 2021-12-13 8:10 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-13 8:10 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
Hi Mathieu,
Thanks for reviews. Please find my comments below.
On 12/4/21 12:25 AM, Mathieu Poirier wrote:
> Hi Tanmay,
>
> On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
>> This driver enables r5f dual core Real time Processing Unit subsystem
>> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
>> (cluster) can be configured in different modes e.g. split mode in which
>> two r5f cores work independent of each other and lock-step mode in which
>> both r5f cores execute same code clock-for-clock and notify if the
>> result is different.
>>
>> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
>> Platform Management Unit that handles the R5 configuration, memory access
>> and R5 lifecycle management. The interface to this manager is done in this
>> driver via zynqmp_pm_* function calls.
>>
>> Signed-off-by: Ben Levinsky<ben.levinsky@xilinx.com>
>> Signed-off-by: Tanmay Shah<tanmay.shah@xilinx.com>
>> ---
>> drivers/remoteproc/Kconfig | 12 +
>> drivers/remoteproc/Makefile | 1 +
>> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
>> 3 files changed, 972 insertions(+)
>> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>>
>> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
>> index f30d00a3aabe..27f66910d8d3 100644
>> --- a/drivers/remoteproc/Kconfig
>> +++ b/drivers/remoteproc/Kconfig
>> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
>> It's safe to say N here if you're not interested in utilizing
>> a slave processor.
>>
>> +config XLNX_R5_REMOTEPROC
>> + tristate "Xilinx R5 remoteproc support"
>> + depends on PM && ARCH_ZYNQMP
>> + depends on ZYNQMP_FIRMWARE
>> + select RPMSG_VIRTIO
>> + select ZYNQMP_IPI_MBOX
>> + help
>> + Say y or m here to support Xilinx R5 remote processors via the remote
>> + processor framework.
>> +
>> + It's safe to say N if not interested in using RPU r5f cores.
>> +
>> endif # REMOTEPROC
>>
>> endmenu
>> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
>> index bb26c9e4ef9c..334a8bed4c14 100644
>> --- a/drivers/remoteproc/Makefile
>> +++ b/drivers/remoteproc/Makefile
>> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
>> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
>> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
>> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
>> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
>> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> new file mode 100644
>> index 000000000000..c2167fd3869d
>> --- /dev/null
>> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> @@ -0,0 +1,959 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ZynqMP R5 Remote Processor driver
>> + *
>> + */
>> +
>> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
>> +#include <linux/firmware/xlnx-zynqmp.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/kernel.h>
>> +#include <linux/list.h>
>> +#include <linux/mailbox_client.h>
>> +#include <linux/mailbox/zynqmp-ipi-message.h>
>> +#include <linux/module.h>
>> +#include <linux/of_address.h>
>> +#include <linux/of_platform.h>
>> +#include <linux/of_reserved_mem.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/remoteproc.h>
>> +#include <linux/skbuff.h>
>> +#include <linux/sysfs.h>
>> +
>> +#include "remoteproc_internal.h"
>> +
>> +/* settings for RPU cluster mode */
>> +enum zynqmp_r5_cluster_mode {
>> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
>> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
>> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
> Please use C style comments, i.e /*...*/
>
>> +};
>> +
>> +/**
>> + * struct mem_bank_data - Memory Bank description
>> + *
>> + * @addr: Start address of memory bank
>> + * @size: Size of Memory bank
>> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
>> + * @bank_name: name of the bank for remoteproc framework
>> + */
>> +struct mem_bank_data {
>> + phys_addr_t addr;
>> + size_t size;
>> + enum pm_node_id pm_domain_id;
>> + char *bank_name;
>> +};
>> +
>> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
>> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
>> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
>> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
>> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
>> +};
> Bizarre - more comments on this below...
I know ideally it should have been in yaml bindings and we should parse
it from device-tree. Please find my explanation below for this approach.
>> +
>> +/**
>> + * struct zynqmp_r5_core - ZynqMP R5 core structure
>> + *
>> + * @dev: device of RPU instance
>> + * @np: device node of RPU instance
>> + * @tcm_bank_count: number TCM banks accessible to this RPU
>> + * @tcm_banks: array of each TCM bank data
>> + * @res_mem_count: number of Reserved Memory regions per core
>> + * @res_mem: array of reserved memory regions
>> + * @rproc: rproc handle
>> + * @pm_domain_id: RPU CPU power domain id
>> + */
>> +struct zynqmp_r5_core {
>> + struct device *dev;
>> + struct device_node *np;
>> + int tcm_bank_count;
>> + struct mem_bank_data *tcm_banks;
>> + int res_mem_count;
>> + struct reserved_mem *res_mem;
>> + struct rproc *rproc;
>> + enum pm_node_id pm_domain_id;
>> +};
>> +
>> +/**
>> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
>> + *
>> + * @dev: r5f subsystem cluster device node
>> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
>> + * @core_count: number of r5 cores used for this cluster mode
>> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
>> + */
>> +struct zynqmp_r5_cluster {
>> + struct device *dev;
>> + enum zynqmp_r5_cluster_mode mode;
>> + int core_count;
>> + struct zynqmp_r5_core *r5_cores;
>> +};
>> +
>> +/*
>> + * zynqmp_r5_set_mode - set RPU operation mode
>> + *
>> + * set RPU operation mode
>> + *
>> + * Return: 0 for success, negative value for failure
>> + */
>> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
>> + enum zynqmp_r5_cluster_mode rpu_mode)
>> +{
>> + enum rpu_tcm_comb tcm_mode;
>> + int ret, reg_val;
>> +
>> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
>> +
>> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
>> + if (ret < 0) {
>> + pr_err("failed to set RPU mode\n");
>> + return ret;
>> + }
>> +
>> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
>> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
>> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
>> + if (ret < 0)
>> + pr_err("failed to configure TCM\n");
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_start
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Start R5 Core from designated boot address.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + enum rpu_boot_mem bootmem;
>> + int ret;
>> +
>> + if (!r5_core) {
>> + pr_err("can't get r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
>> +
>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>> +
>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>> + if (ret)
>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_stop
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Power down R5 Core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + int ret;
>> +
>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret)
>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap memory-region carveout
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + iounmap((void __iomem *)mem->va);
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_mem_regions
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Construct rproc mem carveouts from carveout provided in
>> + * memory-region property
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_mem_regions(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct rproc_mem_entry *mem;
>> + struct reserved_mem *rmem;
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> +
>> + r5_core = rproc->priv;
>> + dev = r5_core->dev;
>> +
>> + /* Register associated reserved memory regions */
>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>> + rmem = &r5_core->res_mem[i];
>> + mem = rproc_mem_entry_init(dev, NULL,
>> + (dma_addr_t)rmem->base,
>> + rmem->size, rmem->base,
>> + zynqmp_r5_rproc_mem_map,
>> + zynqmp_r5_rproc_mem_unmap,
>> + rmem->name);
>> + if (IS_ERR_OR_NULL(mem))
>> + return -ENOMEM;
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap TCM banks when powering down R5 core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>> +{
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> + enum pm_node_id pm_domain_id;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + pr_err("r5 core is not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + iounmap((void __iomem *)mem->va);
>> +
>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + if (zynqmp_pm_release_node(pm_domain_id))
>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * tcm_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to initialize the va and da fields of
>> + *
>> + * Given TCM bank entry, this callback will set device address for R5
>> + * running on TCM and also setup virtual address for TCM bank
>> + * remoteproc carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + /* Update memory entry va */
>> + mem->va = (void *)va;
>> +
>> + /* clear TCMs */
>> + memset_io(va, 0, mem->len);
>> +
>> + /*
>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>> + * while on the Linux side they are at 0xffexxxxx.
>> + *
>> + * Zero out the high 12 bits of the address. This will give
>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>> + */
>> + mem->da &= 0x000fffff;
>> +
>> + /*
>> + * TCM Banks 1A and 1B still have to be translated.
>> + *
>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>> + * 0xffeb0000) and convert to the expected relative addresses
>> + * (0x0 and 0x20000).
>> + */
>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>> + mem->da -= 0x90000;
>> +
>> + /* if translated TCM bank address is not valid report error */
>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>> + return -EINVAL;
>> + }
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; i < num_banks; i++) {
>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>> + bank_name = r5_core->tcm_banks[i].bank_name;
>> + bank_size = r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + /* Turn off all TCM banks turned on before */
>> + do {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x, %x\n",
>> + (u32)pm_domain_id, ret);
>> + } while (i--);
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>> + bank_name = r5_core->tcm_banks[0].bank_name;
>> + for (i = 0; i < num_banks; i++) {
>> + bank_size += r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> + }
>> +
>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + for (i = 0; i < num_banks; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x ret: %x\n",
>> + (u32)pm_domain_id, ret);
>> + }
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_tcm_banks()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Given R5 node in remoteproc instance
>> + * allocate remoteproc carveout for TCM memory
>> + * needed for firmware to be loaded
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_parse_fw()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @fw: ptr to firmware to be loaded onto r5 core
>> + *
>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>> +{
>> + int ret;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + dev_err(&rproc->dev, "r5 core not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + dev = r5_core->dev;
>> +
>> + ret = add_tcm_banks(rproc);
>> + if (ret) {
>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = add_mem_regions(rproc);
>> + if (ret)
>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>> +
>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>> + if (ret == -EINVAL) {
>> + /*
>> + * resource table only required for IPC.
>> + * if not present, this is not necessarily an error;
>> + * for example, loading r5 hello world application
>> + * so simply inform user and keep going.
>> + */
>> + dev_info(&rproc->dev, "no resource table found.\n");
>> + ret = 0;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> +
>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int i, j, k;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + dev_dbg(cluster->dev, "Printing dt node info\n");
> Why mixing dev_dbg() and pr_debug()? Please pick one and stick with it.
>
> Even for debug level output this is very chatty - is all that information really
> needed when it is already available in /proc/device-tree?
I kept this function for debugging. However, looks like we can use
/proc/device-tree for same purpose, so I will remove this function.
Thanks for this suggestion.
>> +
>> + pr_debug("cluster mode = %d\n", cluster->mode);
>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("can't get r5_core\n");
>> + continue;
>> + }
>> +
>> + pr_debug("r5 core %d nodes\n", i);
>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>> + k, r5_core->tcm_banks[k].addr,
>> + r5_core->tcm_banks[k].size,
>> + r5_core->tcm_banks[k].pm_domain_id,
>> + r5_core->tcm_banks[k].bank_name);
>> + }
>> +
>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>> +
>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>> + j, r5_core->res_mem[j].base,
>> + r5_core->res_mem[j].size,
>> + r5_core->res_mem[j].name);
>> + }
>> + }
>> +}
>> +
>> +/**
>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>> + * this is called for each individual R5 core to
>> + * set up mailbox, Xilinx platform manager unique ID,
>> + * add to rproc core
>> + *
>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>> +{
>> + int ret;
>> + struct rproc *r5_rproc;
>> + struct device *dev;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* Set up DMA mask */
>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>> + if (ret)
>> + return ret;
>> +
>> + /* Allocate remoteproc instance */
>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>> + NULL, sizeof(struct zynqmp_r5_core));
>> + if (IS_ERR_OR_NULL(r5_rproc))
>> + return -ENOMEM;
>> +
>> + r5_rproc->auto_boot = false;
>> + r5_rproc->priv = r5_core;
>> +
>> + /* Add R5 remoteproc */
>> + ret = devm_rproc_add(dev, r5_rproc);
>> + if (ret) {
>> + pr_err("failed to add r5 remoteproc\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int tcm_bank_count, tcm_node;
>> + int i = 0, j;
>> + struct zynqmp_r5_core *r5_core;
>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>> + struct device *dev = cluster->dev;
>> +
>> + /* ToDo: Use predefined TCM address space values from driver until
>> + * system-dt spec is not final fot TCM
>> + */
> Multilined comment should be as follow:
>
> /*
> * ToDo: Use predefined TCM address space values from driver until
> * system-dt spec is not final fot TCM
> */
>
> s/"final fot TCM"/"final for TCM"
>
> Any reason this can't be done with "reg" properties like TI did for K3? It
> would be nice to have TCMs included in the yaml file example.
>
R5 TCM on Xilinx platforms contains separate power-domains than RPU
power-domains. So, I believe they can be operated separately than RPU.
That is why I chose not to include as reg property. I believe we will
include TCM as sram property in future.
Now, explanation about why TCM is hardcode in driver:
When I started developing driver, system-dt spec was in progress, to
avoid extra maintenance effort I defined TCM information in driver
instead of defining them in YAML.
I agree with idea that there is no need to maintain two separate
bindings and until system-dt specs are maturing we can hardcode TCM
related information driver. This was discussed previously here:
https://lore.kernel.org/all/CAL_JsqLGo380SRYska+xGgJhgF8NCRvY56ewafvSCU6c-LmhZw@mail.gmail.com/
Patchwork link for the same:
https://patchwork.kernel.org/project/linux-remoteproc/patch/1587749770-15082-5-git-send-email-ben.levinsky@xilinx.com/#23414963
Also, I will sync with system-dt team about its current status and keep
updating about system-dt specs.
Please let me know your opinion / suggestion for any alternative approach.
I agree with rest of the comments in this thread, and I will address
them all in next revision.
>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>> +
>> + /* count per core tcm banks */
>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>> +
>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>> + * 128k. Assign TCM banks to each core accordingly
>> + */
>> + tcm_node = 0;
>> + for (j = 0; j < cluster->core_count; j++) {
>> + r5_core = &cluster->r5_cores[j];
>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>> + tcm_bank_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>> + return -ENOMEM;
>> +
>> + for (i = 0; i < tcm_bank_count; i++) {
>> + /* Use pre-defined TCM reg values.
>> + * Eventually this should be replaced by values
>> + * parsed from dts.
>> + */
>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>> + tcm_node++;
>> + }
>> +
>> + r5_core->tcm_bank_count = tcm_bank_count;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>> +{
>> + int res_mem_count, i;
>> + struct device *dev;
>> + struct device_node *np, *rmem_np;
>> + struct reserved_mem *rmem;
>> +
>> + dev = r5_core->dev;
>> +
>> + np = r5_core->np;
>> + if (IS_ERR_OR_NULL(np)) {
>> + pr_err("invalid device node of r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>> + sizeof(phandle));
>> + if (res_mem_count <= 0) {
>> + dev_warn(dev, "failed to get memory-region property %d\n",
>> + res_mem_count);
>> + return -EINVAL;
>> + }
>> +
>> + r5_core->res_mem = devm_kzalloc(dev,
>> + res_mem_count * sizeof(struct reserved_mem),
>> + GFP_KERNEL);
>> + if (!r5_core->res_mem) {
>> + dev_err(dev, "failed to allocate mem region memory\n");
>> + return -ENOMEM;
>> + }
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int ret, i;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev = cluster->dev;
>> +
>> + ret = zynqmp_r5_get_tcm_node(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("invalid r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>> + if (ret)
>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>> +
>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>> + 1, &r5_core->pm_domain_id);
>> + if (ret) {
>> + dev_err(dev, "failed to get power-domains property\n");
>> + return ret;
>> + }
>> +
>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>> + if (ret) {
>> + dev_err(dev, "failed to init r5 core %d\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + struct device *dev = cluster->dev;
>> + struct device_node *dev_node = dev_of_node(dev);
>> + struct device_node *child;
>> + struct platform_device *child_pdev;
>> + int core_count = 0, ret, i;
>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>> + struct zynqmp_r5_core *r5_cores;
>> +
>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>> +
>> + /* on success returns 0, if not defined then returns -EINVAL,
>> + * In that case, default is LOCKSTEP mode
>> + */
>> + if (ret != -EINVAL && ret != 0) {
>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster_mode == SINGLE_CPU_MODE) {
>> + dev_err(dev, "driver does not support single cpu mode\n");
>> + return -EINVAL;
>> + } else if ((cluster_mode != SPLIT_MODE &&
>> + cluster_mode != LOCKSTEP_MODE)) {
>> + dev_err(dev, "Invalid cluster mode\n");
>> + return -EINVAL;
>> + }
>> +
>> + core_count = of_get_available_child_count(dev_node);
>> + if (core_count <= 0) {
>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>> + return -EINVAL;
>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>> + return -EINVAL;
>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>> + dev_warn(dev, "Only r5 core0 will be used\n");
>> + core_count = 1;
>> + }
>> +
>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>> + core_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_cores)) {
>> + dev_err(dev, "can't allocate memory for cores\n");
>> + return -ENOMEM;
>> + }
>> +
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
>> + return -ENODEV;
>> +
>> + r5_cores[i].dev = &child_pdev->dev;
>> + if (!r5_cores[i].dev) {
>> + pr_err("can't get device for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>> + if (!r5_cores[i].np) {
>> + pr_err("can't get device node for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + i++;
>> + if (i == core_count)
>> + break;
>> + }
>> +
>> + cluster->mode = cluster_mode;
>> + cluster->core_count = core_count;
>> + cluster->r5_cores = r5_cores;
>> +
>> + ret = zynqmp_r5_core_init(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + zynqmp_r5_print_dt_node_info(cluster);
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
>> +
>> + pr_info("Exit r5f subsystem driver\n");
>> +}
>> +
>> +/*
>> + * zynqmp_r5_remoteproc_probe()
>> + *
>> + * @pdev: domain platform device for R5 cluster
>> + *
>> + * called when driver is probed, for each R5 core specified in DT,
>> + * setup as needed to do remoteproc-related operations
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>> +{
>> + int ret;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct device *dev = &pdev->dev;
>> +
>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(cluster))
>> + return -ENOMEM;
> Function devm_kzalloc() does not return an code on error, just NULL. Please fix
> throughout the driver.
>
>> +
>> + cluster->dev = dev;
>> +
>> + ret = devm_of_platform_populate(dev);
>> + if (ret) {
>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /* wire in so each core can be cleaned up at driver remove */
>> + platform_set_drvdata(pdev, cluster);
>> +
>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_cluster_init(cluster);
>> + if (ret) {
>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>> + return ret;
>> + }
>> +
>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> Please remove this.
>
> I am out of time for today and will continue on Monday.
>
> Thanks,
> Mathieu
>
>> + return 0;
>> +}
>> +
>> +/* Match table for OF platform binding */
>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>> + { .compatible = "xlnx,zynqmp-r5fss", },
>> + { /* end of list */ },
>> +};
>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>> +
>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>> + .probe = zynqmp_r5_remoteproc_probe,
>> + .driver = {
>> + .name = "zynqmp_r5_remoteproc",
>> + .of_match_table = zynqmp_r5_remoteproc_match,
>> + },
>> +};
>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>> +
>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>> +MODULE_AUTHOR("Xilinx Inc.");
>> +MODULE_LICENSE("GPL v2");
>> --
>> 2.25.1
>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-13 8:10 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-13 8:10 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
Hi Mathieu,
Thanks for reviews. Please find my comments below.
On 12/4/21 12:25 AM, Mathieu Poirier wrote:
> Hi Tanmay,
>
> On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
>> This driver enables r5f dual core Real time Processing Unit subsystem
>> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
>> (cluster) can be configured in different modes e.g. split mode in which
>> two r5f cores work independent of each other and lock-step mode in which
>> both r5f cores execute same code clock-for-clock and notify if the
>> result is different.
>>
>> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
>> Platform Management Unit that handles the R5 configuration, memory access
>> and R5 lifecycle management. The interface to this manager is done in this
>> driver via zynqmp_pm_* function calls.
>>
>> Signed-off-by: Ben Levinsky<ben.levinsky@xilinx.com>
>> Signed-off-by: Tanmay Shah<tanmay.shah@xilinx.com>
>> ---
>> drivers/remoteproc/Kconfig | 12 +
>> drivers/remoteproc/Makefile | 1 +
>> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
>> 3 files changed, 972 insertions(+)
>> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>>
>> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
>> index f30d00a3aabe..27f66910d8d3 100644
>> --- a/drivers/remoteproc/Kconfig
>> +++ b/drivers/remoteproc/Kconfig
>> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
>> It's safe to say N here if you're not interested in utilizing
>> a slave processor.
>>
>> +config XLNX_R5_REMOTEPROC
>> + tristate "Xilinx R5 remoteproc support"
>> + depends on PM && ARCH_ZYNQMP
>> + depends on ZYNQMP_FIRMWARE
>> + select RPMSG_VIRTIO
>> + select ZYNQMP_IPI_MBOX
>> + help
>> + Say y or m here to support Xilinx R5 remote processors via the remote
>> + processor framework.
>> +
>> + It's safe to say N if not interested in using RPU r5f cores.
>> +
>> endif # REMOTEPROC
>>
>> endmenu
>> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
>> index bb26c9e4ef9c..334a8bed4c14 100644
>> --- a/drivers/remoteproc/Makefile
>> +++ b/drivers/remoteproc/Makefile
>> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
>> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
>> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
>> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
>> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
>> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> new file mode 100644
>> index 000000000000..c2167fd3869d
>> --- /dev/null
>> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> @@ -0,0 +1,959 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ZynqMP R5 Remote Processor driver
>> + *
>> + */
>> +
>> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
>> +#include <linux/firmware/xlnx-zynqmp.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/kernel.h>
>> +#include <linux/list.h>
>> +#include <linux/mailbox_client.h>
>> +#include <linux/mailbox/zynqmp-ipi-message.h>
>> +#include <linux/module.h>
>> +#include <linux/of_address.h>
>> +#include <linux/of_platform.h>
>> +#include <linux/of_reserved_mem.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/remoteproc.h>
>> +#include <linux/skbuff.h>
>> +#include <linux/sysfs.h>
>> +
>> +#include "remoteproc_internal.h"
>> +
>> +/* settings for RPU cluster mode */
>> +enum zynqmp_r5_cluster_mode {
>> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
>> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
>> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
> Please use C style comments, i.e /*...*/
>
>> +};
>> +
>> +/**
>> + * struct mem_bank_data - Memory Bank description
>> + *
>> + * @addr: Start address of memory bank
>> + * @size: Size of Memory bank
>> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
>> + * @bank_name: name of the bank for remoteproc framework
>> + */
>> +struct mem_bank_data {
>> + phys_addr_t addr;
>> + size_t size;
>> + enum pm_node_id pm_domain_id;
>> + char *bank_name;
>> +};
>> +
>> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
>> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
>> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
>> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
>> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
>> +};
> Bizarre - more comments on this below...
I know ideally it should have been in yaml bindings and we should parse
it from device-tree. Please find my explanation below for this approach.
>> +
>> +/**
>> + * struct zynqmp_r5_core - ZynqMP R5 core structure
>> + *
>> + * @dev: device of RPU instance
>> + * @np: device node of RPU instance
>> + * @tcm_bank_count: number TCM banks accessible to this RPU
>> + * @tcm_banks: array of each TCM bank data
>> + * @res_mem_count: number of Reserved Memory regions per core
>> + * @res_mem: array of reserved memory regions
>> + * @rproc: rproc handle
>> + * @pm_domain_id: RPU CPU power domain id
>> + */
>> +struct zynqmp_r5_core {
>> + struct device *dev;
>> + struct device_node *np;
>> + int tcm_bank_count;
>> + struct mem_bank_data *tcm_banks;
>> + int res_mem_count;
>> + struct reserved_mem *res_mem;
>> + struct rproc *rproc;
>> + enum pm_node_id pm_domain_id;
>> +};
>> +
>> +/**
>> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
>> + *
>> + * @dev: r5f subsystem cluster device node
>> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
>> + * @core_count: number of r5 cores used for this cluster mode
>> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
>> + */
>> +struct zynqmp_r5_cluster {
>> + struct device *dev;
>> + enum zynqmp_r5_cluster_mode mode;
>> + int core_count;
>> + struct zynqmp_r5_core *r5_cores;
>> +};
>> +
>> +/*
>> + * zynqmp_r5_set_mode - set RPU operation mode
>> + *
>> + * set RPU operation mode
>> + *
>> + * Return: 0 for success, negative value for failure
>> + */
>> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
>> + enum zynqmp_r5_cluster_mode rpu_mode)
>> +{
>> + enum rpu_tcm_comb tcm_mode;
>> + int ret, reg_val;
>> +
>> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
>> +
>> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
>> + if (ret < 0) {
>> + pr_err("failed to set RPU mode\n");
>> + return ret;
>> + }
>> +
>> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
>> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
>> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
>> + if (ret < 0)
>> + pr_err("failed to configure TCM\n");
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_start
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Start R5 Core from designated boot address.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + enum rpu_boot_mem bootmem;
>> + int ret;
>> +
>> + if (!r5_core) {
>> + pr_err("can't get r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
>> +
>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>> +
>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>> + if (ret)
>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_stop
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Power down R5 Core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + int ret;
>> +
>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret)
>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap memory-region carveout
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + iounmap((void __iomem *)mem->va);
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_mem_regions
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Construct rproc mem carveouts from carveout provided in
>> + * memory-region property
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_mem_regions(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct rproc_mem_entry *mem;
>> + struct reserved_mem *rmem;
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> +
>> + r5_core = rproc->priv;
>> + dev = r5_core->dev;
>> +
>> + /* Register associated reserved memory regions */
>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>> + rmem = &r5_core->res_mem[i];
>> + mem = rproc_mem_entry_init(dev, NULL,
>> + (dma_addr_t)rmem->base,
>> + rmem->size, rmem->base,
>> + zynqmp_r5_rproc_mem_map,
>> + zynqmp_r5_rproc_mem_unmap,
>> + rmem->name);
>> + if (IS_ERR_OR_NULL(mem))
>> + return -ENOMEM;
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap TCM banks when powering down R5 core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>> +{
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> + enum pm_node_id pm_domain_id;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + pr_err("r5 core is not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + iounmap((void __iomem *)mem->va);
>> +
>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + if (zynqmp_pm_release_node(pm_domain_id))
>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * tcm_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to initialize the va and da fields of
>> + *
>> + * Given TCM bank entry, this callback will set device address for R5
>> + * running on TCM and also setup virtual address for TCM bank
>> + * remoteproc carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + /* Update memory entry va */
>> + mem->va = (void *)va;
>> +
>> + /* clear TCMs */
>> + memset_io(va, 0, mem->len);
>> +
>> + /*
>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>> + * while on the Linux side they are at 0xffexxxxx.
>> + *
>> + * Zero out the high 12 bits of the address. This will give
>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>> + */
>> + mem->da &= 0x000fffff;
>> +
>> + /*
>> + * TCM Banks 1A and 1B still have to be translated.
>> + *
>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>> + * 0xffeb0000) and convert to the expected relative addresses
>> + * (0x0 and 0x20000).
>> + */
>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>> + mem->da -= 0x90000;
>> +
>> + /* if translated TCM bank address is not valid report error */
>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>> + return -EINVAL;
>> + }
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; i < num_banks; i++) {
>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>> + bank_name = r5_core->tcm_banks[i].bank_name;
>> + bank_size = r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + /* Turn off all TCM banks turned on before */
>> + do {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x, %x\n",
>> + (u32)pm_domain_id, ret);
>> + } while (i--);
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>> + bank_name = r5_core->tcm_banks[0].bank_name;
>> + for (i = 0; i < num_banks; i++) {
>> + bank_size += r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> + }
>> +
>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + for (i = 0; i < num_banks; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x ret: %x\n",
>> + (u32)pm_domain_id, ret);
>> + }
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_tcm_banks()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Given R5 node in remoteproc instance
>> + * allocate remoteproc carveout for TCM memory
>> + * needed for firmware to be loaded
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_parse_fw()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @fw: ptr to firmware to be loaded onto r5 core
>> + *
>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>> +{
>> + int ret;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + dev_err(&rproc->dev, "r5 core not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + dev = r5_core->dev;
>> +
>> + ret = add_tcm_banks(rproc);
>> + if (ret) {
>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = add_mem_regions(rproc);
>> + if (ret)
>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>> +
>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>> + if (ret == -EINVAL) {
>> + /*
>> + * resource table only required for IPC.
>> + * if not present, this is not necessarily an error;
>> + * for example, loading r5 hello world application
>> + * so simply inform user and keep going.
>> + */
>> + dev_info(&rproc->dev, "no resource table found.\n");
>> + ret = 0;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> +
>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int i, j, k;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + dev_dbg(cluster->dev, "Printing dt node info\n");
> Why mixing dev_dbg() and pr_debug()? Please pick one and stick with it.
>
> Even for debug level output this is very chatty - is all that information really
> needed when it is already available in /proc/device-tree?
I kept this function for debugging. However, looks like we can use
/proc/device-tree for same purpose, so I will remove this function.
Thanks for this suggestion.
>> +
>> + pr_debug("cluster mode = %d\n", cluster->mode);
>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("can't get r5_core\n");
>> + continue;
>> + }
>> +
>> + pr_debug("r5 core %d nodes\n", i);
>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>> + k, r5_core->tcm_banks[k].addr,
>> + r5_core->tcm_banks[k].size,
>> + r5_core->tcm_banks[k].pm_domain_id,
>> + r5_core->tcm_banks[k].bank_name);
>> + }
>> +
>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>> +
>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>> + j, r5_core->res_mem[j].base,
>> + r5_core->res_mem[j].size,
>> + r5_core->res_mem[j].name);
>> + }
>> + }
>> +}
>> +
>> +/**
>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>> + * this is called for each individual R5 core to
>> + * set up mailbox, Xilinx platform manager unique ID,
>> + * add to rproc core
>> + *
>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>> +{
>> + int ret;
>> + struct rproc *r5_rproc;
>> + struct device *dev;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* Set up DMA mask */
>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>> + if (ret)
>> + return ret;
>> +
>> + /* Allocate remoteproc instance */
>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>> + NULL, sizeof(struct zynqmp_r5_core));
>> + if (IS_ERR_OR_NULL(r5_rproc))
>> + return -ENOMEM;
>> +
>> + r5_rproc->auto_boot = false;
>> + r5_rproc->priv = r5_core;
>> +
>> + /* Add R5 remoteproc */
>> + ret = devm_rproc_add(dev, r5_rproc);
>> + if (ret) {
>> + pr_err("failed to add r5 remoteproc\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int tcm_bank_count, tcm_node;
>> + int i = 0, j;
>> + struct zynqmp_r5_core *r5_core;
>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>> + struct device *dev = cluster->dev;
>> +
>> + /* ToDo: Use predefined TCM address space values from driver until
>> + * system-dt spec is not final fot TCM
>> + */
> Multilined comment should be as follow:
>
> /*
> * ToDo: Use predefined TCM address space values from driver until
> * system-dt spec is not final fot TCM
> */
>
> s/"final fot TCM"/"final for TCM"
>
> Any reason this can't be done with "reg" properties like TI did for K3? It
> would be nice to have TCMs included in the yaml file example.
>
R5 TCM on Xilinx platforms contains separate power-domains than RPU
power-domains. So, I believe they can be operated separately than RPU.
That is why I chose not to include as reg property. I believe we will
include TCM as sram property in future.
Now, explanation about why TCM is hardcode in driver:
When I started developing driver, system-dt spec was in progress, to
avoid extra maintenance effort I defined TCM information in driver
instead of defining them in YAML.
I agree with idea that there is no need to maintain two separate
bindings and until system-dt specs are maturing we can hardcode TCM
related information driver. This was discussed previously here:
https://lore.kernel.org/all/CAL_JsqLGo380SRYska+xGgJhgF8NCRvY56ewafvSCU6c-LmhZw@mail.gmail.com/
Patchwork link for the same:
https://patchwork.kernel.org/project/linux-remoteproc/patch/1587749770-15082-5-git-send-email-ben.levinsky@xilinx.com/#23414963
Also, I will sync with system-dt team about its current status and keep
updating about system-dt specs.
Please let me know your opinion / suggestion for any alternative approach.
I agree with rest of the comments in this thread, and I will address
them all in next revision.
>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>> +
>> + /* count per core tcm banks */
>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>> +
>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>> + * 128k. Assign TCM banks to each core accordingly
>> + */
>> + tcm_node = 0;
>> + for (j = 0; j < cluster->core_count; j++) {
>> + r5_core = &cluster->r5_cores[j];
>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>> + tcm_bank_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>> + return -ENOMEM;
>> +
>> + for (i = 0; i < tcm_bank_count; i++) {
>> + /* Use pre-defined TCM reg values.
>> + * Eventually this should be replaced by values
>> + * parsed from dts.
>> + */
>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>> + tcm_node++;
>> + }
>> +
>> + r5_core->tcm_bank_count = tcm_bank_count;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>> +{
>> + int res_mem_count, i;
>> + struct device *dev;
>> + struct device_node *np, *rmem_np;
>> + struct reserved_mem *rmem;
>> +
>> + dev = r5_core->dev;
>> +
>> + np = r5_core->np;
>> + if (IS_ERR_OR_NULL(np)) {
>> + pr_err("invalid device node of r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>> + sizeof(phandle));
>> + if (res_mem_count <= 0) {
>> + dev_warn(dev, "failed to get memory-region property %d\n",
>> + res_mem_count);
>> + return -EINVAL;
>> + }
>> +
>> + r5_core->res_mem = devm_kzalloc(dev,
>> + res_mem_count * sizeof(struct reserved_mem),
>> + GFP_KERNEL);
>> + if (!r5_core->res_mem) {
>> + dev_err(dev, "failed to allocate mem region memory\n");
>> + return -ENOMEM;
>> + }
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int ret, i;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev = cluster->dev;
>> +
>> + ret = zynqmp_r5_get_tcm_node(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("invalid r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>> + if (ret)
>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>> +
>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>> + 1, &r5_core->pm_domain_id);
>> + if (ret) {
>> + dev_err(dev, "failed to get power-domains property\n");
>> + return ret;
>> + }
>> +
>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>> + if (ret) {
>> + dev_err(dev, "failed to init r5 core %d\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + struct device *dev = cluster->dev;
>> + struct device_node *dev_node = dev_of_node(dev);
>> + struct device_node *child;
>> + struct platform_device *child_pdev;
>> + int core_count = 0, ret, i;
>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>> + struct zynqmp_r5_core *r5_cores;
>> +
>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>> +
>> + /* on success returns 0, if not defined then returns -EINVAL,
>> + * In that case, default is LOCKSTEP mode
>> + */
>> + if (ret != -EINVAL && ret != 0) {
>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster_mode == SINGLE_CPU_MODE) {
>> + dev_err(dev, "driver does not support single cpu mode\n");
>> + return -EINVAL;
>> + } else if ((cluster_mode != SPLIT_MODE &&
>> + cluster_mode != LOCKSTEP_MODE)) {
>> + dev_err(dev, "Invalid cluster mode\n");
>> + return -EINVAL;
>> + }
>> +
>> + core_count = of_get_available_child_count(dev_node);
>> + if (core_count <= 0) {
>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>> + return -EINVAL;
>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>> + return -EINVAL;
>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>> + dev_warn(dev, "Only r5 core0 will be used\n");
>> + core_count = 1;
>> + }
>> +
>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>> + core_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_cores)) {
>> + dev_err(dev, "can't allocate memory for cores\n");
>> + return -ENOMEM;
>> + }
>> +
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
>> + return -ENODEV;
>> +
>> + r5_cores[i].dev = &child_pdev->dev;
>> + if (!r5_cores[i].dev) {
>> + pr_err("can't get device for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>> + if (!r5_cores[i].np) {
>> + pr_err("can't get device node for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + i++;
>> + if (i == core_count)
>> + break;
>> + }
>> +
>> + cluster->mode = cluster_mode;
>> + cluster->core_count = core_count;
>> + cluster->r5_cores = r5_cores;
>> +
>> + ret = zynqmp_r5_core_init(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + zynqmp_r5_print_dt_node_info(cluster);
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
>> +
>> + pr_info("Exit r5f subsystem driver\n");
>> +}
>> +
>> +/*
>> + * zynqmp_r5_remoteproc_probe()
>> + *
>> + * @pdev: domain platform device for R5 cluster
>> + *
>> + * called when driver is probed, for each R5 core specified in DT,
>> + * setup as needed to do remoteproc-related operations
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>> +{
>> + int ret;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct device *dev = &pdev->dev;
>> +
>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(cluster))
>> + return -ENOMEM;
> Function devm_kzalloc() does not return an code on error, just NULL. Please fix
> throughout the driver.
>
>> +
>> + cluster->dev = dev;
>> +
>> + ret = devm_of_platform_populate(dev);
>> + if (ret) {
>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /* wire in so each core can be cleaned up at driver remove */
>> + platform_set_drvdata(pdev, cluster);
>> +
>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_cluster_init(cluster);
>> + if (ret) {
>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>> + return ret;
>> + }
>> +
>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> Please remove this.
>
> I am out of time for today and will continue on Monday.
>
> Thanks,
> Mathieu
>
>> + return 0;
>> +}
>> +
>> +/* Match table for OF platform binding */
>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>> + { .compatible = "xlnx,zynqmp-r5fss", },
>> + { /* end of list */ },
>> +};
>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>> +
>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>> + .probe = zynqmp_r5_remoteproc_probe,
>> + .driver = {
>> + .name = "zynqmp_r5_remoteproc",
>> + .of_match_table = zynqmp_r5_remoteproc_match,
>> + },
>> +};
>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>> +
>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>> +MODULE_AUTHOR("Xilinx Inc.");
>> +MODULE_LICENSE("GPL v2");
>> --
>> 2.25.1
>>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-11-23 6:20 ` Tanmay Shah
@ 2021-12-13 11:08 ` Lars-Peter Clausen
-1 siblings, 0 replies; 56+ messages in thread
From: Lars-Peter Clausen @ 2021-12-13 11:08 UTC (permalink / raw)
To: Tanmay Shah, Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
linux-remoteproc, linux-kernel, devicetree, linux-arm-kernel
On 11/23/21 7:20 AM, Tanmay Shah wrote:
> [...]
> +/*
> + * zynqmp_r5_rproc_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
Since you want normal memory and not IO memory a better choice might be
memremap() with MEMREMAP_WC. Internally memremap() will call
ioremap_wc(), but this will make the intention clear and you do not have
to deal with the __iomem type cast.
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + mem->va = (void *)va;
> +
> + return 0;
> +}
> [...]
>
> +static int add_tcm_banks(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + parent_pdev = to_platform_device(dev->parent);
> + if (!parent_pdev) {
> + dev_err(dev, "parent platform dev unavailable\n");
> + return -ENODEV;
> + }
> +
> + cluster = platform_get_drvdata(parent_pdev);
You could just use dev_get_drvdata() without having to cast back to the
platform_device first.
> + if (!cluster) {
> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
> + return -EINVAL;
> + }
> +
> + if (cluster->mode == SPLIT_MODE)
> + return add_tcm_carveout_split_mode(rproc);
> + else if (cluster->mode == LOCKSTEP_MODE)
> + return add_tcm_carveout_lockstep_mode(rproc);
> +
> + dev_err(cluster->dev, "invalid cluster mode\n");
> + return -EINVAL;
> +}
> +
> [...]
> +
> +static struct rproc_ops zynqmp_r5_rproc_ops = {
const
> + .start = zynqmp_r5_rproc_start,
> + .stop = zynqmp_r5_rproc_stop,
> + .load = rproc_elf_load_segments,
> + .parse_fw = zynqmp_r5_parse_fw,
> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> + .sanity_check = rproc_elf_sanity_check,
> + .get_boot_addr = rproc_elf_get_boot_addr,
> +};
> [....]
> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> +{
> [...]
> +
> + for (i = 0; i < res_mem_count; i++) {
> + rmem_np = of_parse_phandle(np, "memory-region", i);
> + if (!rmem_np)
> + return -EINVAL;
> +
> + rmem = of_reserved_mem_lookup(rmem_np);
> + if (!rmem) {
> + of_node_put(rmem_np);
> + return -EINVAL;
> + }
> +
> + memcpy(&r5_core->res_mem[i], rmem,
> + sizeof(struct reserved_mem));
r5_core->res_mem[i] = *mem;
This will give you proper type checking and is also a bit shorter.
> + of_node_put(rmem_np);
> + }
> +
> + r5_core->res_mem_count = res_mem_count;
> +
> + return 0;
> +}
> [...]
> +
> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> +{
> [...]
> +
>
> + i = 0;
> + for_each_available_child_of_node(dev_node, child) {
> + child_pdev = of_find_device_by_node(child);
> + if (!child_pdev)
A return or a break in a for_each_available_child_of_node() will leak
the reference to the child node.
> [...]
> + }
> +
> [...]
> +
> + return 0;
> +}
> +
> +static void zynqmp_r5_cluster_exit(void *data)
> +{
> + struct platform_device *pdev = (struct platform_device *)data;
> +
> + platform_set_drvdata(pdev, NULL);
This is not needed. The device driver core will set drvdata to NULL when
the device is removed.
> +
> + pr_info("Exit r5f subsystem driver\n");
This is probably also not needed.
> +}
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-13 11:08 ` Lars-Peter Clausen
0 siblings, 0 replies; 56+ messages in thread
From: Lars-Peter Clausen @ 2021-12-13 11:08 UTC (permalink / raw)
To: Tanmay Shah, Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
linux-remoteproc, linux-kernel, devicetree, linux-arm-kernel
On 11/23/21 7:20 AM, Tanmay Shah wrote:
> [...]
> +/*
> + * zynqmp_r5_rproc_mem_map
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
> + struct rproc_mem_entry *mem)
> +{
> + void __iomem *va;
> +
> + va = ioremap_wc(mem->dma, mem->len);
Since you want normal memory and not IO memory a better choice might be
memremap() with MEMREMAP_WC. Internally memremap() will call
ioremap_wc(), but this will make the intention clear and you do not have
to deal with the __iomem type cast.
> + if (IS_ERR_OR_NULL(va))
> + return -ENOMEM;
> +
> + mem->va = (void *)va;
> +
> + return 0;
> +}
> [...]
>
> +static int add_tcm_banks(struct rproc *rproc)
> +{
> + struct device *dev;
> + struct platform_device *parent_pdev;
> + struct zynqmp_r5_cluster *cluster;
> + struct zynqmp_r5_core *r5_core;
> +
> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
> + if (!r5_core)
> + return -EINVAL;
> +
> + dev = r5_core->dev;
> + if (!dev) {
> + pr_err("r5 core device unavailable\n");
> + return -ENODEV;
> + }
> +
> + parent_pdev = to_platform_device(dev->parent);
> + if (!parent_pdev) {
> + dev_err(dev, "parent platform dev unavailable\n");
> + return -ENODEV;
> + }
> +
> + cluster = platform_get_drvdata(parent_pdev);
You could just use dev_get_drvdata() without having to cast back to the
platform_device first.
> + if (!cluster) {
> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
> + return -EINVAL;
> + }
> +
> + if (cluster->mode == SPLIT_MODE)
> + return add_tcm_carveout_split_mode(rproc);
> + else if (cluster->mode == LOCKSTEP_MODE)
> + return add_tcm_carveout_lockstep_mode(rproc);
> +
> + dev_err(cluster->dev, "invalid cluster mode\n");
> + return -EINVAL;
> +}
> +
> [...]
> +
> +static struct rproc_ops zynqmp_r5_rproc_ops = {
const
> + .start = zynqmp_r5_rproc_start,
> + .stop = zynqmp_r5_rproc_stop,
> + .load = rproc_elf_load_segments,
> + .parse_fw = zynqmp_r5_parse_fw,
> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> + .sanity_check = rproc_elf_sanity_check,
> + .get_boot_addr = rproc_elf_get_boot_addr,
> +};
> [....]
> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> +{
> [...]
> +
> + for (i = 0; i < res_mem_count; i++) {
> + rmem_np = of_parse_phandle(np, "memory-region", i);
> + if (!rmem_np)
> + return -EINVAL;
> +
> + rmem = of_reserved_mem_lookup(rmem_np);
> + if (!rmem) {
> + of_node_put(rmem_np);
> + return -EINVAL;
> + }
> +
> + memcpy(&r5_core->res_mem[i], rmem,
> + sizeof(struct reserved_mem));
r5_core->res_mem[i] = *mem;
This will give you proper type checking and is also a bit shorter.
> + of_node_put(rmem_np);
> + }
> +
> + r5_core->res_mem_count = res_mem_count;
> +
> + return 0;
> +}
> [...]
> +
> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> +{
> [...]
> +
>
> + i = 0;
> + for_each_available_child_of_node(dev_node, child) {
> + child_pdev = of_find_device_by_node(child);
> + if (!child_pdev)
A return or a break in a for_each_available_child_of_node() will leak
the reference to the child node.
> [...]
> + }
> +
> [...]
> +
> + return 0;
> +}
> +
> +static void zynqmp_r5_cluster_exit(void *data)
> +{
> + struct platform_device *pdev = (struct platform_device *)data;
> +
> + platform_set_drvdata(pdev, NULL);
This is not needed. The device driver core will set drvdata to NULL when
the device is removed.
> +
> + pr_info("Exit r5f subsystem driver\n");
This is probably also not needed.
> +}
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-12-13 8:10 ` Tanmay Shah
@ 2021-12-13 18:39 ` Mathieu Poirier
-1 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-12-13 18:39 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
[...]
> > > +}
> > > +
> > > +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + int tcm_bank_count, tcm_node;
> > > + int i = 0, j;
> > > + struct zynqmp_r5_core *r5_core;
> > > + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> > > + struct device *dev = cluster->dev;
> > > +
> > > + /* ToDo: Use predefined TCM address space values from driver until
> > > + * system-dt spec is not final fot TCM
> > > + */
> > Multilined comment should be as follow:
> >
> > /*
> > * ToDo: Use predefined TCM address space values from driver until
> > * system-dt spec is not final fot TCM
> > */
> >
> > s/"final fot TCM"/"final for TCM"
> >
> > Any reason this can't be done with "reg" properties like TI did for K3? It
> > would be nice to have TCMs included in the yaml file example.
> >
> R5 TCM on Xilinx platforms contains separate power-domains than RPU
> power-domains. So, I believe they can be operated separately than RPU. That
> is why I chose not to include as reg property. I believe we will include TCM
> as sram property in future.
>
> Now, explanation about why TCM is hardcode in driver:
>
> When I started developing driver, system-dt spec was in progress, to avoid
> extra maintenance effort I defined TCM information in driver instead of
> defining them in YAML.
> I agree with idea that there is no need to maintain two separate bindings
> and until system-dt specs are maturing we can hardcode TCM related
> information driver. This was discussed previously here: https://lore.kernel.org/all/CAL_JsqLGo380SRYska+xGgJhgF8NCRvY56ewafvSCU6c-LmhZw@mail.gmail.com/
>
Thanks for the link.
> Patchwork link for the same: https://patchwork.kernel.org/project/linux-remoteproc/patch/1587749770-15082-5-git-send-email-ben.levinsky@xilinx.com/#23414963
>
> Also, I will sync with system-dt team about its current status and keep
> updating about system-dt specs.
>
> Please let me know your opinion / suggestion for any alternative approach.
>
There is no point in burdening Rob any further than he already is - for the time
being the current approach will work.
>
> I agree with rest of the comments in this thread, and I will address them
> all in next revision.
>
> > > + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> > > +
> > > + /* count per core tcm banks */
> > > + tcm_bank_count = tcm_bank_count / cluster->core_count;
> > > +
> > > + /* r5 core 0 will use all of TCM banks in lockstep mode.
> > > + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> > > + * 128k. Assign TCM banks to each core accordingly
> > > + */
> > > + tcm_node = 0;
> > > + for (j = 0; j < cluster->core_count; j++) {
> > > + r5_core = &cluster->r5_cores[j];
> > > + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> > > + tcm_bank_count, GFP_KERNEL);
> > > + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> > > + return -ENOMEM;
> > > +
> > > + for (i = 0; i < tcm_bank_count; i++) {
> > > + /* Use pre-defined TCM reg values.
> > > + * Eventually this should be replaced by values
> > > + * parsed from dts.
> > > + */
> > > + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> > > + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> > > + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> > > + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> > > + tcm_node++;
> > > + }
> > > +
> > > + r5_core->tcm_bank_count = tcm_bank_count;
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> > > +{
> > > + int res_mem_count, i;
> > > + struct device *dev;
> > > + struct device_node *np, *rmem_np;
> > > + struct reserved_mem *rmem;
> > > +
> > > + dev = r5_core->dev;
> > > +
> > > + np = r5_core->np;
> > > + if (IS_ERR_OR_NULL(np)) {
> > > + pr_err("invalid device node of r5 core\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> > > + sizeof(phandle));
> > > + if (res_mem_count <= 0) {
> > > + dev_warn(dev, "failed to get memory-region property %d\n",
> > > + res_mem_count);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + r5_core->res_mem = devm_kzalloc(dev,
> > > + res_mem_count * sizeof(struct reserved_mem),
> > > + GFP_KERNEL);
> > > + if (!r5_core->res_mem) {
> > > + dev_err(dev, "failed to allocate mem region memory\n");
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + for (i = 0; i < res_mem_count; i++) {
> > > + rmem_np = of_parse_phandle(np, "memory-region", i);
> > > + if (!rmem_np)
> > > + return -EINVAL;
> > > +
> > > + rmem = of_reserved_mem_lookup(rmem_np);
> > > + if (!rmem) {
> > > + of_node_put(rmem_np);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + memcpy(&r5_core->res_mem[i], rmem,
> > > + sizeof(struct reserved_mem));
> > > + of_node_put(rmem_np);
> > > + }
> > > +
> > > + r5_core->res_mem_count = res_mem_count;
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + int ret, i;
> > > + struct zynqmp_r5_core *r5_core;
> > > + struct device *dev = cluster->dev;
> > > +
> > > + ret = zynqmp_r5_get_tcm_node(cluster);
> > > + if (ret < 0) {
> > > + dev_err(dev, "can't get tcm node, err %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + for (i = 0; i < cluster->core_count; i++) {
> > > + r5_core = &cluster->r5_cores[i];
> > > + if (!r5_core) {
> > > + pr_err("invalid r5 core\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + ret = zynqmp_r5_get_mem_region_node(r5_core);
> > > + if (ret)
> > > + dev_warn(dev, "memory-region prop failed %d\n", ret);
> > > +
> > > + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> > > + 1, &r5_core->pm_domain_id);
> > > + if (ret) {
> > > + dev_err(dev, "failed to get power-domains property\n");
> > > + return ret;
> > > + }
> > > +
> > > + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> > > + if (ret)
> > > + return ret;
> > > +
> > > + ret = zynqmp_r5_add_rproc_core(r5_core);
> > > + if (ret) {
> > > + dev_err(dev, "failed to init r5 core %d\n", i);
> > > + return ret;
> > > + }
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + struct device *dev = cluster->dev;
> > > + struct device_node *dev_node = dev_of_node(dev);
> > > + struct device_node *child;
> > > + struct platform_device *child_pdev;
> > > + int core_count = 0, ret, i;
> > > + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> > > + struct zynqmp_r5_core *r5_cores;
> > > +
> > > + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> > > +
> > > + /* on success returns 0, if not defined then returns -EINVAL,
> > > + * In that case, default is LOCKSTEP mode
> > > + */
> > > + if (ret != -EINVAL && ret != 0) {
> > > + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + if (cluster_mode == SINGLE_CPU_MODE) {
> > > + dev_err(dev, "driver does not support single cpu mode\n");
> > > + return -EINVAL;
> > > + } else if ((cluster_mode != SPLIT_MODE &&
> > > + cluster_mode != LOCKSTEP_MODE)) {
> > > + dev_err(dev, "Invalid cluster mode\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + core_count = of_get_available_child_count(dev_node);
> > > + if (core_count <= 0) {
> > > + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> > > + return -EINVAL;
> > > + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> > > + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> > > + return -EINVAL;
> > > + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> > > + dev_warn(dev, "Only r5 core0 will be used\n");
> > > + core_count = 1;
> > > + }
> > > +
> > > + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> > > + core_count, GFP_KERNEL);
> > > + if (IS_ERR_OR_NULL(r5_cores)) {
> > > + dev_err(dev, "can't allocate memory for cores\n");
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + i = 0;
> > > + for_each_available_child_of_node(dev_node, child) {
> > > + child_pdev = of_find_device_by_node(child);
> > > + if (!child_pdev)
> > > + return -ENODEV;
> > > +
> > > + r5_cores[i].dev = &child_pdev->dev;
> > > + if (!r5_cores[i].dev) {
> > > + pr_err("can't get device for r5 core %d\n", i);
> > > + return -ENODEV;
> > > + }
> > > +
> > > + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> > > + if (!r5_cores[i].np) {
> > > + pr_err("can't get device node for r5 core %d\n", i);
> > > + return -ENODEV;
> > > + }
> > > +
> > > + i++;
> > > + if (i == core_count)
> > > + break;
> > > + }
> > > +
> > > + cluster->mode = cluster_mode;
> > > + cluster->core_count = core_count;
> > > + cluster->r5_cores = r5_cores;
> > > +
> > > + ret = zynqmp_r5_core_init(cluster);
> > > + if (ret < 0) {
> > > + dev_err(dev, "failed to init r5 core err %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + zynqmp_r5_print_dt_node_info(cluster);
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static void zynqmp_r5_cluster_exit(void *data)
> > > +{
> > > + struct platform_device *pdev = (struct platform_device *)data;
> > > +
> > > + platform_set_drvdata(pdev, NULL);
> > > +
> > > + pr_info("Exit r5f subsystem driver\n");
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_remoteproc_probe()
> > > + *
> > > + * @pdev: domain platform device for R5 cluster
> > > + *
> > > + * called when driver is probed, for each R5 core specified in DT,
> > > + * setup as needed to do remoteproc-related operations
> > > + *
> > > + * Return: 0 for success, negative value for failure.
> > > + */
> > > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > > +{
> > > + int ret;
> > > + struct zynqmp_r5_cluster *cluster;
> > > + struct device *dev = &pdev->dev;
> > > +
> > > + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > > + if (IS_ERR_OR_NULL(cluster))
> > > + return -ENOMEM;
> > Function devm_kzalloc() does not return an code on error, just NULL. Please fix
> > throughout the driver.
> >
> > > +
> > > + cluster->dev = dev;
> > > +
> > > + ret = devm_of_platform_populate(dev);
> > > + if (ret) {
> > > + dev_err(dev, "failed to populate platform dev %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + /* wire in so each core can be cleaned up at driver remove */
> > > + platform_set_drvdata(pdev, cluster);
> > > +
> > > + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> > > + if (ret)
> > > + return ret;
> > > +
> > > + ret = zynqmp_r5_cluster_init(cluster);
> > > + if (ret) {
> > > + dev_err(dev, "Invalid r5f subsystem device tree\n");
> > > + return ret;
> > > + }
> > > +
> > > + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> > Please remove this.
> >
> > I am out of time for today and will continue on Monday.
> >
> > Thanks,
> > Mathieu
> >
> > > + return 0;
> > > +}
> > > +
> > > +/* Match table for OF platform binding */
> > > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > > + { .compatible = "xlnx,zynqmp-r5fss", },
> > > + { /* end of list */ },
> > > +};
> > > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > > +
> > > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > > + .probe = zynqmp_r5_remoteproc_probe,
> > > + .driver = {
> > > + .name = "zynqmp_r5_remoteproc",
> > > + .of_match_table = zynqmp_r5_remoteproc_match,
> > > + },
> > > +};
> > > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > > +
> > > +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> > > +MODULE_AUTHOR("Xilinx Inc.");
> > > +MODULE_LICENSE("GPL v2");
> > > --
> > > 2.25.1
> > >
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-13 18:39 ` Mathieu Poirier
0 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2021-12-13 18:39 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
[...]
> > > +}
> > > +
> > > +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + int tcm_bank_count, tcm_node;
> > > + int i = 0, j;
> > > + struct zynqmp_r5_core *r5_core;
> > > + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> > > + struct device *dev = cluster->dev;
> > > +
> > > + /* ToDo: Use predefined TCM address space values from driver until
> > > + * system-dt spec is not final fot TCM
> > > + */
> > Multilined comment should be as follow:
> >
> > /*
> > * ToDo: Use predefined TCM address space values from driver until
> > * system-dt spec is not final fot TCM
> > */
> >
> > s/"final fot TCM"/"final for TCM"
> >
> > Any reason this can't be done with "reg" properties like TI did for K3? It
> > would be nice to have TCMs included in the yaml file example.
> >
> R5 TCM on Xilinx platforms contains separate power-domains than RPU
> power-domains. So, I believe they can be operated separately than RPU. That
> is why I chose not to include as reg property. I believe we will include TCM
> as sram property in future.
>
> Now, explanation about why TCM is hardcode in driver:
>
> When I started developing driver, system-dt spec was in progress, to avoid
> extra maintenance effort I defined TCM information in driver instead of
> defining them in YAML.
> I agree with idea that there is no need to maintain two separate bindings
> and until system-dt specs are maturing we can hardcode TCM related
> information driver. This was discussed previously here: https://lore.kernel.org/all/CAL_JsqLGo380SRYska+xGgJhgF8NCRvY56ewafvSCU6c-LmhZw@mail.gmail.com/
>
Thanks for the link.
> Patchwork link for the same: https://patchwork.kernel.org/project/linux-remoteproc/patch/1587749770-15082-5-git-send-email-ben.levinsky@xilinx.com/#23414963
>
> Also, I will sync with system-dt team about its current status and keep
> updating about system-dt specs.
>
> Please let me know your opinion / suggestion for any alternative approach.
>
There is no point in burdening Rob any further than he already is - for the time
being the current approach will work.
>
> I agree with rest of the comments in this thread, and I will address them
> all in next revision.
>
> > > + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> > > +
> > > + /* count per core tcm banks */
> > > + tcm_bank_count = tcm_bank_count / cluster->core_count;
> > > +
> > > + /* r5 core 0 will use all of TCM banks in lockstep mode.
> > > + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> > > + * 128k. Assign TCM banks to each core accordingly
> > > + */
> > > + tcm_node = 0;
> > > + for (j = 0; j < cluster->core_count; j++) {
> > > + r5_core = &cluster->r5_cores[j];
> > > + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> > > + tcm_bank_count, GFP_KERNEL);
> > > + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> > > + return -ENOMEM;
> > > +
> > > + for (i = 0; i < tcm_bank_count; i++) {
> > > + /* Use pre-defined TCM reg values.
> > > + * Eventually this should be replaced by values
> > > + * parsed from dts.
> > > + */
> > > + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> > > + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> > > + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> > > + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> > > + tcm_node++;
> > > + }
> > > +
> > > + r5_core->tcm_bank_count = tcm_bank_count;
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> > > +{
> > > + int res_mem_count, i;
> > > + struct device *dev;
> > > + struct device_node *np, *rmem_np;
> > > + struct reserved_mem *rmem;
> > > +
> > > + dev = r5_core->dev;
> > > +
> > > + np = r5_core->np;
> > > + if (IS_ERR_OR_NULL(np)) {
> > > + pr_err("invalid device node of r5 core\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> > > + sizeof(phandle));
> > > + if (res_mem_count <= 0) {
> > > + dev_warn(dev, "failed to get memory-region property %d\n",
> > > + res_mem_count);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + r5_core->res_mem = devm_kzalloc(dev,
> > > + res_mem_count * sizeof(struct reserved_mem),
> > > + GFP_KERNEL);
> > > + if (!r5_core->res_mem) {
> > > + dev_err(dev, "failed to allocate mem region memory\n");
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + for (i = 0; i < res_mem_count; i++) {
> > > + rmem_np = of_parse_phandle(np, "memory-region", i);
> > > + if (!rmem_np)
> > > + return -EINVAL;
> > > +
> > > + rmem = of_reserved_mem_lookup(rmem_np);
> > > + if (!rmem) {
> > > + of_node_put(rmem_np);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + memcpy(&r5_core->res_mem[i], rmem,
> > > + sizeof(struct reserved_mem));
> > > + of_node_put(rmem_np);
> > > + }
> > > +
> > > + r5_core->res_mem_count = res_mem_count;
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + int ret, i;
> > > + struct zynqmp_r5_core *r5_core;
> > > + struct device *dev = cluster->dev;
> > > +
> > > + ret = zynqmp_r5_get_tcm_node(cluster);
> > > + if (ret < 0) {
> > > + dev_err(dev, "can't get tcm node, err %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + for (i = 0; i < cluster->core_count; i++) {
> > > + r5_core = &cluster->r5_cores[i];
> > > + if (!r5_core) {
> > > + pr_err("invalid r5 core\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + ret = zynqmp_r5_get_mem_region_node(r5_core);
> > > + if (ret)
> > > + dev_warn(dev, "memory-region prop failed %d\n", ret);
> > > +
> > > + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> > > + 1, &r5_core->pm_domain_id);
> > > + if (ret) {
> > > + dev_err(dev, "failed to get power-domains property\n");
> > > + return ret;
> > > + }
> > > +
> > > + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> > > + if (ret)
> > > + return ret;
> > > +
> > > + ret = zynqmp_r5_add_rproc_core(r5_core);
> > > + if (ret) {
> > > + dev_err(dev, "failed to init r5 core %d\n", i);
> > > + return ret;
> > > + }
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + struct device *dev = cluster->dev;
> > > + struct device_node *dev_node = dev_of_node(dev);
> > > + struct device_node *child;
> > > + struct platform_device *child_pdev;
> > > + int core_count = 0, ret, i;
> > > + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> > > + struct zynqmp_r5_core *r5_cores;
> > > +
> > > + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> > > +
> > > + /* on success returns 0, if not defined then returns -EINVAL,
> > > + * In that case, default is LOCKSTEP mode
> > > + */
> > > + if (ret != -EINVAL && ret != 0) {
> > > + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + if (cluster_mode == SINGLE_CPU_MODE) {
> > > + dev_err(dev, "driver does not support single cpu mode\n");
> > > + return -EINVAL;
> > > + } else if ((cluster_mode != SPLIT_MODE &&
> > > + cluster_mode != LOCKSTEP_MODE)) {
> > > + dev_err(dev, "Invalid cluster mode\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + core_count = of_get_available_child_count(dev_node);
> > > + if (core_count <= 0) {
> > > + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> > > + return -EINVAL;
> > > + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> > > + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> > > + return -EINVAL;
> > > + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> > > + dev_warn(dev, "Only r5 core0 will be used\n");
> > > + core_count = 1;
> > > + }
> > > +
> > > + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> > > + core_count, GFP_KERNEL);
> > > + if (IS_ERR_OR_NULL(r5_cores)) {
> > > + dev_err(dev, "can't allocate memory for cores\n");
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + i = 0;
> > > + for_each_available_child_of_node(dev_node, child) {
> > > + child_pdev = of_find_device_by_node(child);
> > > + if (!child_pdev)
> > > + return -ENODEV;
> > > +
> > > + r5_cores[i].dev = &child_pdev->dev;
> > > + if (!r5_cores[i].dev) {
> > > + pr_err("can't get device for r5 core %d\n", i);
> > > + return -ENODEV;
> > > + }
> > > +
> > > + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> > > + if (!r5_cores[i].np) {
> > > + pr_err("can't get device node for r5 core %d\n", i);
> > > + return -ENODEV;
> > > + }
> > > +
> > > + i++;
> > > + if (i == core_count)
> > > + break;
> > > + }
> > > +
> > > + cluster->mode = cluster_mode;
> > > + cluster->core_count = core_count;
> > > + cluster->r5_cores = r5_cores;
> > > +
> > > + ret = zynqmp_r5_core_init(cluster);
> > > + if (ret < 0) {
> > > + dev_err(dev, "failed to init r5 core err %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + zynqmp_r5_print_dt_node_info(cluster);
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static void zynqmp_r5_cluster_exit(void *data)
> > > +{
> > > + struct platform_device *pdev = (struct platform_device *)data;
> > > +
> > > + platform_set_drvdata(pdev, NULL);
> > > +
> > > + pr_info("Exit r5f subsystem driver\n");
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_remoteproc_probe()
> > > + *
> > > + * @pdev: domain platform device for R5 cluster
> > > + *
> > > + * called when driver is probed, for each R5 core specified in DT,
> > > + * setup as needed to do remoteproc-related operations
> > > + *
> > > + * Return: 0 for success, negative value for failure.
> > > + */
> > > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > > +{
> > > + int ret;
> > > + struct zynqmp_r5_cluster *cluster;
> > > + struct device *dev = &pdev->dev;
> > > +
> > > + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > > + if (IS_ERR_OR_NULL(cluster))
> > > + return -ENOMEM;
> > Function devm_kzalloc() does not return an code on error, just NULL. Please fix
> > throughout the driver.
> >
> > > +
> > > + cluster->dev = dev;
> > > +
> > > + ret = devm_of_platform_populate(dev);
> > > + if (ret) {
> > > + dev_err(dev, "failed to populate platform dev %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + /* wire in so each core can be cleaned up at driver remove */
> > > + platform_set_drvdata(pdev, cluster);
> > > +
> > > + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> > > + if (ret)
> > > + return ret;
> > > +
> > > + ret = zynqmp_r5_cluster_init(cluster);
> > > + if (ret) {
> > > + dev_err(dev, "Invalid r5f subsystem device tree\n");
> > > + return ret;
> > > + }
> > > +
> > > + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> > Please remove this.
> >
> > I am out of time for today and will continue on Monday.
> >
> > Thanks,
> > Mathieu
> >
> > > + return 0;
> > > +}
> > > +
> > > +/* Match table for OF platform binding */
> > > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > > + { .compatible = "xlnx,zynqmp-r5fss", },
> > > + { /* end of list */ },
> > > +};
> > > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > > +
> > > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > > + .probe = zynqmp_r5_remoteproc_probe,
> > > + .driver = {
> > > + .name = "zynqmp_r5_remoteproc",
> > > + .of_match_table = zynqmp_r5_remoteproc_match,
> > > + },
> > > +};
> > > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > > +
> > > +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> > > +MODULE_AUTHOR("Xilinx Inc.");
> > > +MODULE_LICENSE("GPL v2");
> > > --
> > > 2.25.1
> > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-12-06 19:04 ` Mathieu Poirier
@ 2021-12-13 19:11 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-13 19:11 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
Thanks Mathieu for reviews.
[...]
>> + * zynqmp_r5_set_mode - set RPU operation mode
>> + *
>> + * set RPU operation mode
>> + *
>> + * Return: 0 for success, negative value for failure
>> + */
>> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
>> + enum zynqmp_r5_cluster_mode rpu_mode)
>> +{
>> + enum rpu_tcm_comb tcm_mode;
>> + int ret, reg_val;
>> +
>> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
> Here split and single-cpu mode are treated the same... Is this an omission or
> really how the HW works? A good dose of documentation is definitely needed.
In this patchset, driver probe is failing in zynqmp_r5_cluster_init
if single-cpu mode is set in device-tree with message that
single-cpu mode is not supported by driver. So, here it is only for
split mode.
Later when single-cpu mode is supported, this will be modified accordingly.
Sure adding comments sounds good idea. I will take care it in next revision.
Also, I agree with all the comments in this email. I will address them
in next revision.
If I have any question while development, I will ask before posting next
revision.
>
>> +
>> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
>> + if (ret < 0) {
>> + pr_err("failed to set RPU mode\n");
>> + return ret;
>> + }
>> +
>> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
>> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
>> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
>> + if (ret < 0)
>> + pr_err("failed to configure TCM\n");
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_start
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Start R5 Core from designated boot address.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + enum rpu_boot_mem bootmem;
>> + int ret;
>> +
>> + if (!r5_core) {
>> + pr_err("can't get r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
>> +
>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>> +
>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>> + if (ret)
>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_stop
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Power down R5 Core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + int ret;
>> +
>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret)
>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap memory-region carveout
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + iounmap((void __iomem *)mem->va);
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_mem_regions
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Construct rproc mem carveouts from carveout provided in
>> + * memory-region property
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_mem_regions(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct rproc_mem_entry *mem;
>> + struct reserved_mem *rmem;
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> +
>> + r5_core = rproc->priv;
>> + dev = r5_core->dev;
>> +
>> + /* Register associated reserved memory regions */
>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>> + rmem = &r5_core->res_mem[i];
>> + mem = rproc_mem_entry_init(dev, NULL,
>> + (dma_addr_t)rmem->base,
>> + rmem->size, rmem->base,
>> + zynqmp_r5_rproc_mem_map,
>> + zynqmp_r5_rproc_mem_unmap,
>> + rmem->name);
>> + if (IS_ERR_OR_NULL(mem))
>> + return -ENOMEM;
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap TCM banks when powering down R5 core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>> +{
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> + enum pm_node_id pm_domain_id;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + pr_err("r5 core is not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + iounmap((void __iomem *)mem->va);
>> +
>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + if (zynqmp_pm_release_node(pm_domain_id))
>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * tcm_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to initialize the va and da fields of
>> + *
>> + * Given TCM bank entry, this callback will set device address for R5
>> + * running on TCM and also setup virtual address for TCM bank
>> + * remoteproc carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + /* Update memory entry va */
>> + mem->va = (void *)va;
>> +
>> + /* clear TCMs */
>> + memset_io(va, 0, mem->len);
>> +
>> + /*
>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>> + * while on the Linux side they are at 0xffexxxxx.
>> + *
>> + * Zero out the high 12 bits of the address. This will give
>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>> + */
>> + mem->da &= 0x000fffff;
>> +
>> + /*
>> + * TCM Banks 1A and 1B still have to be translated.
>> + *
>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>> + * 0xffeb0000) and convert to the expected relative addresses
>> + * (0x0 and 0x20000).
>> + */
>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>> + mem->da -= 0x90000;
>> +
>> + /* if translated TCM bank address is not valid report error */
>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>> + return -EINVAL;
>> + }
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; i < num_banks; i++) {
>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>> + bank_name = r5_core->tcm_banks[i].bank_name;
>> + bank_size = r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + /* Turn off all TCM banks turned on before */
>> + do {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x, %x\n",
>> + (u32)pm_domain_id, ret);
>> + } while (i--);
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>> + bank_name = r5_core->tcm_banks[0].bank_name;
>> + for (i = 0; i < num_banks; i++) {
>> + bank_size += r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> + }
>> +
>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + for (i = 0; i < num_banks; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x ret: %x\n",
>> + (u32)pm_domain_id, ret);
>> + }
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_tcm_banks()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Given R5 node in remoteproc instance
>> + * allocate remoteproc carveout for TCM memory
>> + * needed for firmware to be loaded
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_parse_fw()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @fw: ptr to firmware to be loaded onto r5 core
>> + *
>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>> +{
>> + int ret;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + dev_err(&rproc->dev, "r5 core not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + dev = r5_core->dev;
>> +
>> + ret = add_tcm_banks(rproc);
>> + if (ret) {
>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = add_mem_regions(rproc);
>> + if (ret)
>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>> +
>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>> + if (ret == -EINVAL) {
>> + /*
>> + * resource table only required for IPC.
>> + * if not present, this is not necessarily an error;
>> + * for example, loading r5 hello world application
>> + * so simply inform user and keep going.
>> + */
>> + dev_info(&rproc->dev, "no resource table found.\n");
>> + ret = 0;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> +
>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int i, j, k;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + dev_dbg(cluster->dev, "Printing dt node info\n");
>> +
>> + pr_debug("cluster mode = %d\n", cluster->mode);
>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("can't get r5_core\n");
>> + continue;
>> + }
>> +
>> + pr_debug("r5 core %d nodes\n", i);
>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>> + k, r5_core->tcm_banks[k].addr,
>> + r5_core->tcm_banks[k].size,
>> + r5_core->tcm_banks[k].pm_domain_id,
>> + r5_core->tcm_banks[k].bank_name);
>> + }
>> +
>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>> +
>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>> + j, r5_core->res_mem[j].base,
>> + r5_core->res_mem[j].size,
>> + r5_core->res_mem[j].name);
>> + }
>> + }
>> +}
>> +
>> +/**
>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>> + * this is called for each individual R5 core to
>> + * set up mailbox, Xilinx platform manager unique ID,
>> + * add to rproc core
>> + *
> Please get rid of the indentations. That way it conforms to the rest of the
> style in this file.
>
>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>> +{
>> + int ret;
>> + struct rproc *r5_rproc;
>> + struct device *dev;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* Set up DMA mask */
>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>> + if (ret)
>> + return ret;
>> +
>> + /* Allocate remoteproc instance */
>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>> + NULL, sizeof(struct zynqmp_r5_core));
>> + if (IS_ERR_OR_NULL(r5_rproc))
>> + return -ENOMEM;
> devm_rproc_alloc() doesn't return an error code.
>
>> +
>> + r5_rproc->auto_boot = false;
>> + r5_rproc->priv = r5_core;
> Here the original value of rproc->priv allocated by rproc_alloc() is replaced by
> @r5_core, resulting in a memory leak.
>
>> +
>> + /* Add R5 remoteproc */
>> + ret = devm_rproc_add(dev, r5_rproc);
>> + if (ret) {
>> + pr_err("failed to add r5 remoteproc\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int tcm_bank_count, tcm_node;
>> + int i = 0, j;
>> + struct zynqmp_r5_core *r5_core;
>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>> + struct device *dev = cluster->dev;
>> +
>> + /* ToDo: Use predefined TCM address space values from driver until
>> + * system-dt spec is not final fot TCM
>> + */
>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>> +
>> + /* count per core tcm banks */
>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>> +
>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>> + * 128k. Assign TCM banks to each core accordingly
>> + */
>> + tcm_node = 0;
>> + for (j = 0; j < cluster->core_count; j++) {
>> + r5_core = &cluster->r5_cores[j];
>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>> + tcm_bank_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>> + return -ENOMEM;
>> +
>> + for (i = 0; i < tcm_bank_count; i++) {
>> + /* Use pre-defined TCM reg values.
>> + * Eventually this should be replaced by values
>> + * parsed from dts.
>> + */
>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>> + tcm_node++;
>> + }
>> +
>> + r5_core->tcm_bank_count = tcm_bank_count;
>> + }
> Please use 'i' for the outer loop and 'j' for the inner one. Otherwise people
> are looking for a subtlety that isn't here.
>
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>> +{
>> + int res_mem_count, i;
>> + struct device *dev;
>> + struct device_node *np, *rmem_np;
>> + struct reserved_mem *rmem;
>> +
>> + dev = r5_core->dev;
>> +
>> + np = r5_core->np;
>> + if (IS_ERR_OR_NULL(np)) {
>> + pr_err("invalid device node of r5 core\n");
>> + return -EINVAL;
>> + }
> In normal cases this kind of defensive programming makes sense but here
> r5_core->np has just been set in zynqmp_r5_cluster_init() where there is
> plenty of gards against errors. Please remove.
>> +
>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>> + sizeof(phandle));
>> + if (res_mem_count <= 0) {
>> + dev_warn(dev, "failed to get memory-region property %d\n",
>> + res_mem_count);
>> + return -EINVAL;
>> + }
>> +
>> + r5_core->res_mem = devm_kzalloc(dev,
>> + res_mem_count * sizeof(struct reserved_mem),
>> + GFP_KERNEL);
> Use devm_kcalloc() when more than one element is required. Here and throughout
> the driver.
>
>> + if (!r5_core->res_mem) {
>> + dev_err(dev, "failed to allocate mem region memory\n");
>> + return -ENOMEM;
>> + }
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
> Why is the whole reserved_mem is duplicated when only ->base, ->size and ->name
> are used? And even those I'm not sure about really needing them...
>
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int ret, i;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev = cluster->dev;
>> +
>> + ret = zynqmp_r5_get_tcm_node(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("invalid r5 core\n");
>> + return -EINVAL;
>> + }
> Same here - please remove.
>
>> +
>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>> + if (ret)
>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>> +
>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>> + 1, &r5_core->pm_domain_id);
>> + if (ret) {
>> + dev_err(dev, "failed to get power-domains property\n");
>> + return ret;
>> + }
>> +
>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>> + if (ret) {
>> + dev_err(dev, "failed to init r5 core %d\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + struct device *dev = cluster->dev;
>> + struct device_node *dev_node = dev_of_node(dev);
>> + struct device_node *child;
>> + struct platform_device *child_pdev;
>> + int core_count = 0, ret, i;
>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>> + struct zynqmp_r5_core *r5_cores;
>> +
>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>> +
>> + /* on success returns 0, if not defined then returns -EINVAL,
>> + * In that case, default is LOCKSTEP mode
>> + */
>> + if (ret != -EINVAL && ret != 0) {
>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster_mode == SINGLE_CPU_MODE) {
>> + dev_err(dev, "driver does not support single cpu mode\n");
>> + return -EINVAL;
>> + } else if ((cluster_mode != SPLIT_MODE &&
>> + cluster_mode != LOCKSTEP_MODE)) {
>> + dev_err(dev, "Invalid cluster mode\n");
>> + return -EINVAL;
>> + }
>> +
>> + core_count = of_get_available_child_count(dev_node);
>> + if (core_count <= 0) {
>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>> + return -EINVAL;
>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>> + return -EINVAL;
>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>> + dev_warn(dev, "Only r5 core0 will be used\n");
>> + core_count = 1;
>> + }
>> +
>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>> + core_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_cores)) {
>> + dev_err(dev, "can't allocate memory for cores\n");
>> + return -ENOMEM;
>> + }
>> +
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
>> + return -ENODEV;
>> +
>> + r5_cores[i].dev = &child_pdev->dev;
>> + if (!r5_cores[i].dev) {
>> + pr_err("can't get device for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>> + if (!r5_cores[i].np) {
>> + pr_err("can't get device node for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
> The error path for both if() conditions above needs to release the
> child_pdev->dev, as instructed by the documentation for
> of_find_device_by_node(). The same needs to be done when there is no errors.
>
> More comments to come tomorrow.
>
> Thanks,
> Mathieu
>
>> +
>> + i++;
>> + if (i == core_count)
>> + break;
>> + }
>> +
>> + cluster->mode = cluster_mode;
>> + cluster->core_count = core_count;
>> + cluster->r5_cores = r5_cores;
>> +
>> + ret = zynqmp_r5_core_init(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + zynqmp_r5_print_dt_node_info(cluster);
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
>> +
>> + pr_info("Exit r5f subsystem driver\n");
>> +}
>> +
>> +/*
>> + * zynqmp_r5_remoteproc_probe()
>> + *
>> + * @pdev: domain platform device for R5 cluster
>> + *
>> + * called when driver is probed, for each R5 core specified in DT,
>> + * setup as needed to do remoteproc-related operations
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>> +{
>> + int ret;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct device *dev = &pdev->dev;
>> +
>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(cluster))
>> + return -ENOMEM;
>> +
>> + cluster->dev = dev;
>> +
>> + ret = devm_of_platform_populate(dev);
>> + if (ret) {
>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /* wire in so each core can be cleaned up at driver remove */
>> + platform_set_drvdata(pdev, cluster);
>> +
>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_cluster_init(cluster);
>> + if (ret) {
>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>> + return ret;
>> + }
>> +
>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
>> + return 0;
>> +}
>> +
>> +/* Match table for OF platform binding */
>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>> + { .compatible = "xlnx,zynqmp-r5fss", },
>> + { /* end of list */ },
>> +};
>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>> +
>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>> + .probe = zynqmp_r5_remoteproc_probe,
>> + .driver = {
>> + .name = "zynqmp_r5_remoteproc",
>> + .of_match_table = zynqmp_r5_remoteproc_match,
>> + },
>> +};
>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>> +
>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>> +MODULE_AUTHOR("Xilinx Inc.");
>> +MODULE_LICENSE("GPL v2");
>> --
>> 2.25.1
>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-13 19:11 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-13 19:11 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
Thanks Mathieu for reviews.
[...]
>> + * zynqmp_r5_set_mode - set RPU operation mode
>> + *
>> + * set RPU operation mode
>> + *
>> + * Return: 0 for success, negative value for failure
>> + */
>> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
>> + enum zynqmp_r5_cluster_mode rpu_mode)
>> +{
>> + enum rpu_tcm_comb tcm_mode;
>> + int ret, reg_val;
>> +
>> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
> Here split and single-cpu mode are treated the same... Is this an omission or
> really how the HW works? A good dose of documentation is definitely needed.
In this patchset, driver probe is failing in zynqmp_r5_cluster_init
if single-cpu mode is set in device-tree with message that
single-cpu mode is not supported by driver. So, here it is only for
split mode.
Later when single-cpu mode is supported, this will be modified accordingly.
Sure adding comments sounds good idea. I will take care it in next revision.
Also, I agree with all the comments in this email. I will address them
in next revision.
If I have any question while development, I will ask before posting next
revision.
>
>> +
>> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
>> + if (ret < 0) {
>> + pr_err("failed to set RPU mode\n");
>> + return ret;
>> + }
>> +
>> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
>> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
>> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
>> + if (ret < 0)
>> + pr_err("failed to configure TCM\n");
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_start
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Start R5 Core from designated boot address.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + enum rpu_boot_mem bootmem;
>> + int ret;
>> +
>> + if (!r5_core) {
>> + pr_err("can't get r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
>> +
>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>> +
>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>> + if (ret)
>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_stop
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Power down R5 Core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + int ret;
>> +
>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret)
>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap memory-region carveout
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + iounmap((void __iomem *)mem->va);
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_mem_regions
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Construct rproc mem carveouts from carveout provided in
>> + * memory-region property
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_mem_regions(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct rproc_mem_entry *mem;
>> + struct reserved_mem *rmem;
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> +
>> + r5_core = rproc->priv;
>> + dev = r5_core->dev;
>> +
>> + /* Register associated reserved memory regions */
>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>> + rmem = &r5_core->res_mem[i];
>> + mem = rproc_mem_entry_init(dev, NULL,
>> + (dma_addr_t)rmem->base,
>> + rmem->size, rmem->base,
>> + zynqmp_r5_rproc_mem_map,
>> + zynqmp_r5_rproc_mem_unmap,
>> + rmem->name);
>> + if (IS_ERR_OR_NULL(mem))
>> + return -ENOMEM;
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap TCM banks when powering down R5 core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>> +{
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> + enum pm_node_id pm_domain_id;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + pr_err("r5 core is not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + iounmap((void __iomem *)mem->va);
>> +
>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + if (zynqmp_pm_release_node(pm_domain_id))
>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * tcm_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to initialize the va and da fields of
>> + *
>> + * Given TCM bank entry, this callback will set device address for R5
>> + * running on TCM and also setup virtual address for TCM bank
>> + * remoteproc carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + /* Update memory entry va */
>> + mem->va = (void *)va;
>> +
>> + /* clear TCMs */
>> + memset_io(va, 0, mem->len);
>> +
>> + /*
>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>> + * while on the Linux side they are at 0xffexxxxx.
>> + *
>> + * Zero out the high 12 bits of the address. This will give
>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>> + */
>> + mem->da &= 0x000fffff;
>> +
>> + /*
>> + * TCM Banks 1A and 1B still have to be translated.
>> + *
>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>> + * 0xffeb0000) and convert to the expected relative addresses
>> + * (0x0 and 0x20000).
>> + */
>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>> + mem->da -= 0x90000;
>> +
>> + /* if translated TCM bank address is not valid report error */
>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>> + return -EINVAL;
>> + }
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; i < num_banks; i++) {
>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>> + bank_name = r5_core->tcm_banks[i].bank_name;
>> + bank_size = r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + /* Turn off all TCM banks turned on before */
>> + do {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x, %x\n",
>> + (u32)pm_domain_id, ret);
>> + } while (i--);
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
>> + char *bank_name;
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>> + bank_name = r5_core->tcm_banks[0].bank_name;
>> + for (i = 0; i < num_banks; i++) {
>> + bank_size += r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> + }
>> +
>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + for (i = 0; i < num_banks; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x ret: %x\n",
>> + (u32)pm_domain_id, ret);
>> + }
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_tcm_banks()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Given R5 node in remoteproc instance
>> + * allocate remoteproc carveout for TCM memory
>> + * needed for firmware to be loaded
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_parse_fw()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @fw: ptr to firmware to be loaded onto r5 core
>> + *
>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>> +{
>> + int ret;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + dev_err(&rproc->dev, "r5 core not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + dev = r5_core->dev;
>> +
>> + ret = add_tcm_banks(rproc);
>> + if (ret) {
>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = add_mem_regions(rproc);
>> + if (ret)
>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>> +
>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>> + if (ret == -EINVAL) {
>> + /*
>> + * resource table only required for IPC.
>> + * if not present, this is not necessarily an error;
>> + * for example, loading r5 hello world application
>> + * so simply inform user and keep going.
>> + */
>> + dev_info(&rproc->dev, "no resource table found.\n");
>> + ret = 0;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> +
>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int i, j, k;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + dev_dbg(cluster->dev, "Printing dt node info\n");
>> +
>> + pr_debug("cluster mode = %d\n", cluster->mode);
>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("can't get r5_core\n");
>> + continue;
>> + }
>> +
>> + pr_debug("r5 core %d nodes\n", i);
>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>> + k, r5_core->tcm_banks[k].addr,
>> + r5_core->tcm_banks[k].size,
>> + r5_core->tcm_banks[k].pm_domain_id,
>> + r5_core->tcm_banks[k].bank_name);
>> + }
>> +
>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>> +
>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>> + j, r5_core->res_mem[j].base,
>> + r5_core->res_mem[j].size,
>> + r5_core->res_mem[j].name);
>> + }
>> + }
>> +}
>> +
>> +/**
>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>> + * this is called for each individual R5 core to
>> + * set up mailbox, Xilinx platform manager unique ID,
>> + * add to rproc core
>> + *
> Please get rid of the indentations. That way it conforms to the rest of the
> style in this file.
>
>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>> +{
>> + int ret;
>> + struct rproc *r5_rproc;
>> + struct device *dev;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* Set up DMA mask */
>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>> + if (ret)
>> + return ret;
>> +
>> + /* Allocate remoteproc instance */
>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>> + NULL, sizeof(struct zynqmp_r5_core));
>> + if (IS_ERR_OR_NULL(r5_rproc))
>> + return -ENOMEM;
> devm_rproc_alloc() doesn't return an error code.
>
>> +
>> + r5_rproc->auto_boot = false;
>> + r5_rproc->priv = r5_core;
> Here the original value of rproc->priv allocated by rproc_alloc() is replaced by
> @r5_core, resulting in a memory leak.
>
>> +
>> + /* Add R5 remoteproc */
>> + ret = devm_rproc_add(dev, r5_rproc);
>> + if (ret) {
>> + pr_err("failed to add r5 remoteproc\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int tcm_bank_count, tcm_node;
>> + int i = 0, j;
>> + struct zynqmp_r5_core *r5_core;
>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>> + struct device *dev = cluster->dev;
>> +
>> + /* ToDo: Use predefined TCM address space values from driver until
>> + * system-dt spec is not final fot TCM
>> + */
>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>> +
>> + /* count per core tcm banks */
>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>> +
>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>> + * 128k. Assign TCM banks to each core accordingly
>> + */
>> + tcm_node = 0;
>> + for (j = 0; j < cluster->core_count; j++) {
>> + r5_core = &cluster->r5_cores[j];
>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>> + tcm_bank_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>> + return -ENOMEM;
>> +
>> + for (i = 0; i < tcm_bank_count; i++) {
>> + /* Use pre-defined TCM reg values.
>> + * Eventually this should be replaced by values
>> + * parsed from dts.
>> + */
>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>> + tcm_node++;
>> + }
>> +
>> + r5_core->tcm_bank_count = tcm_bank_count;
>> + }
> Please use 'i' for the outer loop and 'j' for the inner one. Otherwise people
> are looking for a subtlety that isn't here.
>
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>> +{
>> + int res_mem_count, i;
>> + struct device *dev;
>> + struct device_node *np, *rmem_np;
>> + struct reserved_mem *rmem;
>> +
>> + dev = r5_core->dev;
>> +
>> + np = r5_core->np;
>> + if (IS_ERR_OR_NULL(np)) {
>> + pr_err("invalid device node of r5 core\n");
>> + return -EINVAL;
>> + }
> In normal cases this kind of defensive programming makes sense but here
> r5_core->np has just been set in zynqmp_r5_cluster_init() where there is
> plenty of gards against errors. Please remove.
>> +
>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>> + sizeof(phandle));
>> + if (res_mem_count <= 0) {
>> + dev_warn(dev, "failed to get memory-region property %d\n",
>> + res_mem_count);
>> + return -EINVAL;
>> + }
>> +
>> + r5_core->res_mem = devm_kzalloc(dev,
>> + res_mem_count * sizeof(struct reserved_mem),
>> + GFP_KERNEL);
> Use devm_kcalloc() when more than one element is required. Here and throughout
> the driver.
>
>> + if (!r5_core->res_mem) {
>> + dev_err(dev, "failed to allocate mem region memory\n");
>> + return -ENOMEM;
>> + }
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
> Why is the whole reserved_mem is duplicated when only ->base, ->size and ->name
> are used? And even those I'm not sure about really needing them...
>
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int ret, i;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev = cluster->dev;
>> +
>> + ret = zynqmp_r5_get_tcm_node(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("invalid r5 core\n");
>> + return -EINVAL;
>> + }
> Same here - please remove.
>
>> +
>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>> + if (ret)
>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>> +
>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>> + 1, &r5_core->pm_domain_id);
>> + if (ret) {
>> + dev_err(dev, "failed to get power-domains property\n");
>> + return ret;
>> + }
>> +
>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>> + if (ret) {
>> + dev_err(dev, "failed to init r5 core %d\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + struct device *dev = cluster->dev;
>> + struct device_node *dev_node = dev_of_node(dev);
>> + struct device_node *child;
>> + struct platform_device *child_pdev;
>> + int core_count = 0, ret, i;
>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>> + struct zynqmp_r5_core *r5_cores;
>> +
>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>> +
>> + /* on success returns 0, if not defined then returns -EINVAL,
>> + * In that case, default is LOCKSTEP mode
>> + */
>> + if (ret != -EINVAL && ret != 0) {
>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster_mode == SINGLE_CPU_MODE) {
>> + dev_err(dev, "driver does not support single cpu mode\n");
>> + return -EINVAL;
>> + } else if ((cluster_mode != SPLIT_MODE &&
>> + cluster_mode != LOCKSTEP_MODE)) {
>> + dev_err(dev, "Invalid cluster mode\n");
>> + return -EINVAL;
>> + }
>> +
>> + core_count = of_get_available_child_count(dev_node);
>> + if (core_count <= 0) {
>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>> + return -EINVAL;
>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>> + return -EINVAL;
>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>> + dev_warn(dev, "Only r5 core0 will be used\n");
>> + core_count = 1;
>> + }
>> +
>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>> + core_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_cores)) {
>> + dev_err(dev, "can't allocate memory for cores\n");
>> + return -ENOMEM;
>> + }
>> +
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
>> + return -ENODEV;
>> +
>> + r5_cores[i].dev = &child_pdev->dev;
>> + if (!r5_cores[i].dev) {
>> + pr_err("can't get device for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>> + if (!r5_cores[i].np) {
>> + pr_err("can't get device node for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
> The error path for both if() conditions above needs to release the
> child_pdev->dev, as instructed by the documentation for
> of_find_device_by_node(). The same needs to be done when there is no errors.
>
> More comments to come tomorrow.
>
> Thanks,
> Mathieu
>
>> +
>> + i++;
>> + if (i == core_count)
>> + break;
>> + }
>> +
>> + cluster->mode = cluster_mode;
>> + cluster->core_count = core_count;
>> + cluster->r5_cores = r5_cores;
>> +
>> + ret = zynqmp_r5_core_init(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + zynqmp_r5_print_dt_node_info(cluster);
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
>> +
>> + pr_info("Exit r5f subsystem driver\n");
>> +}
>> +
>> +/*
>> + * zynqmp_r5_remoteproc_probe()
>> + *
>> + * @pdev: domain platform device for R5 cluster
>> + *
>> + * called when driver is probed, for each R5 core specified in DT,
>> + * setup as needed to do remoteproc-related operations
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>> +{
>> + int ret;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct device *dev = &pdev->dev;
>> +
>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(cluster))
>> + return -ENOMEM;
>> +
>> + cluster->dev = dev;
>> +
>> + ret = devm_of_platform_populate(dev);
>> + if (ret) {
>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /* wire in so each core can be cleaned up at driver remove */
>> + platform_set_drvdata(pdev, cluster);
>> +
>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_cluster_init(cluster);
>> + if (ret) {
>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>> + return ret;
>> + }
>> +
>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
>> + return 0;
>> +}
>> +
>> +/* Match table for OF platform binding */
>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>> + { .compatible = "xlnx,zynqmp-r5fss", },
>> + { /* end of list */ },
>> +};
>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>> +
>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>> + .probe = zynqmp_r5_remoteproc_probe,
>> + .driver = {
>> + .name = "zynqmp_r5_remoteproc",
>> + .of_match_table = zynqmp_r5_remoteproc_match,
>> + },
>> +};
>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>> +
>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>> +MODULE_AUTHOR("Xilinx Inc.");
>> +MODULE_LICENSE("GPL v2");
>> --
>> 2.25.1
>>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-12-07 18:31 ` Mathieu Poirier
@ 2021-12-13 19:17 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-13 19:17 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On 12/8/21 12:01 AM, Mathieu Poirier wrote:
[...]
>> +
>> +/*
>> + * zynqmp_r5_rproc_start
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Start R5 Core from designated boot address.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + enum rpu_boot_mem bootmem;
>> + int ret;
>> +
>> + if (!r5_core) {
>> + pr_err("can't get r5 core\n");
>> + return -EINVAL;
>> + }
> Please remove.
>
>> +
>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> Some comments as to what is happening here would be appreciated.
>
>> +
>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>> +
>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>> + if (ret)
>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_stop
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Power down R5 Core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + int ret;
>> +
>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret)
>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap memory-region carveout
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + iounmap((void __iomem *)mem->va);
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_mem_regions
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Construct rproc mem carveouts from carveout provided in
>> + * memory-region property
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_mem_regions(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct rproc_mem_entry *mem;
>> + struct reserved_mem *rmem;
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> +
>> + r5_core = rproc->priv;
>> + dev = r5_core->dev;
>> +
>> + /* Register associated reserved memory regions */
>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>> + rmem = &r5_core->res_mem[i];
> Indentation problem
>
>> + mem = rproc_mem_entry_init(dev, NULL,
>> + (dma_addr_t)rmem->base,
>> + rmem->size, rmem->base,
>> + zynqmp_r5_rproc_mem_map,
>> + zynqmp_r5_rproc_mem_unmap,
>> + rmem->name);
>> + if (IS_ERR_OR_NULL(mem))
>> + return -ENOMEM;
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
> Not sure why this wasn't done in zynqmp_r5_get_mem_region_node()... That would
> prevent the need to keep a copy of the reserved memories in the zynqmp_r5_core
> structure.
Sure, I will analyze this optimization.
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap TCM banks when powering down R5 core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>> +{
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> + enum pm_node_id pm_domain_id;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + pr_err("r5 core is not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + iounmap((void __iomem *)mem->va);
>> +
>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + if (zynqmp_pm_release_node(pm_domain_id))
>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * tcm_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to initialize the va and da fields of
>> + *
>> + * Given TCM bank entry, this callback will set device address for R5
>> + * running on TCM and also setup virtual address for TCM bank
>> + * remoteproc carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
> Indentation problem
>
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + /* Update memory entry va */
>> + mem->va = (void *)va;
>> +
>> + /* clear TCMs */
>> + memset_io(va, 0, mem->len);
>> +
>> + /*
>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>> + * while on the Linux side they are at 0xffexxxxx.
>> + *
>> + * Zero out the high 12 bits of the address. This will give
>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>> + */
>> + mem->da &= 0x000fffff;
>> +
>> + /*
>> + * TCM Banks 1A and 1B still have to be translated.
>> + *
>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>> + * 0xffeb0000) and convert to the expected relative addresses
>> + * (0x0 and 0x20000).
>> + */
>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>> + mem->da -= 0x90000;
>> +
>> + /* if translated TCM bank address is not valid report error */
>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>> + return -EINVAL;
>> + }
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
> phys_addr_t bank_addr;
>
>> + size_t bank_size = 0;
> No initialisation needed.
>
>> + char *bank_name;
>> + struct device *dev;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
> Remove.
>
>> +
>> + dev = r5_core->dev;
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; i < num_banks; i++) {
>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>> + bank_name = r5_core->tcm_banks[i].bank_name;
>> + bank_size = r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
> I'm pretty sure that if you run the sparse checker on this you will get a lot of
> angry warnings.
>
>
>> + if (IS_ERR_OR_NULL(mem)) {
> Function rproc_mem_entry_init() does not return an error code.
>
>> + /* Turn off all TCM banks turned on before */
>> + do {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> Here @pm_domain_id is casted as a u32 but it isn't above when calling
> zynqmp_pm_request_node(), and that despite both functions taking a "const u32
> node". Again, I'm sure the sparse checker will complain here.
>
>
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x, %x\n",
>> + (u32)pm_domain_id, ret);
>> + } while (i--);
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
> Same as above.
>
>> + char *bank_name;
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
> Same as above.
>
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>> + bank_name = r5_core->tcm_banks[0].bank_name;
>> + for (i = 0; i < num_banks; i++) {
>> + bank_size += r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> + }
> I know what is happening above but in 6 months it will all be gone from
> memory. Please add enough documentation for someone not familiar with the
> platform to understand why things are done that way. Here and throughout the
> driver.
>
>> +
>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + for (i = 0; i < num_banks; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x ret: %x\n",
>> + (u32)pm_domain_id, ret);
>> + }
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_tcm_banks()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Given R5 node in remoteproc instance
>> + * allocate remoteproc carveout for TCM memory
>> + * needed for firmware to be loaded
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
> Remove.
>
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
> Why not simply calling dev_get_drvdata(dev->parent) instead of going through the
> platform API?
>
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_parse_fw()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @fw: ptr to firmware to be loaded onto r5 core
>> + *
>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>> +{
>> + int ret;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + dev_err(&rproc->dev, "r5 core not available\n");
>> + return -EINVAL;
>> + }
> Same as above and throughout.
>
> I am done reviewing this set - the overall structure is sound but there is a
> fair amount of work to be done. There is no need to rush another revision as I
> won't have time to review it before the holidays. On that front, change
> _only_ the things I have commented on. We can address other issues as we move
> forward. Last but not least, please run the sparse checker on your work.
>
> Thanks,
> Mathieu
>
Sure Mathieu. I will address all the comments you have posted.
Running sparse checker is also good idea. I will run it for all the next
revisions.
Thanks,
Tanmay
>> +
>> + dev = r5_core->dev;
>> +
>> + ret = add_tcm_banks(rproc);
>> + if (ret) {
>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = add_mem_regions(rproc);
>> + if (ret)
>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>> +
>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>> + if (ret == -EINVAL) {
>> + /*
>> + * resource table only required for IPC.
>> + * if not present, this is not necessarily an error;
>> + * for example, loading r5 hello world application
>> + * so simply inform user and keep going.
>> + */
>> + dev_info(&rproc->dev, "no resource table found.\n");
>> + ret = 0;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> +
>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int i, j, k;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + dev_dbg(cluster->dev, "Printing dt node info\n");
>> +
>> + pr_debug("cluster mode = %d\n", cluster->mode);
>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("can't get r5_core\n");
>> + continue;
>> + }
>> +
>> + pr_debug("r5 core %d nodes\n", i);
>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>> + k, r5_core->tcm_banks[k].addr,
>> + r5_core->tcm_banks[k].size,
>> + r5_core->tcm_banks[k].pm_domain_id,
>> + r5_core->tcm_banks[k].bank_name);
>> + }
>> +
>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>> +
>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>> + j, r5_core->res_mem[j].base,
>> + r5_core->res_mem[j].size,
>> + r5_core->res_mem[j].name);
>> + }
>> + }
>> +}
>> +
>> +/**
>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>> + * this is called for each individual R5 core to
>> + * set up mailbox, Xilinx platform manager unique ID,
>> + * add to rproc core
>> + *
>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>> +{
>> + int ret;
>> + struct rproc *r5_rproc;
>> + struct device *dev;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* Set up DMA mask */
>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>> + if (ret)
>> + return ret;
>> +
>> + /* Allocate remoteproc instance */
>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>> + NULL, sizeof(struct zynqmp_r5_core));
>> + if (IS_ERR_OR_NULL(r5_rproc))
>> + return -ENOMEM;
>> +
>> + r5_rproc->auto_boot = false;
>> + r5_rproc->priv = r5_core;
>> +
>> + /* Add R5 remoteproc */
>> + ret = devm_rproc_add(dev, r5_rproc);
>> + if (ret) {
>> + pr_err("failed to add r5 remoteproc\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int tcm_bank_count, tcm_node;
>> + int i = 0, j;
>> + struct zynqmp_r5_core *r5_core;
>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>> + struct device *dev = cluster->dev;
>> +
>> + /* ToDo: Use predefined TCM address space values from driver until
>> + * system-dt spec is not final fot TCM
>> + */
>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>> +
>> + /* count per core tcm banks */
>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>> +
>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>> + * 128k. Assign TCM banks to each core accordingly
>> + */
>> + tcm_node = 0;
>> + for (j = 0; j < cluster->core_count; j++) {
>> + r5_core = &cluster->r5_cores[j];
>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>> + tcm_bank_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>> + return -ENOMEM;
>> +
>> + for (i = 0; i < tcm_bank_count; i++) {
>> + /* Use pre-defined TCM reg values.
>> + * Eventually this should be replaced by values
>> + * parsed from dts.
>> + */
>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>> + tcm_node++;
>> + }
>> +
>> + r5_core->tcm_bank_count = tcm_bank_count;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>> +{
>> + int res_mem_count, i;
>> + struct device *dev;
>> + struct device_node *np, *rmem_np;
>> + struct reserved_mem *rmem;
>> +
>> + dev = r5_core->dev;
>> +
>> + np = r5_core->np;
>> + if (IS_ERR_OR_NULL(np)) {
>> + pr_err("invalid device node of r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>> + sizeof(phandle));
>> + if (res_mem_count <= 0) {
>> + dev_warn(dev, "failed to get memory-region property %d\n",
>> + res_mem_count);
>> + return -EINVAL;
>> + }
>> +
>> + r5_core->res_mem = devm_kzalloc(dev,
>> + res_mem_count * sizeof(struct reserved_mem),
>> + GFP_KERNEL);
>> + if (!r5_core->res_mem) {
>> + dev_err(dev, "failed to allocate mem region memory\n");
>> + return -ENOMEM;
>> + }
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int ret, i;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev = cluster->dev;
>> +
>> + ret = zynqmp_r5_get_tcm_node(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("invalid r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>> + if (ret)
>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>> +
>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>> + 1, &r5_core->pm_domain_id);
>> + if (ret) {
>> + dev_err(dev, "failed to get power-domains property\n");
>> + return ret;
>> + }
>> +
>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>> + if (ret) {
>> + dev_err(dev, "failed to init r5 core %d\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + struct device *dev = cluster->dev;
>> + struct device_node *dev_node = dev_of_node(dev);
>> + struct device_node *child;
>> + struct platform_device *child_pdev;
>> + int core_count = 0, ret, i;
>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>> + struct zynqmp_r5_core *r5_cores;
>> +
>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>> +
>> + /* on success returns 0, if not defined then returns -EINVAL,
>> + * In that case, default is LOCKSTEP mode
>> + */
>> + if (ret != -EINVAL && ret != 0) {
>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster_mode == SINGLE_CPU_MODE) {
>> + dev_err(dev, "driver does not support single cpu mode\n");
>> + return -EINVAL;
>> + } else if ((cluster_mode != SPLIT_MODE &&
>> + cluster_mode != LOCKSTEP_MODE)) {
>> + dev_err(dev, "Invalid cluster mode\n");
>> + return -EINVAL;
>> + }
>> +
>> + core_count = of_get_available_child_count(dev_node);
>> + if (core_count <= 0) {
>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>> + return -EINVAL;
>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>> + return -EINVAL;
>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>> + dev_warn(dev, "Only r5 core0 will be used\n");
>> + core_count = 1;
>> + }
>> +
>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>> + core_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_cores)) {
>> + dev_err(dev, "can't allocate memory for cores\n");
>> + return -ENOMEM;
>> + }
>> +
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
>> + return -ENODEV;
>> +
>> + r5_cores[i].dev = &child_pdev->dev;
>> + if (!r5_cores[i].dev) {
>> + pr_err("can't get device for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>> + if (!r5_cores[i].np) {
>> + pr_err("can't get device node for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + i++;
>> + if (i == core_count)
>> + break;
>> + }
>> +
>> + cluster->mode = cluster_mode;
>> + cluster->core_count = core_count;
>> + cluster->r5_cores = r5_cores;
>> +
>> + ret = zynqmp_r5_core_init(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + zynqmp_r5_print_dt_node_info(cluster);
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
>> +
>> + pr_info("Exit r5f subsystem driver\n");
>> +}
>> +
>> +/*
>> + * zynqmp_r5_remoteproc_probe()
>> + *
>> + * @pdev: domain platform device for R5 cluster
>> + *
>> + * called when driver is probed, for each R5 core specified in DT,
>> + * setup as needed to do remoteproc-related operations
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>> +{
>> + int ret;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct device *dev = &pdev->dev;
>> +
>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(cluster))
>> + return -ENOMEM;
>> +
>> + cluster->dev = dev;
>> +
>> + ret = devm_of_platform_populate(dev);
>> + if (ret) {
>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /* wire in so each core can be cleaned up at driver remove */
>> + platform_set_drvdata(pdev, cluster);
>> +
>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_cluster_init(cluster);
>> + if (ret) {
>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>> + return ret;
>> + }
>> +
>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
>> + return 0;
>> +}
>> +
>> +/* Match table for OF platform binding */
>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>> + { .compatible = "xlnx,zynqmp-r5fss", },
>> + { /* end of list */ },
>> +};
>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>> +
>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>> + .probe = zynqmp_r5_remoteproc_probe,
>> + .driver = {
>> + .name = "zynqmp_r5_remoteproc",
>> + .of_match_table = zynqmp_r5_remoteproc_match,
>> + },
>> +};
>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>> +
>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>> +MODULE_AUTHOR("Xilinx Inc.");
>> +MODULE_LICENSE("GPL v2");
>> --
>> 2.25.1
>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-13 19:17 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-13 19:17 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On 12/8/21 12:01 AM, Mathieu Poirier wrote:
[...]
>> +
>> +/*
>> + * zynqmp_r5_rproc_start
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Start R5 Core from designated boot address.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + enum rpu_boot_mem bootmem;
>> + int ret;
>> +
>> + if (!r5_core) {
>> + pr_err("can't get r5 core\n");
>> + return -EINVAL;
>> + }
> Please remove.
>
>> +
>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> Some comments as to what is happening here would be appreciated.
>
>> +
>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>> +
>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>> + if (ret)
>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_stop
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Power down R5 Core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + int ret;
>> +
>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret)
>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap memory-region carveout
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + iounmap((void __iomem *)mem->va);
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_mem_regions
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Construct rproc mem carveouts from carveout provided in
>> + * memory-region property
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_mem_regions(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct rproc_mem_entry *mem;
>> + struct reserved_mem *rmem;
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> +
>> + r5_core = rproc->priv;
>> + dev = r5_core->dev;
>> +
>> + /* Register associated reserved memory regions */
>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>> + rmem = &r5_core->res_mem[i];
> Indentation problem
>
>> + mem = rproc_mem_entry_init(dev, NULL,
>> + (dma_addr_t)rmem->base,
>> + rmem->size, rmem->base,
>> + zynqmp_r5_rproc_mem_map,
>> + zynqmp_r5_rproc_mem_unmap,
>> + rmem->name);
>> + if (IS_ERR_OR_NULL(mem))
>> + return -ENOMEM;
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
> Not sure why this wasn't done in zynqmp_r5_get_mem_region_node()... That would
> prevent the need to keep a copy of the reserved memories in the zynqmp_r5_core
> structure.
Sure, I will analyze this optimization.
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap TCM banks when powering down R5 core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>> +{
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> + enum pm_node_id pm_domain_id;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + pr_err("r5 core is not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + iounmap((void __iomem *)mem->va);
>> +
>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + if (zynqmp_pm_release_node(pm_domain_id))
>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * tcm_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to initialize the va and da fields of
>> + *
>> + * Given TCM bank entry, this callback will set device address for R5
>> + * running on TCM and also setup virtual address for TCM bank
>> + * remoteproc carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
> Indentation problem
>
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + /* Update memory entry va */
>> + mem->va = (void *)va;
>> +
>> + /* clear TCMs */
>> + memset_io(va, 0, mem->len);
>> +
>> + /*
>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>> + * while on the Linux side they are at 0xffexxxxx.
>> + *
>> + * Zero out the high 12 bits of the address. This will give
>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>> + */
>> + mem->da &= 0x000fffff;
>> +
>> + /*
>> + * TCM Banks 1A and 1B still have to be translated.
>> + *
>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>> + * 0xffeb0000) and convert to the expected relative addresses
>> + * (0x0 and 0x20000).
>> + */
>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>> + mem->da -= 0x90000;
>> +
>> + /* if translated TCM bank address is not valid report error */
>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>> + return -EINVAL;
>> + }
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
> phys_addr_t bank_addr;
>
>> + size_t bank_size = 0;
> No initialisation needed.
>
>> + char *bank_name;
>> + struct device *dev;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
> Remove.
>
>> +
>> + dev = r5_core->dev;
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; i < num_banks; i++) {
>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>> + bank_name = r5_core->tcm_banks[i].bank_name;
>> + bank_size = r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
> I'm pretty sure that if you run the sparse checker on this you will get a lot of
> angry warnings.
>
>
>> + if (IS_ERR_OR_NULL(mem)) {
> Function rproc_mem_entry_init() does not return an error code.
>
>> + /* Turn off all TCM banks turned on before */
>> + do {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> Here @pm_domain_id is casted as a u32 but it isn't above when calling
> zynqmp_pm_request_node(), and that despite both functions taking a "const u32
> node". Again, I'm sure the sparse checker will complain here.
>
>
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x, %x\n",
>> + (u32)pm_domain_id, ret);
>> + } while (i--);
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
> Same as above.
>
>> + char *bank_name;
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
> Same as above.
>
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>> + bank_name = r5_core->tcm_banks[0].bank_name;
>> + for (i = 0; i < num_banks; i++) {
>> + bank_size += r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> + }
> I know what is happening above but in 6 months it will all be gone from
> memory. Please add enough documentation for someone not familiar with the
> platform to understand why things are done that way. Here and throughout the
> driver.
>
>> +
>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + for (i = 0; i < num_banks; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x ret: %x\n",
>> + (u32)pm_domain_id, ret);
>> + }
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_tcm_banks()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Given R5 node in remoteproc instance
>> + * allocate remoteproc carveout for TCM memory
>> + * needed for firmware to be loaded
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
> Remove.
>
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
> Why not simply calling dev_get_drvdata(dev->parent) instead of going through the
> platform API?
>
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_parse_fw()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @fw: ptr to firmware to be loaded onto r5 core
>> + *
>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>> +{
>> + int ret;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + dev_err(&rproc->dev, "r5 core not available\n");
>> + return -EINVAL;
>> + }
> Same as above and throughout.
>
> I am done reviewing this set - the overall structure is sound but there is a
> fair amount of work to be done. There is no need to rush another revision as I
> won't have time to review it before the holidays. On that front, change
> _only_ the things I have commented on. We can address other issues as we move
> forward. Last but not least, please run the sparse checker on your work.
>
> Thanks,
> Mathieu
>
Sure Mathieu. I will address all the comments you have posted.
Running sparse checker is also good idea. I will run it for all the next
revisions.
Thanks,
Tanmay
>> +
>> + dev = r5_core->dev;
>> +
>> + ret = add_tcm_banks(rproc);
>> + if (ret) {
>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = add_mem_regions(rproc);
>> + if (ret)
>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>> +
>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>> + if (ret == -EINVAL) {
>> + /*
>> + * resource table only required for IPC.
>> + * if not present, this is not necessarily an error;
>> + * for example, loading r5 hello world application
>> + * so simply inform user and keep going.
>> + */
>> + dev_info(&rproc->dev, "no resource table found.\n");
>> + ret = 0;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> +
>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int i, j, k;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + dev_dbg(cluster->dev, "Printing dt node info\n");
>> +
>> + pr_debug("cluster mode = %d\n", cluster->mode);
>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("can't get r5_core\n");
>> + continue;
>> + }
>> +
>> + pr_debug("r5 core %d nodes\n", i);
>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>> + k, r5_core->tcm_banks[k].addr,
>> + r5_core->tcm_banks[k].size,
>> + r5_core->tcm_banks[k].pm_domain_id,
>> + r5_core->tcm_banks[k].bank_name);
>> + }
>> +
>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>> +
>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>> + j, r5_core->res_mem[j].base,
>> + r5_core->res_mem[j].size,
>> + r5_core->res_mem[j].name);
>> + }
>> + }
>> +}
>> +
>> +/**
>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>> + * this is called for each individual R5 core to
>> + * set up mailbox, Xilinx platform manager unique ID,
>> + * add to rproc core
>> + *
>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>> +{
>> + int ret;
>> + struct rproc *r5_rproc;
>> + struct device *dev;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* Set up DMA mask */
>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>> + if (ret)
>> + return ret;
>> +
>> + /* Allocate remoteproc instance */
>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>> + NULL, sizeof(struct zynqmp_r5_core));
>> + if (IS_ERR_OR_NULL(r5_rproc))
>> + return -ENOMEM;
>> +
>> + r5_rproc->auto_boot = false;
>> + r5_rproc->priv = r5_core;
>> +
>> + /* Add R5 remoteproc */
>> + ret = devm_rproc_add(dev, r5_rproc);
>> + if (ret) {
>> + pr_err("failed to add r5 remoteproc\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int tcm_bank_count, tcm_node;
>> + int i = 0, j;
>> + struct zynqmp_r5_core *r5_core;
>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>> + struct device *dev = cluster->dev;
>> +
>> + /* ToDo: Use predefined TCM address space values from driver until
>> + * system-dt spec is not final fot TCM
>> + */
>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>> +
>> + /* count per core tcm banks */
>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>> +
>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>> + * 128k. Assign TCM banks to each core accordingly
>> + */
>> + tcm_node = 0;
>> + for (j = 0; j < cluster->core_count; j++) {
>> + r5_core = &cluster->r5_cores[j];
>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>> + tcm_bank_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>> + return -ENOMEM;
>> +
>> + for (i = 0; i < tcm_bank_count; i++) {
>> + /* Use pre-defined TCM reg values.
>> + * Eventually this should be replaced by values
>> + * parsed from dts.
>> + */
>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>> + tcm_node++;
>> + }
>> +
>> + r5_core->tcm_bank_count = tcm_bank_count;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>> +{
>> + int res_mem_count, i;
>> + struct device *dev;
>> + struct device_node *np, *rmem_np;
>> + struct reserved_mem *rmem;
>> +
>> + dev = r5_core->dev;
>> +
>> + np = r5_core->np;
>> + if (IS_ERR_OR_NULL(np)) {
>> + pr_err("invalid device node of r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>> + sizeof(phandle));
>> + if (res_mem_count <= 0) {
>> + dev_warn(dev, "failed to get memory-region property %d\n",
>> + res_mem_count);
>> + return -EINVAL;
>> + }
>> +
>> + r5_core->res_mem = devm_kzalloc(dev,
>> + res_mem_count * sizeof(struct reserved_mem),
>> + GFP_KERNEL);
>> + if (!r5_core->res_mem) {
>> + dev_err(dev, "failed to allocate mem region memory\n");
>> + return -ENOMEM;
>> + }
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int ret, i;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev = cluster->dev;
>> +
>> + ret = zynqmp_r5_get_tcm_node(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("invalid r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>> + if (ret)
>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>> +
>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>> + 1, &r5_core->pm_domain_id);
>> + if (ret) {
>> + dev_err(dev, "failed to get power-domains property\n");
>> + return ret;
>> + }
>> +
>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>> + if (ret) {
>> + dev_err(dev, "failed to init r5 core %d\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + struct device *dev = cluster->dev;
>> + struct device_node *dev_node = dev_of_node(dev);
>> + struct device_node *child;
>> + struct platform_device *child_pdev;
>> + int core_count = 0, ret, i;
>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>> + struct zynqmp_r5_core *r5_cores;
>> +
>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>> +
>> + /* on success returns 0, if not defined then returns -EINVAL,
>> + * In that case, default is LOCKSTEP mode
>> + */
>> + if (ret != -EINVAL && ret != 0) {
>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster_mode == SINGLE_CPU_MODE) {
>> + dev_err(dev, "driver does not support single cpu mode\n");
>> + return -EINVAL;
>> + } else if ((cluster_mode != SPLIT_MODE &&
>> + cluster_mode != LOCKSTEP_MODE)) {
>> + dev_err(dev, "Invalid cluster mode\n");
>> + return -EINVAL;
>> + }
>> +
>> + core_count = of_get_available_child_count(dev_node);
>> + if (core_count <= 0) {
>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>> + return -EINVAL;
>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>> + return -EINVAL;
>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>> + dev_warn(dev, "Only r5 core0 will be used\n");
>> + core_count = 1;
>> + }
>> +
>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>> + core_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_cores)) {
>> + dev_err(dev, "can't allocate memory for cores\n");
>> + return -ENOMEM;
>> + }
>> +
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
>> + return -ENODEV;
>> +
>> + r5_cores[i].dev = &child_pdev->dev;
>> + if (!r5_cores[i].dev) {
>> + pr_err("can't get device for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>> + if (!r5_cores[i].np) {
>> + pr_err("can't get device node for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + i++;
>> + if (i == core_count)
>> + break;
>> + }
>> +
>> + cluster->mode = cluster_mode;
>> + cluster->core_count = core_count;
>> + cluster->r5_cores = r5_cores;
>> +
>> + ret = zynqmp_r5_core_init(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + zynqmp_r5_print_dt_node_info(cluster);
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
>> +
>> + pr_info("Exit r5f subsystem driver\n");
>> +}
>> +
>> +/*
>> + * zynqmp_r5_remoteproc_probe()
>> + *
>> + * @pdev: domain platform device for R5 cluster
>> + *
>> + * called when driver is probed, for each R5 core specified in DT,
>> + * setup as needed to do remoteproc-related operations
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>> +{
>> + int ret;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct device *dev = &pdev->dev;
>> +
>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(cluster))
>> + return -ENOMEM;
>> +
>> + cluster->dev = dev;
>> +
>> + ret = devm_of_platform_populate(dev);
>> + if (ret) {
>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /* wire in so each core can be cleaned up at driver remove */
>> + platform_set_drvdata(pdev, cluster);
>> +
>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_cluster_init(cluster);
>> + if (ret) {
>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>> + return ret;
>> + }
>> +
>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
>> + return 0;
>> +}
>> +
>> +/* Match table for OF platform binding */
>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>> + { .compatible = "xlnx,zynqmp-r5fss", },
>> + { /* end of list */ },
>> +};
>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>> +
>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>> + .probe = zynqmp_r5_remoteproc_probe,
>> + .driver = {
>> + .name = "zynqmp_r5_remoteproc",
>> + .of_match_table = zynqmp_r5_remoteproc_match,
>> + },
>> +};
>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>> +
>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>> +MODULE_AUTHOR("Xilinx Inc.");
>> +MODULE_LICENSE("GPL v2");
>> --
>> 2.25.1
>>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-12-13 11:08 ` Lars-Peter Clausen
@ 2021-12-13 19:31 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-13 19:31 UTC (permalink / raw)
To: Lars-Peter Clausen, Bjorn Andersson, Mathieu Poirier,
Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
linux-remoteproc, linux-kernel, devicetree, linux-arm-kernel
On 12/13/21 4:38 PM, Lars-Peter Clausen wrote:
> On 11/23/21 7:20 AM, Tanmay Shah wrote:
>> [...]
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>
> Since you want normal memory and not IO memory a better choice might
> be memremap() with MEMREMAP_WC. Internally memremap() will call
> ioremap_wc(), but this will make the intention clear and you do not
> have to deal with the __iomem type cast.
>
Thanks Lars-Peter. Yes you are right. I found this article
https://lwn.net/Articles/653585/ about use of memremap after I posted
this patchset and I have planned to replace ioremap_wc function with
memremap with MEMREMAP_WC flag.
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> [...]
>>
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>
> You could just use dev_get_drvdata() without having to cast back to
> the platform_device first.
>
>
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> [...]
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> const
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> [....]
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core
>> *r5_core)
>> +{
>> [...]
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
>
> r5_core->res_mem[i] = *mem;
>
> This will give you proper type checking and is also a bit shorter.
>
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> [...]
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> [...]
>> +
>>
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
> A return or a break in a for_each_available_child_of_node() will leak
> the reference to the child node.
Do you mean I have to use of_put_node for each child?
>> [...]
>> + }
>> +
>> [...]
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
> This is not needed. The device driver core will set drvdata to NULL
> when the device is removed.
>> +
>> + pr_info("Exit r5f subsystem driver\n");
> This is probably also not needed.
>> +}
>
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-13 19:31 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-13 19:31 UTC (permalink / raw)
To: Lars-Peter Clausen, Bjorn Andersson, Mathieu Poirier,
Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
linux-remoteproc, linux-kernel, devicetree, linux-arm-kernel
On 12/13/21 4:38 PM, Lars-Peter Clausen wrote:
> On 11/23/21 7:20 AM, Tanmay Shah wrote:
>> [...]
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>
> Since you want normal memory and not IO memory a better choice might
> be memremap() with MEMREMAP_WC. Internally memremap() will call
> ioremap_wc(), but this will make the intention clear and you do not
> have to deal with the __iomem type cast.
>
Thanks Lars-Peter. Yes you are right. I found this article
https://lwn.net/Articles/653585/ about use of memremap after I posted
this patchset and I have planned to replace ioremap_wc function with
memremap with MEMREMAP_WC flag.
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> [...]
>>
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>
> You could just use dev_get_drvdata() without having to cast back to
> the platform_device first.
>
>
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> [...]
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> const
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> [....]
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core
>> *r5_core)
>> +{
>> [...]
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
>
> r5_core->res_mem[i] = *mem;
>
> This will give you proper type checking and is also a bit shorter.
>
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> [...]
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> [...]
>> +
>>
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
> A return or a break in a for_each_available_child_of_node() will leak
> the reference to the child node.
Do you mean I have to use of_put_node for each child?
>> [...]
>> + }
>> +
>> [...]
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
> This is not needed. The device driver core will set drvdata to NULL
> when the device is removed.
>> +
>> + pr_info("Exit r5f subsystem driver\n");
> This is probably also not needed.
>> +}
>
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-12-13 19:31 ` Tanmay Shah
@ 2021-12-13 19:43 ` Lars-Peter Clausen
-1 siblings, 0 replies; 56+ messages in thread
From: Lars-Peter Clausen @ 2021-12-13 19:43 UTC (permalink / raw)
To: Tanmay Shah, Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
linux-remoteproc, linux-kernel, devicetree, linux-arm-kernel
On 12/13/21 8:31 PM, Tanmay Shah wrote:
>>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>>> +{
>>> [...]
>>> +
>>>
>>> + i = 0;
>>> + for_each_available_child_of_node(dev_node, child) {
>>> + child_pdev = of_find_device_by_node(child);
>>> + if (!child_pdev)
>> A return or a break in a for_each_available_child_of_node() will leak
>> the reference to the child node.
>
> Do you mean I have to use of_put_node for each child?
You have to put a `of_put_node(child)` before each break or return in
the loop.
If you search the kernel commit history for
for_each_available_child_of_node or for_each_child_of_node you'll find
many fixes for similar problems. For example
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e1b391e9712db
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-13 19:43 ` Lars-Peter Clausen
0 siblings, 0 replies; 56+ messages in thread
From: Lars-Peter Clausen @ 2021-12-13 19:43 UTC (permalink / raw)
To: Tanmay Shah, Bjorn Andersson, Mathieu Poirier, Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
linux-remoteproc, linux-kernel, devicetree, linux-arm-kernel
On 12/13/21 8:31 PM, Tanmay Shah wrote:
>>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>>> +{
>>> [...]
>>> +
>>>
>>> + i = 0;
>>> + for_each_available_child_of_node(dev_node, child) {
>>> + child_pdev = of_find_device_by_node(child);
>>> + if (!child_pdev)
>> A return or a break in a for_each_available_child_of_node() will leak
>> the reference to the child node.
>
> Do you mean I have to use of_put_node for each child?
You have to put a `of_put_node(child)` before each break or return in
the loop.
If you search the kernel commit history for
for_each_available_child_of_node or for_each_child_of_node you'll find
many fixes for similar problems. For example
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e1b391e9712db
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-12-13 19:43 ` Lars-Peter Clausen
@ 2021-12-13 19:46 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-13 19:46 UTC (permalink / raw)
To: Lars-Peter Clausen, Bjorn Andersson, Mathieu Poirier,
Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
linux-remoteproc, linux-kernel, devicetree, linux-arm-kernel
On 12/14/21 1:13 AM, Lars-Peter Clausen wrote:
> On 12/13/21 8:31 PM, Tanmay Shah wrote:
>>>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> [...]
>>>> +
>>>>
>>>> + i = 0;
>>>> + for_each_available_child_of_node(dev_node, child) {
>>>> + child_pdev = of_find_device_by_node(child);
>>>> + if (!child_pdev)
>>> A return or a break in a for_each_available_child_of_node() will
>>> leak the reference to the child node.
>>
>> Do you mean I have to use of_put_node for each child?
>
> You have to put a `of_put_node(child)` before each break or return in
> the loop.
>
> If you search the kernel commit history for
> for_each_available_child_of_node or for_each_child_of_node you'll find
> many fixes for similar problems. For example
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e1b391e9712db
>
I got it. Same concern was posted by Mathieu also. I will address these
comments in next revision.
Thanks,
Tanmay
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2021-12-13 19:46 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2021-12-13 19:46 UTC (permalink / raw)
To: Lars-Peter Clausen, Bjorn Andersson, Mathieu Poirier,
Rob Herring, Michal Simek
Cc: Laurent Pinchart, Ben Levinsky, Bill Mills, Sergei Korneichuk,
linux-remoteproc, linux-kernel, devicetree, linux-arm-kernel
On 12/14/21 1:13 AM, Lars-Peter Clausen wrote:
> On 12/13/21 8:31 PM, Tanmay Shah wrote:
>>>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> [...]
>>>> +
>>>>
>>>> + i = 0;
>>>> + for_each_available_child_of_node(dev_node, child) {
>>>> + child_pdev = of_find_device_by_node(child);
>>>> + if (!child_pdev)
>>> A return or a break in a for_each_available_child_of_node() will
>>> leak the reference to the child node.
>>
>> Do you mean I have to use of_put_node for each child?
>
> You have to put a `of_put_node(child)` before each break or return in
> the loop.
>
> If you search the kernel commit history for
> for_each_available_child_of_node or for_each_child_of_node you'll find
> many fixes for similar problems. For example
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e1b391e9712db
>
I got it. Same concern was posted by Mathieu also. I will address these
comments in next revision.
Thanks,
Tanmay
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2021-12-07 18:31 ` Mathieu Poirier
@ 2022-02-07 18:57 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2022-02-07 18:57 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On 12/8/21 12:01 AM, Mathieu Poirier wrote:
> On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
>> This driver enables r5f dual core Real time Processing Unit subsystem
>> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
>> (cluster) can be configured in different modes e.g. split mode in which
>> two r5f cores work independent of each other and lock-step mode in which
>> both r5f cores execute same code clock-for-clock and notify if the
>> result is different.
>>
>> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
>> Platform Management Unit that handles the R5 configuration, memory access
>> and R5 lifecycle management. The interface to this manager is done in this
>> driver via zynqmp_pm_* function calls.
>>
>> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
>> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
>> ---
>> drivers/remoteproc/Kconfig | 12 +
>> drivers/remoteproc/Makefile | 1 +
>> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
>> 3 files changed, 972 insertions(+)
>> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>>
>> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
>> index f30d00a3aabe..27f66910d8d3 100644
>> --- a/drivers/remoteproc/Kconfig
>> +++ b/drivers/remoteproc/Kconfig
>> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
>> It's safe to say N here if you're not interested in utilizing
>> a slave processor.
>>
>> +config XLNX_R5_REMOTEPROC
>> + tristate "Xilinx R5 remoteproc support"
>> + depends on PM && ARCH_ZYNQMP
>> + depends on ZYNQMP_FIRMWARE
>> + select RPMSG_VIRTIO
>> + select ZYNQMP_IPI_MBOX
>> + help
>> + Say y or m here to support Xilinx R5 remote processors via the remote
>> + processor framework.
>> +
>> + It's safe to say N if not interested in using RPU r5f cores.
>> +
>> endif # REMOTEPROC
>>
>> endmenu
>> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
>> index bb26c9e4ef9c..334a8bed4c14 100644
>> --- a/drivers/remoteproc/Makefile
>> +++ b/drivers/remoteproc/Makefile
>> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
>> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
>> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
>> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
>> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
>> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> new file mode 100644
>> index 000000000000..c2167fd3869d
>> --- /dev/null
>> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> @@ -0,0 +1,959 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ZynqMP R5 Remote Processor driver
>> + *
>> + */
>> +
>> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
>> +#include <linux/firmware/xlnx-zynqmp.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/kernel.h>
>> +#include <linux/list.h>
>> +#include <linux/mailbox_client.h>
>> +#include <linux/mailbox/zynqmp-ipi-message.h>
>> +#include <linux/module.h>
>> +#include <linux/of_address.h>
>> +#include <linux/of_platform.h>
>> +#include <linux/of_reserved_mem.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/remoteproc.h>
>> +#include <linux/skbuff.h>
>> +#include <linux/sysfs.h>
>> +
>> +#include "remoteproc_internal.h"
>> +
>> +/* settings for RPU cluster mode */
>> +enum zynqmp_r5_cluster_mode {
>> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
>> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
>> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
>> +};
>> +
>> +/**
>> + * struct mem_bank_data - Memory Bank description
>> + *
>> + * @addr: Start address of memory bank
>> + * @size: Size of Memory bank
>> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
>> + * @bank_name: name of the bank for remoteproc framework
>> + */
>> +struct mem_bank_data {
>> + phys_addr_t addr;
>> + size_t size;
>> + enum pm_node_id pm_domain_id;
>> + char *bank_name;
>> +};
>> +
>> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
>> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
>> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
>> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
>> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
>> +};
>> +
>> +/**
>> + * struct zynqmp_r5_core - ZynqMP R5 core structure
>> + *
>> + * @dev: device of RPU instance
>> + * @np: device node of RPU instance
>> + * @tcm_bank_count: number TCM banks accessible to this RPU
>> + * @tcm_banks: array of each TCM bank data
>> + * @res_mem_count: number of Reserved Memory regions per core
>> + * @res_mem: array of reserved memory regions
>> + * @rproc: rproc handle
>> + * @pm_domain_id: RPU CPU power domain id
>> + */
>> +struct zynqmp_r5_core {
>> + struct device *dev;
>> + struct device_node *np;
>> + int tcm_bank_count;
>> + struct mem_bank_data *tcm_banks;
>> + int res_mem_count;
>> + struct reserved_mem *res_mem;
>> + struct rproc *rproc;
>> + enum pm_node_id pm_domain_id;
>> +};
>> +
>> +/**
>> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
>> + *
>> + * @dev: r5f subsystem cluster device node
>> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
>> + * @core_count: number of r5 cores used for this cluster mode
>> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
>> + */
>> +struct zynqmp_r5_cluster {
>> + struct device *dev;
>> + enum zynqmp_r5_cluster_mode mode;
>> + int core_count;
>> + struct zynqmp_r5_core *r5_cores;
>> +};
>> +
>> +/*
>> + * zynqmp_r5_set_mode - set RPU operation mode
>> + *
>> + * set RPU operation mode
>> + *
>> + * Return: 0 for success, negative value for failure
>> + */
>> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
>> + enum zynqmp_r5_cluster_mode rpu_mode)
>> +{
>> + enum rpu_tcm_comb tcm_mode;
>> + int ret, reg_val;
>> +
>> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
>> +
>> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
>> + if (ret < 0) {
>> + pr_err("failed to set RPU mode\n");
>> + return ret;
>> + }
>> +
>> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
>> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
>> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
>> + if (ret < 0)
>> + pr_err("failed to configure TCM\n");
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_start
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Start R5 Core from designated boot address.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + enum rpu_boot_mem bootmem;
>> + int ret;
>> +
>> + if (!r5_core) {
>> + pr_err("can't get r5 core\n");
>> + return -EINVAL;
>> + }
> Please remove.
>
>> +
>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> Some comments as to what is happening here would be appreciated.
>
>> +
>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>> +
>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>> + if (ret)
>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_stop
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Power down R5 Core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + int ret;
>> +
>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret)
>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap memory-region carveout
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + iounmap((void __iomem *)mem->va);
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_mem_regions
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Construct rproc mem carveouts from carveout provided in
>> + * memory-region property
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_mem_regions(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct rproc_mem_entry *mem;
>> + struct reserved_mem *rmem;
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> +
>> + r5_core = rproc->priv;
>> + dev = r5_core->dev;
>> +
>> + /* Register associated reserved memory regions */
>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>> + rmem = &r5_core->res_mem[i];
> Indentation problem
>
>> + mem = rproc_mem_entry_init(dev, NULL,
>> + (dma_addr_t)rmem->base,
>> + rmem->size, rmem->base,
>> + zynqmp_r5_rproc_mem_map,
>> + zynqmp_r5_rproc_mem_unmap,
>> + rmem->name);
>> + if (IS_ERR_OR_NULL(mem))
>> + return -ENOMEM;
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
> Not sure why this wasn't done in zynqmp_r5_get_mem_region_node()... That would
> prevent the need to keep a copy of the reserved memories in the zynqmp_r5_core
> structure.
>
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap TCM banks when powering down R5 core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>> +{
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> + enum pm_node_id pm_domain_id;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + pr_err("r5 core is not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + iounmap((void __iomem *)mem->va);
>> +
>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + if (zynqmp_pm_release_node(pm_domain_id))
>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * tcm_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to initialize the va and da fields of
>> + *
>> + * Given TCM bank entry, this callback will set device address for R5
>> + * running on TCM and also setup virtual address for TCM bank
>> + * remoteproc carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
> Indentation problem
>
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + /* Update memory entry va */
>> + mem->va = (void *)va;
>> +
>> + /* clear TCMs */
>> + memset_io(va, 0, mem->len);
>> +
>> + /*
>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>> + * while on the Linux side they are at 0xffexxxxx.
>> + *
>> + * Zero out the high 12 bits of the address. This will give
>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>> + */
>> + mem->da &= 0x000fffff;
>> +
>> + /*
>> + * TCM Banks 1A and 1B still have to be translated.
>> + *
>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>> + * 0xffeb0000) and convert to the expected relative addresses
>> + * (0x0 and 0x20000).
>> + */
>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>> + mem->da -= 0x90000;
>> +
>> + /* if translated TCM bank address is not valid report error */
>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>> + return -EINVAL;
>> + }
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
> phys_addr_t bank_addr;
>
>> + size_t bank_size = 0;
> No initialisation needed.
>
>> + char *bank_name;
>> + struct device *dev;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
> Remove.
>
>> +
>> + dev = r5_core->dev;
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; i < num_banks; i++) {
>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>> + bank_name = r5_core->tcm_banks[i].bank_name;
>> + bank_size = r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
> I'm pretty sure that if you run the sparse checker on this you will get a lot of
> angry warnings.
>
>
>> + if (IS_ERR_OR_NULL(mem)) {
> Function rproc_mem_entry_init() does not return an error code.
>
>> + /* Turn off all TCM banks turned on before */
>> + do {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> Here @pm_domain_id is casted as a u32 but it isn't above when calling
> zynqmp_pm_request_node(), and that despite both functions taking a "const u32
> node". Again, I'm sure the sparse checker will complain here.
>
>
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x, %x\n",
>> + (u32)pm_domain_id, ret);
>> + } while (i--);
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
> Same as above.
>
>> + char *bank_name;
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
> Same as above.
>
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>> + bank_name = r5_core->tcm_banks[0].bank_name;
>> + for (i = 0; i < num_banks; i++) {
>> + bank_size += r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> + }
> I know what is happening above but in 6 months it will all be gone from
> memory. Please add enough documentation for someone not familiar with the
> platform to understand why things are done that way. Here and throughout the
> driver.
>
>> +
>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + for (i = 0; i < num_banks; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x ret: %x\n",
>> + (u32)pm_domain_id, ret);
>> + }
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_tcm_banks()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Given R5 node in remoteproc instance
>> + * allocate remoteproc carveout for TCM memory
>> + * needed for firmware to be loaded
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
> Remove.
>
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
> Why not simply calling dev_get_drvdata(dev->parent) instead of going through the
> platform API?
>
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_parse_fw()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @fw: ptr to firmware to be loaded onto r5 core
>> + *
>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>> +{
>> + int ret;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + dev_err(&rproc->dev, "r5 core not available\n");
>> + return -EINVAL;
>> + }
> Same as above and throughout.
>
> I am done reviewing this set - the overall structure is sound but there is a
> fair amount of work to be done. There is no need to rush another revision as I
> won't have time to review it before the holidays. On that front, change
> _only_ the things I have commented on. We can address other issues as we move
> forward. Last but not least, please run the sparse checker on your work.
>
> Thanks,
> Mathieu
Hi Mathieu,
I am ready to post v3 in this patch series.
I have developed v3 based on rproc-next. However, when I rebased my work
on tip of linux-next branch,
I faced merge-conflicts.
What is preferred way to proceed in such case? Should I post my patch
based on rproc-next or linux-next?
Or any other way to proceed further?
>> +
>> + dev = r5_core->dev;
>> +
>> + ret = add_tcm_banks(rproc);
>> + if (ret) {
>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = add_mem_regions(rproc);
>> + if (ret)
>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>> +
>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>> + if (ret == -EINVAL) {
>> + /*
>> + * resource table only required for IPC.
>> + * if not present, this is not necessarily an error;
>> + * for example, loading r5 hello world application
>> + * so simply inform user and keep going.
>> + */
>> + dev_info(&rproc->dev, "no resource table found.\n");
>> + ret = 0;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> +
>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int i, j, k;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + dev_dbg(cluster->dev, "Printing dt node info\n");
>> +
>> + pr_debug("cluster mode = %d\n", cluster->mode);
>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("can't get r5_core\n");
>> + continue;
>> + }
>> +
>> + pr_debug("r5 core %d nodes\n", i);
>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>> + k, r5_core->tcm_banks[k].addr,
>> + r5_core->tcm_banks[k].size,
>> + r5_core->tcm_banks[k].pm_domain_id,
>> + r5_core->tcm_banks[k].bank_name);
>> + }
>> +
>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>> +
>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>> + j, r5_core->res_mem[j].base,
>> + r5_core->res_mem[j].size,
>> + r5_core->res_mem[j].name);
>> + }
>> + }
>> +}
>> +
>> +/**
>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>> + * this is called for each individual R5 core to
>> + * set up mailbox, Xilinx platform manager unique ID,
>> + * add to rproc core
>> + *
>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>> +{
>> + int ret;
>> + struct rproc *r5_rproc;
>> + struct device *dev;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* Set up DMA mask */
>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>> + if (ret)
>> + return ret;
>> +
>> + /* Allocate remoteproc instance */
>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>> + NULL, sizeof(struct zynqmp_r5_core));
>> + if (IS_ERR_OR_NULL(r5_rproc))
>> + return -ENOMEM;
>> +
>> + r5_rproc->auto_boot = false;
>> + r5_rproc->priv = r5_core;
>> +
>> + /* Add R5 remoteproc */
>> + ret = devm_rproc_add(dev, r5_rproc);
>> + if (ret) {
>> + pr_err("failed to add r5 remoteproc\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int tcm_bank_count, tcm_node;
>> + int i = 0, j;
>> + struct zynqmp_r5_core *r5_core;
>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>> + struct device *dev = cluster->dev;
>> +
>> + /* ToDo: Use predefined TCM address space values from driver until
>> + * system-dt spec is not final fot TCM
>> + */
>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>> +
>> + /* count per core tcm banks */
>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>> +
>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>> + * 128k. Assign TCM banks to each core accordingly
>> + */
>> + tcm_node = 0;
>> + for (j = 0; j < cluster->core_count; j++) {
>> + r5_core = &cluster->r5_cores[j];
>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>> + tcm_bank_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>> + return -ENOMEM;
>> +
>> + for (i = 0; i < tcm_bank_count; i++) {
>> + /* Use pre-defined TCM reg values.
>> + * Eventually this should be replaced by values
>> + * parsed from dts.
>> + */
>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>> + tcm_node++;
>> + }
>> +
>> + r5_core->tcm_bank_count = tcm_bank_count;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>> +{
>> + int res_mem_count, i;
>> + struct device *dev;
>> + struct device_node *np, *rmem_np;
>> + struct reserved_mem *rmem;
>> +
>> + dev = r5_core->dev;
>> +
>> + np = r5_core->np;
>> + if (IS_ERR_OR_NULL(np)) {
>> + pr_err("invalid device node of r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>> + sizeof(phandle));
>> + if (res_mem_count <= 0) {
>> + dev_warn(dev, "failed to get memory-region property %d\n",
>> + res_mem_count);
>> + return -EINVAL;
>> + }
>> +
>> + r5_core->res_mem = devm_kzalloc(dev,
>> + res_mem_count * sizeof(struct reserved_mem),
>> + GFP_KERNEL);
>> + if (!r5_core->res_mem) {
>> + dev_err(dev, "failed to allocate mem region memory\n");
>> + return -ENOMEM;
>> + }
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int ret, i;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev = cluster->dev;
>> +
>> + ret = zynqmp_r5_get_tcm_node(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("invalid r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>> + if (ret)
>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>> +
>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>> + 1, &r5_core->pm_domain_id);
>> + if (ret) {
>> + dev_err(dev, "failed to get power-domains property\n");
>> + return ret;
>> + }
>> +
>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>> + if (ret) {
>> + dev_err(dev, "failed to init r5 core %d\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + struct device *dev = cluster->dev;
>> + struct device_node *dev_node = dev_of_node(dev);
>> + struct device_node *child;
>> + struct platform_device *child_pdev;
>> + int core_count = 0, ret, i;
>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>> + struct zynqmp_r5_core *r5_cores;
>> +
>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>> +
>> + /* on success returns 0, if not defined then returns -EINVAL,
>> + * In that case, default is LOCKSTEP mode
>> + */
>> + if (ret != -EINVAL && ret != 0) {
>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster_mode == SINGLE_CPU_MODE) {
>> + dev_err(dev, "driver does not support single cpu mode\n");
>> + return -EINVAL;
>> + } else if ((cluster_mode != SPLIT_MODE &&
>> + cluster_mode != LOCKSTEP_MODE)) {
>> + dev_err(dev, "Invalid cluster mode\n");
>> + return -EINVAL;
>> + }
>> +
>> + core_count = of_get_available_child_count(dev_node);
>> + if (core_count <= 0) {
>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>> + return -EINVAL;
>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>> + return -EINVAL;
>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>> + dev_warn(dev, "Only r5 core0 will be used\n");
>> + core_count = 1;
>> + }
>> +
>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>> + core_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_cores)) {
>> + dev_err(dev, "can't allocate memory for cores\n");
>> + return -ENOMEM;
>> + }
>> +
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
>> + return -ENODEV;
>> +
>> + r5_cores[i].dev = &child_pdev->dev;
>> + if (!r5_cores[i].dev) {
>> + pr_err("can't get device for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>> + if (!r5_cores[i].np) {
>> + pr_err("can't get device node for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + i++;
>> + if (i == core_count)
>> + break;
>> + }
>> +
>> + cluster->mode = cluster_mode;
>> + cluster->core_count = core_count;
>> + cluster->r5_cores = r5_cores;
>> +
>> + ret = zynqmp_r5_core_init(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + zynqmp_r5_print_dt_node_info(cluster);
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
>> +
>> + pr_info("Exit r5f subsystem driver\n");
>> +}
>> +
>> +/*
>> + * zynqmp_r5_remoteproc_probe()
>> + *
>> + * @pdev: domain platform device for R5 cluster
>> + *
>> + * called when driver is probed, for each R5 core specified in DT,
>> + * setup as needed to do remoteproc-related operations
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>> +{
>> + int ret;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct device *dev = &pdev->dev;
>> +
>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(cluster))
>> + return -ENOMEM;
>> +
>> + cluster->dev = dev;
>> +
>> + ret = devm_of_platform_populate(dev);
>> + if (ret) {
>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /* wire in so each core can be cleaned up at driver remove */
>> + platform_set_drvdata(pdev, cluster);
>> +
>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_cluster_init(cluster);
>> + if (ret) {
>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>> + return ret;
>> + }
>> +
>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
>> + return 0;
>> +}
>> +
>> +/* Match table for OF platform binding */
>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>> + { .compatible = "xlnx,zynqmp-r5fss", },
>> + { /* end of list */ },
>> +};
>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>> +
>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>> + .probe = zynqmp_r5_remoteproc_probe,
>> + .driver = {
>> + .name = "zynqmp_r5_remoteproc",
>> + .of_match_table = zynqmp_r5_remoteproc_match,
>> + },
>> +};
>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>> +
>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>> +MODULE_AUTHOR("Xilinx Inc.");
>> +MODULE_LICENSE("GPL v2");
>> --
>> 2.25.1
>>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2022-02-07 18:57 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2022-02-07 18:57 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On 12/8/21 12:01 AM, Mathieu Poirier wrote:
> On Mon, Nov 22, 2021 at 10:20:50PM -0800, Tanmay Shah wrote:
>> This driver enables r5f dual core Real time Processing Unit subsystem
>> available on Xilinx Zynq Ultrascale MPSoC Platform. RPU subsystem
>> (cluster) can be configured in different modes e.g. split mode in which
>> two r5f cores work independent of each other and lock-step mode in which
>> both r5f cores execute same code clock-for-clock and notify if the
>> result is different.
>>
>> The Xilinx r5 Remoteproc Driver boots the RPU cores via calls to the Xilinx
>> Platform Management Unit that handles the R5 configuration, memory access
>> and R5 lifecycle management. The interface to this manager is done in this
>> driver via zynqmp_pm_* function calls.
>>
>> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
>> Signed-off-by: Tanmay Shah <tanmay.shah@xilinx.com>
>> ---
>> drivers/remoteproc/Kconfig | 12 +
>> drivers/remoteproc/Makefile | 1 +
>> drivers/remoteproc/xlnx_r5_remoteproc.c | 959 ++++++++++++++++++++++++
>> 3 files changed, 972 insertions(+)
>> create mode 100644 drivers/remoteproc/xlnx_r5_remoteproc.c
>>
>> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
>> index f30d00a3aabe..27f66910d8d3 100644
>> --- a/drivers/remoteproc/Kconfig
>> +++ b/drivers/remoteproc/Kconfig
>> @@ -315,6 +315,18 @@ config TI_K3_R5_REMOTEPROC
>> It's safe to say N here if you're not interested in utilizing
>> a slave processor.
>>
>> +config XLNX_R5_REMOTEPROC
>> + tristate "Xilinx R5 remoteproc support"
>> + depends on PM && ARCH_ZYNQMP
>> + depends on ZYNQMP_FIRMWARE
>> + select RPMSG_VIRTIO
>> + select ZYNQMP_IPI_MBOX
>> + help
>> + Say y or m here to support Xilinx R5 remote processors via the remote
>> + processor framework.
>> +
>> + It's safe to say N if not interested in using RPU r5f cores.
>> +
>> endif # REMOTEPROC
>>
>> endmenu
>> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
>> index bb26c9e4ef9c..334a8bed4c14 100644
>> --- a/drivers/remoteproc/Makefile
>> +++ b/drivers/remoteproc/Makefile
>> @@ -35,3 +35,4 @@ obj-$(CONFIG_ST_SLIM_REMOTEPROC) += st_slim_rproc.o
>> obj-$(CONFIG_STM32_RPROC) += stm32_rproc.o
>> obj-$(CONFIG_TI_K3_DSP_REMOTEPROC) += ti_k3_dsp_remoteproc.o
>> obj-$(CONFIG_TI_K3_R5_REMOTEPROC) += ti_k3_r5_remoteproc.o
>> +obj-$(CONFIG_XLNX_R5_REMOTEPROC) += xlnx_r5_remoteproc.o
>> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> new file mode 100644
>> index 000000000000..c2167fd3869d
>> --- /dev/null
>> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
>> @@ -0,0 +1,959 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * ZynqMP R5 Remote Processor driver
>> + *
>> + */
>> +
>> +#include <dt-bindings/power/xlnx-zynqmp-power.h>
>> +#include <linux/firmware/xlnx-zynqmp.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/kernel.h>
>> +#include <linux/list.h>
>> +#include <linux/mailbox_client.h>
>> +#include <linux/mailbox/zynqmp-ipi-message.h>
>> +#include <linux/module.h>
>> +#include <linux/of_address.h>
>> +#include <linux/of_platform.h>
>> +#include <linux/of_reserved_mem.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/remoteproc.h>
>> +#include <linux/skbuff.h>
>> +#include <linux/sysfs.h>
>> +
>> +#include "remoteproc_internal.h"
>> +
>> +/* settings for RPU cluster mode */
>> +enum zynqmp_r5_cluster_mode {
>> + SPLIT_MODE = 0, // RPU cluster mode when cores run as separate processor
>> + LOCKSTEP_MODE = 1, // cores execute same code in lockstep,clk-for-clk
>> + SINGLE_CPU_MODE = 2, // core0 is held in reset and only core1 runs
>> +};
>> +
>> +/**
>> + * struct mem_bank_data - Memory Bank description
>> + *
>> + * @addr: Start address of memory bank
>> + * @size: Size of Memory bank
>> + * @pm_domain_id: Power-domains id of memory bank for firmware to turn on/off
>> + * @bank_name: name of the bank for remoteproc framework
>> + */
>> +struct mem_bank_data {
>> + phys_addr_t addr;
>> + size_t size;
>> + enum pm_node_id pm_domain_id;
>> + char *bank_name;
>> +};
>> +
>> +static const struct mem_bank_data zynqmp_tcm_banks[] = {
>> + {0xffe00000UL, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
>> + {0xffe20000UL, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
>> + {0xffe90000UL, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
>> + {0xffeb0000UL, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
>> +};
>> +
>> +/**
>> + * struct zynqmp_r5_core - ZynqMP R5 core structure
>> + *
>> + * @dev: device of RPU instance
>> + * @np: device node of RPU instance
>> + * @tcm_bank_count: number TCM banks accessible to this RPU
>> + * @tcm_banks: array of each TCM bank data
>> + * @res_mem_count: number of Reserved Memory regions per core
>> + * @res_mem: array of reserved memory regions
>> + * @rproc: rproc handle
>> + * @pm_domain_id: RPU CPU power domain id
>> + */
>> +struct zynqmp_r5_core {
>> + struct device *dev;
>> + struct device_node *np;
>> + int tcm_bank_count;
>> + struct mem_bank_data *tcm_banks;
>> + int res_mem_count;
>> + struct reserved_mem *res_mem;
>> + struct rproc *rproc;
>> + enum pm_node_id pm_domain_id;
>> +};
>> +
>> +/**
>> + * struct zynqmp_r5_cluster - ZynqMP R5 cluster structure
>> + *
>> + * @dev: r5f subsystem cluster device node
>> + * @mode: cluster mode of type zynqmp_r5_cluster_mode
>> + * @core_count: number of r5 cores used for this cluster mode
>> + * @r5_cores: Array of r5 cores of type struct zynqmp_r5_core
>> + */
>> +struct zynqmp_r5_cluster {
>> + struct device *dev;
>> + enum zynqmp_r5_cluster_mode mode;
>> + int core_count;
>> + struct zynqmp_r5_core *r5_cores;
>> +};
>> +
>> +/*
>> + * zynqmp_r5_set_mode - set RPU operation mode
>> + *
>> + * set RPU operation mode
>> + *
>> + * Return: 0 for success, negative value for failure
>> + */
>> +static int zynqmp_r5_set_mode(struct zynqmp_r5_core *r5_core,
>> + enum zynqmp_r5_cluster_mode rpu_mode)
>> +{
>> + enum rpu_tcm_comb tcm_mode;
>> + int ret, reg_val;
>> +
>> + reg_val = (rpu_mode == LOCKSTEP_MODE ? 0 : 1);
>> +
>> + ret = zynqmp_pm_set_rpu_mode(r5_core->pm_domain_id, reg_val);
>> + if (ret < 0) {
>> + pr_err("failed to set RPU mode\n");
>> + return ret;
>> + }
>> +
>> + tcm_mode = (rpu_mode == LOCKSTEP_MODE) ?
>> + PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
>> + ret = zynqmp_pm_set_tcm_config(r5_core->pm_domain_id, tcm_mode);
>> + if (ret < 0)
>> + pr_err("failed to configure TCM\n");
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_start
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Start R5 Core from designated boot address.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + enum rpu_boot_mem bootmem;
>> + int ret;
>> +
>> + if (!r5_core) {
>> + pr_err("can't get r5 core\n");
>> + return -EINVAL;
>> + }
> Please remove.
>
>> +
>> + bootmem = (rproc->bootaddr >= 0xFFFC0000) ?
>> + PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> Some comments as to what is happening here would be appreciated.
>
>> +
>> + dev_dbg(r5_core->dev, "RPU boot addr 0x%llx from %s.", rproc->bootaddr,
>> + bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
>> +
>> + ret = zynqmp_pm_request_wake(r5_core->pm_domain_id, 1,
>> + bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
>> + if (ret)
>> + pr_err("failed to start RPU = %d\n", r5_core->pm_domain_id);
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_stop
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Power down R5 Core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
>> +{
>> + struct zynqmp_r5_core *r5_core = rproc->priv;
>> + int ret;
>> +
>> + ret = zynqmp_pm_force_pwrdwn(r5_core->pm_domain_id,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret)
>> + pr_err("failed to stop remoteproc RPU %d\n", ret);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to map
>> + *
>> + * Callback to map va for memory-region's carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + mem->va = (void *)va;
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap memory-region carveout
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_rproc_mem_unmap(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
>> +{
>> + iounmap((void __iomem *)mem->va);
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_mem_regions
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Construct rproc mem carveouts from carveout provided in
>> + * memory-region property
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_mem_regions(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct rproc_mem_entry *mem;
>> + struct reserved_mem *rmem;
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> +
>> + r5_core = rproc->priv;
>> + dev = r5_core->dev;
>> +
>> + /* Register associated reserved memory regions */
>> + for (i = 0; i < r5_core->res_mem_count; i++) {
>> + rmem = &r5_core->res_mem[i];
> Indentation problem
>
>> + mem = rproc_mem_entry_init(dev, NULL,
>> + (dma_addr_t)rmem->base,
>> + rmem->size, rmem->base,
>> + zynqmp_r5_rproc_mem_map,
>> + zynqmp_r5_rproc_mem_unmap,
>> + rmem->name);
>> + if (IS_ERR_OR_NULL(mem))
>> + return -ENOMEM;
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
> Not sure why this wasn't done in zynqmp_r5_get_mem_region_node()... That would
> prevent the need to keep a copy of the reserved memories in the zynqmp_r5_core
> structure.
>
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_rproc_mem_unmap
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to unmap
>> + *
>> + * Unmap TCM banks when powering down R5 core.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_unmap(struct rproc *rproc, struct rproc_mem_entry *mem)
>> +{
>> + struct zynqmp_r5_core *r5_core;
>> + int i;
>> + enum pm_node_id pm_domain_id;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + pr_err("r5 core is not available\n");
>> + return -EINVAL;
>> + }
>> +
>> + iounmap((void __iomem *)mem->va);
>> +
>> + for (i = 0; i < r5_core->tcm_bank_count; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + if (zynqmp_pm_release_node(pm_domain_id))
>> + pr_warn("can't turn off TCM bank %d", pm_domain_id);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * tcm_mem_map
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @mem: mem entry to initialize the va and da fields of
>> + *
>> + * Given TCM bank entry, this callback will set device address for R5
>> + * running on TCM and also setup virtual address for TCM bank
>> + * remoteproc carveout.
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int tcm_mem_map(struct rproc *rproc,
>> + struct rproc_mem_entry *mem)
> Indentation problem
>
>> +{
>> + void __iomem *va;
>> +
>> + va = ioremap_wc(mem->dma, mem->len);
>> + if (IS_ERR_OR_NULL(va))
>> + return -ENOMEM;
>> +
>> + /* Update memory entry va */
>> + mem->va = (void *)va;
>> +
>> + /* clear TCMs */
>> + memset_io(va, 0, mem->len);
>> +
>> + /*
>> + * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
>> + * while on the Linux side they are at 0xffexxxxx.
>> + *
>> + * Zero out the high 12 bits of the address. This will give
>> + * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
>> + */
>> + mem->da &= 0x000fffff;
>> +
>> + /*
>> + * TCM Banks 1A and 1B still have to be translated.
>> + *
>> + * Below handle these two banks' absolute addresses (0xffe90000 and
>> + * 0xffeb0000) and convert to the expected relative addresses
>> + * (0x0 and 0x20000).
>> + */
>> + if (mem->da == 0x90000 || mem->da == 0xB0000)
>> + mem->da -= 0x90000;
>> +
>> + /* if translated TCM bank address is not valid report error */
>> + if (mem->da != 0x0 && mem->da != 0x20000) {
>> + dev_err(&rproc->dev, "invalid TCM address: %x\n", mem->da);
>> + return -EINVAL;
>> + }
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_split_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
> phys_addr_t bank_addr;
>
>> + size_t bank_size = 0;
> No initialisation needed.
>
>> + char *bank_name;
>> + struct device *dev;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
> Remove.
>
>> +
>> + dev = r5_core->dev;
>> +
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; i < num_banks; i++) {
>> + bank_addr = (u32)r5_core->tcm_banks[i].addr;
>> + bank_name = r5_core->tcm_banks[i].bank_name;
>> + bank_size = r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> +
>> + dev_dbg(dev, "TCM carveout split mode %s addr=%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
> I'm pretty sure that if you run the sparse checker on this you will get a lot of
> angry warnings.
>
>
>> + if (IS_ERR_OR_NULL(mem)) {
> Function rproc_mem_entry_init() does not return an error code.
>
>> + /* Turn off all TCM banks turned on before */
>> + do {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
> Here @pm_domain_id is casted as a u32 but it isn't above when calling
> zynqmp_pm_request_node(), and that despite both functions taking a "const u32
> node". Again, I'm sure the sparse checker will complain here.
>
>
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x, %x\n",
>> + (u32)pm_domain_id, ret);
>> + } while (i--);
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>> +{
>> + int i, num_banks, ret;
>> + struct rproc_mem_entry *mem;
>> + enum pm_node_id pm_domain_id;
>> + u32 bank_addr;
>> + size_t bank_size = 0;
> Same as above.
>
>> + char *bank_name;
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
>> +
> Same as above.
>
>> + /* go through zynqmp banks for r5 node */
>> + num_banks = r5_core->tcm_bank_count;
>> + if (num_banks <= 0) {
>> + dev_err(dev, "need to specify TCM banks\n");
>> + return -EINVAL;
>> + }
>> +
>> + bank_addr = (u32)r5_core->tcm_banks[0].addr;
>> + bank_name = r5_core->tcm_banks[0].bank_name;
>> + for (i = 0; i < num_banks; i++) {
>> + bank_size += r5_core->tcm_banks[i].size;
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> +
>> + ret = zynqmp_pm_request_node(pm_domain_id,
>> + ZYNQMP_PM_CAPABILITY_ACCESS, 0,
>> + ZYNQMP_PM_REQUEST_ACK_BLOCKING);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to turn on TCM %d", pm_domain_id);
>> + return ret;
>> + }
>> + }
> I know what is happening above but in 6 months it will all be gone from
> memory. Please add enough documentation for someone not familiar with the
> platform to understand why things are done that way. Here and throughout the
> driver.
>
>> +
>> + dev_dbg(dev, "TCM add carveout lockstep mode %s addr=0x%x, size=0x%lx",
>> + bank_name, bank_addr, bank_size);
>> +
>> + /* add carveout */
>> + mem = rproc_mem_entry_init(dev, NULL, bank_addr,
>> + bank_size, bank_addr,
>> + tcm_mem_map, tcm_mem_unmap,
>> + bank_name);
>> + if (IS_ERR_OR_NULL(mem)) {
>> + for (i = 0; i < num_banks; i++) {
>> + pm_domain_id = r5_core->tcm_banks[i].pm_domain_id;
>> + ret = zynqmp_pm_release_node((u32)pm_domain_id);
>> + if (ret)
>> + dev_warn(dev,
>> + "fail to release node: %x ret: %x\n",
>> + (u32)pm_domain_id, ret);
>> + }
>> + return -ENOMEM;
>> + }
>> +
>> + rproc_add_carveout(rproc, mem);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * add_tcm_banks()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + *
>> + * Given R5 node in remoteproc instance
>> + * allocate remoteproc carveout for TCM memory
>> + * needed for firmware to be loaded
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int add_tcm_banks(struct rproc *rproc)
>> +{
>> + struct device *dev;
>> + struct platform_device *parent_pdev;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + r5_core = (struct zynqmp_r5_core *)rproc->priv;
>> + if (!r5_core)
>> + return -EINVAL;
>> +
>> + dev = r5_core->dev;
>> + if (!dev) {
>> + pr_err("r5 core device unavailable\n");
>> + return -ENODEV;
>> + }
> Remove.
>
>> +
>> + parent_pdev = to_platform_device(dev->parent);
>> + if (!parent_pdev) {
>> + dev_err(dev, "parent platform dev unavailable\n");
>> + return -ENODEV;
>> + }
>> +
>> + cluster = platform_get_drvdata(parent_pdev);
>> + if (!cluster) {
>> + dev_err(&parent_pdev->dev, "Invalid driver data\n");
>> + return -EINVAL;
>> + }
> Why not simply calling dev_get_drvdata(dev->parent) instead of going through the
> platform API?
>
>> +
>> + if (cluster->mode == SPLIT_MODE)
>> + return add_tcm_carveout_split_mode(rproc);
>> + else if (cluster->mode == LOCKSTEP_MODE)
>> + return add_tcm_carveout_lockstep_mode(rproc);
>> +
>> + dev_err(cluster->dev, "invalid cluster mode\n");
>> + return -EINVAL;
>> +}
>> +
>> +/*
>> + * zynqmp_r5_parse_fw()
>> + * @rproc: single R5 core's corresponding rproc instance
>> + * @fw: ptr to firmware to be loaded onto r5 core
>> + *
>> + * When loading firmware, ensure the necessary carveouts are in remoteproc
>> + *
>> + * return 0 on success, otherwise non-zero value on failure
>> + */
>> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
>> +{
>> + int ret;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev;
>> +
>> + r5_core = rproc->priv;
>> + if (!r5_core) {
>> + dev_err(&rproc->dev, "r5 core not available\n");
>> + return -EINVAL;
>> + }
> Same as above and throughout.
>
> I am done reviewing this set - the overall structure is sound but there is a
> fair amount of work to be done. There is no need to rush another revision as I
> won't have time to review it before the holidays. On that front, change
> _only_ the things I have commented on. We can address other issues as we move
> forward. Last but not least, please run the sparse checker on your work.
>
> Thanks,
> Mathieu
Hi Mathieu,
I am ready to post v3 in this patch series.
I have developed v3 based on rproc-next. However, when I rebased my work
on tip of linux-next branch,
I faced merge-conflicts.
What is preferred way to proceed in such case? Should I post my patch
based on rproc-next or linux-next?
Or any other way to proceed further?
>> +
>> + dev = r5_core->dev;
>> +
>> + ret = add_tcm_banks(rproc);
>> + if (ret) {
>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = add_mem_regions(rproc);
>> + if (ret)
>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>> +
>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>> + if (ret == -EINVAL) {
>> + /*
>> + * resource table only required for IPC.
>> + * if not present, this is not necessarily an error;
>> + * for example, loading r5 hello world application
>> + * so simply inform user and keep going.
>> + */
>> + dev_info(&rproc->dev, "no resource table found.\n");
>> + ret = 0;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>> + .start = zynqmp_r5_rproc_start,
>> + .stop = zynqmp_r5_rproc_stop,
>> + .load = rproc_elf_load_segments,
>> + .parse_fw = zynqmp_r5_parse_fw,
>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>> + .sanity_check = rproc_elf_sanity_check,
>> + .get_boot_addr = rproc_elf_get_boot_addr,
>> +};
>> +
>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int i, j, k;
>> + struct zynqmp_r5_core *r5_core;
>> +
>> + dev_dbg(cluster->dev, "Printing dt node info\n");
>> +
>> + pr_debug("cluster mode = %d\n", cluster->mode);
>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("can't get r5_core\n");
>> + continue;
>> + }
>> +
>> + pr_debug("r5 core %d nodes\n", i);
>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>> + k, r5_core->tcm_banks[k].addr,
>> + r5_core->tcm_banks[k].size,
>> + r5_core->tcm_banks[k].pm_domain_id,
>> + r5_core->tcm_banks[k].bank_name);
>> + }
>> +
>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>> +
>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>> + j, r5_core->res_mem[j].base,
>> + r5_core->res_mem[j].size,
>> + r5_core->res_mem[j].name);
>> + }
>> + }
>> +}
>> +
>> +/**
>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>> + * this is called for each individual R5 core to
>> + * set up mailbox, Xilinx platform manager unique ID,
>> + * add to rproc core
>> + *
>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>> +{
>> + int ret;
>> + struct rproc *r5_rproc;
>> + struct device *dev;
>> +
>> + dev = r5_core->dev;
>> +
>> + /* Set up DMA mask */
>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>> + if (ret)
>> + return ret;
>> +
>> + /* Allocate remoteproc instance */
>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>> + NULL, sizeof(struct zynqmp_r5_core));
>> + if (IS_ERR_OR_NULL(r5_rproc))
>> + return -ENOMEM;
>> +
>> + r5_rproc->auto_boot = false;
>> + r5_rproc->priv = r5_core;
>> +
>> + /* Add R5 remoteproc */
>> + ret = devm_rproc_add(dev, r5_rproc);
>> + if (ret) {
>> + pr_err("failed to add r5 remoteproc\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int tcm_bank_count, tcm_node;
>> + int i = 0, j;
>> + struct zynqmp_r5_core *r5_core;
>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>> + struct device *dev = cluster->dev;
>> +
>> + /* ToDo: Use predefined TCM address space values from driver until
>> + * system-dt spec is not final fot TCM
>> + */
>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>> +
>> + /* count per core tcm banks */
>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>> +
>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>> + * 128k. Assign TCM banks to each core accordingly
>> + */
>> + tcm_node = 0;
>> + for (j = 0; j < cluster->core_count; j++) {
>> + r5_core = &cluster->r5_cores[j];
>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>> + tcm_bank_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>> + return -ENOMEM;
>> +
>> + for (i = 0; i < tcm_bank_count; i++) {
>> + /* Use pre-defined TCM reg values.
>> + * Eventually this should be replaced by values
>> + * parsed from dts.
>> + */
>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>> + tcm_node++;
>> + }
>> +
>> + r5_core->tcm_bank_count = tcm_bank_count;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>> +{
>> + int res_mem_count, i;
>> + struct device *dev;
>> + struct device_node *np, *rmem_np;
>> + struct reserved_mem *rmem;
>> +
>> + dev = r5_core->dev;
>> +
>> + np = r5_core->np;
>> + if (IS_ERR_OR_NULL(np)) {
>> + pr_err("invalid device node of r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>> + sizeof(phandle));
>> + if (res_mem_count <= 0) {
>> + dev_warn(dev, "failed to get memory-region property %d\n",
>> + res_mem_count);
>> + return -EINVAL;
>> + }
>> +
>> + r5_core->res_mem = devm_kzalloc(dev,
>> + res_mem_count * sizeof(struct reserved_mem),
>> + GFP_KERNEL);
>> + if (!r5_core->res_mem) {
>> + dev_err(dev, "failed to allocate mem region memory\n");
>> + return -ENOMEM;
>> + }
>> +
>> + for (i = 0; i < res_mem_count; i++) {
>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>> + if (!rmem_np)
>> + return -EINVAL;
>> +
>> + rmem = of_reserved_mem_lookup(rmem_np);
>> + if (!rmem) {
>> + of_node_put(rmem_np);
>> + return -EINVAL;
>> + }
>> +
>> + memcpy(&r5_core->res_mem[i], rmem,
>> + sizeof(struct reserved_mem));
>> + of_node_put(rmem_np);
>> + }
>> +
>> + r5_core->res_mem_count = res_mem_count;
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + int ret, i;
>> + struct zynqmp_r5_core *r5_core;
>> + struct device *dev = cluster->dev;
>> +
>> + ret = zynqmp_r5_get_tcm_node(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + for (i = 0; i < cluster->core_count; i++) {
>> + r5_core = &cluster->r5_cores[i];
>> + if (!r5_core) {
>> + pr_err("invalid r5 core\n");
>> + return -EINVAL;
>> + }
>> +
>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>> + if (ret)
>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>> +
>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>> + 1, &r5_core->pm_domain_id);
>> + if (ret) {
>> + dev_err(dev, "failed to get power-domains property\n");
>> + return ret;
>> + }
>> +
>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>> + if (ret) {
>> + dev_err(dev, "failed to init r5 core %d\n", i);
>> + return ret;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>> +{
>> + struct device *dev = cluster->dev;
>> + struct device_node *dev_node = dev_of_node(dev);
>> + struct device_node *child;
>> + struct platform_device *child_pdev;
>> + int core_count = 0, ret, i;
>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>> + struct zynqmp_r5_core *r5_cores;
>> +
>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>> +
>> + /* on success returns 0, if not defined then returns -EINVAL,
>> + * In that case, default is LOCKSTEP mode
>> + */
>> + if (ret != -EINVAL && ret != 0) {
>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>> + return -EINVAL;
>> + }
>> +
>> + if (cluster_mode == SINGLE_CPU_MODE) {
>> + dev_err(dev, "driver does not support single cpu mode\n");
>> + return -EINVAL;
>> + } else if ((cluster_mode != SPLIT_MODE &&
>> + cluster_mode != LOCKSTEP_MODE)) {
>> + dev_err(dev, "Invalid cluster mode\n");
>> + return -EINVAL;
>> + }
>> +
>> + core_count = of_get_available_child_count(dev_node);
>> + if (core_count <= 0) {
>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>> + return -EINVAL;
>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>> + return -EINVAL;
>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>> + dev_warn(dev, "Only r5 core0 will be used\n");
>> + core_count = 1;
>> + }
>> +
>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>> + core_count, GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(r5_cores)) {
>> + dev_err(dev, "can't allocate memory for cores\n");
>> + return -ENOMEM;
>> + }
>> +
>> + i = 0;
>> + for_each_available_child_of_node(dev_node, child) {
>> + child_pdev = of_find_device_by_node(child);
>> + if (!child_pdev)
>> + return -ENODEV;
>> +
>> + r5_cores[i].dev = &child_pdev->dev;
>> + if (!r5_cores[i].dev) {
>> + pr_err("can't get device for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>> + if (!r5_cores[i].np) {
>> + pr_err("can't get device node for r5 core %d\n", i);
>> + return -ENODEV;
>> + }
>> +
>> + i++;
>> + if (i == core_count)
>> + break;
>> + }
>> +
>> + cluster->mode = cluster_mode;
>> + cluster->core_count = core_count;
>> + cluster->r5_cores = r5_cores;
>> +
>> + ret = zynqmp_r5_core_init(cluster);
>> + if (ret < 0) {
>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>> + return ret;
>> + }
>> +
>> + zynqmp_r5_print_dt_node_info(cluster);
>> +
>> + return 0;
>> +}
>> +
>> +static void zynqmp_r5_cluster_exit(void *data)
>> +{
>> + struct platform_device *pdev = (struct platform_device *)data;
>> +
>> + platform_set_drvdata(pdev, NULL);
>> +
>> + pr_info("Exit r5f subsystem driver\n");
>> +}
>> +
>> +/*
>> + * zynqmp_r5_remoteproc_probe()
>> + *
>> + * @pdev: domain platform device for R5 cluster
>> + *
>> + * called when driver is probed, for each R5 core specified in DT,
>> + * setup as needed to do remoteproc-related operations
>> + *
>> + * Return: 0 for success, negative value for failure.
>> + */
>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>> +{
>> + int ret;
>> + struct zynqmp_r5_cluster *cluster;
>> + struct device *dev = &pdev->dev;
>> +
>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>> + if (IS_ERR_OR_NULL(cluster))
>> + return -ENOMEM;
>> +
>> + cluster->dev = dev;
>> +
>> + ret = devm_of_platform_populate(dev);
>> + if (ret) {
>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>> + return ret;
>> + }
>> +
>> + /* wire in so each core can be cleaned up at driver remove */
>> + platform_set_drvdata(pdev, cluster);
>> +
>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>> + if (ret)
>> + return ret;
>> +
>> + ret = zynqmp_r5_cluster_init(cluster);
>> + if (ret) {
>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>> + return ret;
>> + }
>> +
>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
>> + return 0;
>> +}
>> +
>> +/* Match table for OF platform binding */
>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>> + { .compatible = "xlnx,zynqmp-r5fss", },
>> + { /* end of list */ },
>> +};
>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>> +
>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>> + .probe = zynqmp_r5_remoteproc_probe,
>> + .driver = {
>> + .name = "zynqmp_r5_remoteproc",
>> + .of_match_table = zynqmp_r5_remoteproc_match,
>> + },
>> +};
>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>> +
>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>> +MODULE_AUTHOR("Xilinx Inc.");
>> +MODULE_LICENSE("GPL v2");
>> --
>> 2.25.1
>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2022-02-07 18:57 ` Tanmay Shah
@ 2022-02-08 16:54 ` Mathieu Poirier
-1 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2022-02-08 16:54 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
Hi Tanmay,
[...]
> >
> > I am done reviewing this set - the overall structure is sound but there is a
> > fair amount of work to be done. There is no need to rush another revision as I
> > won't have time to review it before the holidays. On that front, change
> > _only_ the things I have commented on. We can address other issues as we move
> > forward. Last but not least, please run the sparse checker on your work.
> >
> > Thanks,
> > Mathieu
>
> Hi Mathieu,
>
> I am ready to post v3 in this patch series.
>
> I have developed v3 based on rproc-next. However, when I rebased my work on
> tip of linux-next branch,
Are the merge conflicts related to drivers/remoteproc or drivers/rpmsg? That
would be surprising since the code that is in rproc-next and rpmsg-next is also
in linux-next.
In any case I would suggest to base your v3 on _today's_ for-next branch [1] and
explicitly mention any dependency in the cover letter. From there I might be
able to see where the problems are and advise further.
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/remoteproc/linux.git/log/?h=for-next
>
> I faced merge-conflicts.
>
> What is preferred way to proceed in such case? Should I post my patch based
> on rproc-next or linux-next?
>
> Or any other way to proceed further?
>
>
> > > +
> > > + dev = r5_core->dev;
> > > +
> > > + ret = add_tcm_banks(rproc);
> > > + if (ret) {
> > > + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + ret = add_mem_regions(rproc);
> > > + if (ret)
> > > + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
> > > +
> > > + ret = rproc_elf_load_rsc_table(rproc, fw);
> > > + if (ret == -EINVAL) {
> > > + /*
> > > + * resource table only required for IPC.
> > > + * if not present, this is not necessarily an error;
> > > + * for example, loading r5 hello world application
> > > + * so simply inform user and keep going.
> > > + */
> > > + dev_info(&rproc->dev, "no resource table found.\n");
> > > + ret = 0;
> > > + }
> > > + return ret;
> > > +}
> > > +
> > > +static struct rproc_ops zynqmp_r5_rproc_ops = {
> > > + .start = zynqmp_r5_rproc_start,
> > > + .stop = zynqmp_r5_rproc_stop,
> > > + .load = rproc_elf_load_segments,
> > > + .parse_fw = zynqmp_r5_parse_fw,
> > > + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > > + .sanity_check = rproc_elf_sanity_check,
> > > + .get_boot_addr = rproc_elf_get_boot_addr,
> > > +};
> > > +
> > > +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + int i, j, k;
> > > + struct zynqmp_r5_core *r5_core;
> > > +
> > > + dev_dbg(cluster->dev, "Printing dt node info\n");
> > > +
> > > + pr_debug("cluster mode = %d\n", cluster->mode);
> > > + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
> > > + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
> > > + pr_debug("r5f num cores = %d\n", cluster->core_count);
> > > +
> > > + for (i = 0; i < cluster->core_count; i++) {
> > > + r5_core = &cluster->r5_cores[i];
> > > + if (!r5_core) {
> > > + pr_err("can't get r5_core\n");
> > > + continue;
> > > + }
> > > +
> > > + pr_debug("r5 core %d nodes\n", i);
> > > + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
> > > + for (k = 0; k < r5_core->tcm_bank_count; k++) {
> > > + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
> > > + k, r5_core->tcm_banks[k].addr,
> > > + r5_core->tcm_banks[k].size,
> > > + r5_core->tcm_banks[k].pm_domain_id,
> > > + r5_core->tcm_banks[k].bank_name);
> > > + }
> > > +
> > > + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
> > > +
> > > + for (j = 0; j < r5_core->res_mem_count; j++) {
> > > + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
> > > + j, r5_core->res_mem[j].base,
> > > + r5_core->res_mem[j].size,
> > > + r5_core->res_mem[j].name);
> > > + }
> > > + }
> > > +}
> > > +
> > > +/**
> > > + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
> > > + * this is called for each individual R5 core to
> > > + * set up mailbox, Xilinx platform manager unique ID,
> > > + * add to rproc core
> > > + *
> > > + * @r5_core: zynqmp_r5_core r5 core object to initialize
> > > + *
> > > + * Return: 0 for success, negative value for failure.
> > > + */
> > > +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
> > > +{
> > > + int ret;
> > > + struct rproc *r5_rproc;
> > > + struct device *dev;
> > > +
> > > + dev = r5_core->dev;
> > > +
> > > + /* Set up DMA mask */
> > > + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> > > + if (ret)
> > > + return ret;
> > > +
> > > + /* Allocate remoteproc instance */
> > > + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> > > + NULL, sizeof(struct zynqmp_r5_core));
> > > + if (IS_ERR_OR_NULL(r5_rproc))
> > > + return -ENOMEM;
> > > +
> > > + r5_rproc->auto_boot = false;
> > > + r5_rproc->priv = r5_core;
> > > +
> > > + /* Add R5 remoteproc */
> > > + ret = devm_rproc_add(dev, r5_rproc);
> > > + if (ret) {
> > > + pr_err("failed to add r5 remoteproc\n");
> > > + return ret;
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + int tcm_bank_count, tcm_node;
> > > + int i = 0, j;
> > > + struct zynqmp_r5_core *r5_core;
> > > + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> > > + struct device *dev = cluster->dev;
> > > +
> > > + /* ToDo: Use predefined TCM address space values from driver until
> > > + * system-dt spec is not final fot TCM
> > > + */
> > > + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> > > +
> > > + /* count per core tcm banks */
> > > + tcm_bank_count = tcm_bank_count / cluster->core_count;
> > > +
> > > + /* r5 core 0 will use all of TCM banks in lockstep mode.
> > > + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> > > + * 128k. Assign TCM banks to each core accordingly
> > > + */
> > > + tcm_node = 0;
> > > + for (j = 0; j < cluster->core_count; j++) {
> > > + r5_core = &cluster->r5_cores[j];
> > > + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> > > + tcm_bank_count, GFP_KERNEL);
> > > + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> > > + return -ENOMEM;
> > > +
> > > + for (i = 0; i < tcm_bank_count; i++) {
> > > + /* Use pre-defined TCM reg values.
> > > + * Eventually this should be replaced by values
> > > + * parsed from dts.
> > > + */
> > > + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> > > + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> > > + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> > > + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> > > + tcm_node++;
> > > + }
> > > +
> > > + r5_core->tcm_bank_count = tcm_bank_count;
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> > > +{
> > > + int res_mem_count, i;
> > > + struct device *dev;
> > > + struct device_node *np, *rmem_np;
> > > + struct reserved_mem *rmem;
> > > +
> > > + dev = r5_core->dev;
> > > +
> > > + np = r5_core->np;
> > > + if (IS_ERR_OR_NULL(np)) {
> > > + pr_err("invalid device node of r5 core\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> > > + sizeof(phandle));
> > > + if (res_mem_count <= 0) {
> > > + dev_warn(dev, "failed to get memory-region property %d\n",
> > > + res_mem_count);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + r5_core->res_mem = devm_kzalloc(dev,
> > > + res_mem_count * sizeof(struct reserved_mem),
> > > + GFP_KERNEL);
> > > + if (!r5_core->res_mem) {
> > > + dev_err(dev, "failed to allocate mem region memory\n");
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + for (i = 0; i < res_mem_count; i++) {
> > > + rmem_np = of_parse_phandle(np, "memory-region", i);
> > > + if (!rmem_np)
> > > + return -EINVAL;
> > > +
> > > + rmem = of_reserved_mem_lookup(rmem_np);
> > > + if (!rmem) {
> > > + of_node_put(rmem_np);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + memcpy(&r5_core->res_mem[i], rmem,
> > > + sizeof(struct reserved_mem));
> > > + of_node_put(rmem_np);
> > > + }
> > > +
> > > + r5_core->res_mem_count = res_mem_count;
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + int ret, i;
> > > + struct zynqmp_r5_core *r5_core;
> > > + struct device *dev = cluster->dev;
> > > +
> > > + ret = zynqmp_r5_get_tcm_node(cluster);
> > > + if (ret < 0) {
> > > + dev_err(dev, "can't get tcm node, err %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + for (i = 0; i < cluster->core_count; i++) {
> > > + r5_core = &cluster->r5_cores[i];
> > > + if (!r5_core) {
> > > + pr_err("invalid r5 core\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + ret = zynqmp_r5_get_mem_region_node(r5_core);
> > > + if (ret)
> > > + dev_warn(dev, "memory-region prop failed %d\n", ret);
> > > +
> > > + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> > > + 1, &r5_core->pm_domain_id);
> > > + if (ret) {
> > > + dev_err(dev, "failed to get power-domains property\n");
> > > + return ret;
> > > + }
> > > +
> > > + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> > > + if (ret)
> > > + return ret;
> > > +
> > > + ret = zynqmp_r5_add_rproc_core(r5_core);
> > > + if (ret) {
> > > + dev_err(dev, "failed to init r5 core %d\n", i);
> > > + return ret;
> > > + }
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + struct device *dev = cluster->dev;
> > > + struct device_node *dev_node = dev_of_node(dev);
> > > + struct device_node *child;
> > > + struct platform_device *child_pdev;
> > > + int core_count = 0, ret, i;
> > > + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> > > + struct zynqmp_r5_core *r5_cores;
> > > +
> > > + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> > > +
> > > + /* on success returns 0, if not defined then returns -EINVAL,
> > > + * In that case, default is LOCKSTEP mode
> > > + */
> > > + if (ret != -EINVAL && ret != 0) {
> > > + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + if (cluster_mode == SINGLE_CPU_MODE) {
> > > + dev_err(dev, "driver does not support single cpu mode\n");
> > > + return -EINVAL;
> > > + } else if ((cluster_mode != SPLIT_MODE &&
> > > + cluster_mode != LOCKSTEP_MODE)) {
> > > + dev_err(dev, "Invalid cluster mode\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + core_count = of_get_available_child_count(dev_node);
> > > + if (core_count <= 0) {
> > > + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> > > + return -EINVAL;
> > > + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> > > + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> > > + return -EINVAL;
> > > + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> > > + dev_warn(dev, "Only r5 core0 will be used\n");
> > > + core_count = 1;
> > > + }
> > > +
> > > + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> > > + core_count, GFP_KERNEL);
> > > + if (IS_ERR_OR_NULL(r5_cores)) {
> > > + dev_err(dev, "can't allocate memory for cores\n");
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + i = 0;
> > > + for_each_available_child_of_node(dev_node, child) {
> > > + child_pdev = of_find_device_by_node(child);
> > > + if (!child_pdev)
> > > + return -ENODEV;
> > > +
> > > + r5_cores[i].dev = &child_pdev->dev;
> > > + if (!r5_cores[i].dev) {
> > > + pr_err("can't get device for r5 core %d\n", i);
> > > + return -ENODEV;
> > > + }
> > > +
> > > + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> > > + if (!r5_cores[i].np) {
> > > + pr_err("can't get device node for r5 core %d\n", i);
> > > + return -ENODEV;
> > > + }
> > > +
> > > + i++;
> > > + if (i == core_count)
> > > + break;
> > > + }
> > > +
> > > + cluster->mode = cluster_mode;
> > > + cluster->core_count = core_count;
> > > + cluster->r5_cores = r5_cores;
> > > +
> > > + ret = zynqmp_r5_core_init(cluster);
> > > + if (ret < 0) {
> > > + dev_err(dev, "failed to init r5 core err %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + zynqmp_r5_print_dt_node_info(cluster);
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static void zynqmp_r5_cluster_exit(void *data)
> > > +{
> > > + struct platform_device *pdev = (struct platform_device *)data;
> > > +
> > > + platform_set_drvdata(pdev, NULL);
> > > +
> > > + pr_info("Exit r5f subsystem driver\n");
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_remoteproc_probe()
> > > + *
> > > + * @pdev: domain platform device for R5 cluster
> > > + *
> > > + * called when driver is probed, for each R5 core specified in DT,
> > > + * setup as needed to do remoteproc-related operations
> > > + *
> > > + * Return: 0 for success, negative value for failure.
> > > + */
> > > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > > +{
> > > + int ret;
> > > + struct zynqmp_r5_cluster *cluster;
> > > + struct device *dev = &pdev->dev;
> > > +
> > > + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > > + if (IS_ERR_OR_NULL(cluster))
> > > + return -ENOMEM;
> > > +
> > > + cluster->dev = dev;
> > > +
> > > + ret = devm_of_platform_populate(dev);
> > > + if (ret) {
> > > + dev_err(dev, "failed to populate platform dev %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + /* wire in so each core can be cleaned up at driver remove */
> > > + platform_set_drvdata(pdev, cluster);
> > > +
> > > + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> > > + if (ret)
> > > + return ret;
> > > +
> > > + ret = zynqmp_r5_cluster_init(cluster);
> > > + if (ret) {
> > > + dev_err(dev, "Invalid r5f subsystem device tree\n");
> > > + return ret;
> > > + }
> > > +
> > > + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> > > + return 0;
> > > +}
> > > +
> > > +/* Match table for OF platform binding */
> > > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > > + { .compatible = "xlnx,zynqmp-r5fss", },
> > > + { /* end of list */ },
> > > +};
> > > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > > +
> > > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > > + .probe = zynqmp_r5_remoteproc_probe,
> > > + .driver = {
> > > + .name = "zynqmp_r5_remoteproc",
> > > + .of_match_table = zynqmp_r5_remoteproc_match,
> > > + },
> > > +};
> > > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > > +
> > > +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> > > +MODULE_AUTHOR("Xilinx Inc.");
> > > +MODULE_LICENSE("GPL v2");
> > > --
> > > 2.25.1
> > >
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2022-02-08 16:54 ` Mathieu Poirier
0 siblings, 0 replies; 56+ messages in thread
From: Mathieu Poirier @ 2022-02-08 16:54 UTC (permalink / raw)
To: Tanmay Shah
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
Hi Tanmay,
[...]
> >
> > I am done reviewing this set - the overall structure is sound but there is a
> > fair amount of work to be done. There is no need to rush another revision as I
> > won't have time to review it before the holidays. On that front, change
> > _only_ the things I have commented on. We can address other issues as we move
> > forward. Last but not least, please run the sparse checker on your work.
> >
> > Thanks,
> > Mathieu
>
> Hi Mathieu,
>
> I am ready to post v3 in this patch series.
>
> I have developed v3 based on rproc-next. However, when I rebased my work on
> tip of linux-next branch,
Are the merge conflicts related to drivers/remoteproc or drivers/rpmsg? That
would be surprising since the code that is in rproc-next and rpmsg-next is also
in linux-next.
In any case I would suggest to base your v3 on _today's_ for-next branch [1] and
explicitly mention any dependency in the cover letter. From there I might be
able to see where the problems are and advise further.
[1]. https://git.kernel.org/pub/scm/linux/kernel/git/remoteproc/linux.git/log/?h=for-next
>
> I faced merge-conflicts.
>
> What is preferred way to proceed in such case? Should I post my patch based
> on rproc-next or linux-next?
>
> Or any other way to proceed further?
>
>
> > > +
> > > + dev = r5_core->dev;
> > > +
> > > + ret = add_tcm_banks(rproc);
> > > + if (ret) {
> > > + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + ret = add_mem_regions(rproc);
> > > + if (ret)
> > > + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
> > > +
> > > + ret = rproc_elf_load_rsc_table(rproc, fw);
> > > + if (ret == -EINVAL) {
> > > + /*
> > > + * resource table only required for IPC.
> > > + * if not present, this is not necessarily an error;
> > > + * for example, loading r5 hello world application
> > > + * so simply inform user and keep going.
> > > + */
> > > + dev_info(&rproc->dev, "no resource table found.\n");
> > > + ret = 0;
> > > + }
> > > + return ret;
> > > +}
> > > +
> > > +static struct rproc_ops zynqmp_r5_rproc_ops = {
> > > + .start = zynqmp_r5_rproc_start,
> > > + .stop = zynqmp_r5_rproc_stop,
> > > + .load = rproc_elf_load_segments,
> > > + .parse_fw = zynqmp_r5_parse_fw,
> > > + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > > + .sanity_check = rproc_elf_sanity_check,
> > > + .get_boot_addr = rproc_elf_get_boot_addr,
> > > +};
> > > +
> > > +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + int i, j, k;
> > > + struct zynqmp_r5_core *r5_core;
> > > +
> > > + dev_dbg(cluster->dev, "Printing dt node info\n");
> > > +
> > > + pr_debug("cluster mode = %d\n", cluster->mode);
> > > + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
> > > + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
> > > + pr_debug("r5f num cores = %d\n", cluster->core_count);
> > > +
> > > + for (i = 0; i < cluster->core_count; i++) {
> > > + r5_core = &cluster->r5_cores[i];
> > > + if (!r5_core) {
> > > + pr_err("can't get r5_core\n");
> > > + continue;
> > > + }
> > > +
> > > + pr_debug("r5 core %d nodes\n", i);
> > > + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
> > > + for (k = 0; k < r5_core->tcm_bank_count; k++) {
> > > + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
> > > + k, r5_core->tcm_banks[k].addr,
> > > + r5_core->tcm_banks[k].size,
> > > + r5_core->tcm_banks[k].pm_domain_id,
> > > + r5_core->tcm_banks[k].bank_name);
> > > + }
> > > +
> > > + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
> > > +
> > > + for (j = 0; j < r5_core->res_mem_count; j++) {
> > > + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
> > > + j, r5_core->res_mem[j].base,
> > > + r5_core->res_mem[j].size,
> > > + r5_core->res_mem[j].name);
> > > + }
> > > + }
> > > +}
> > > +
> > > +/**
> > > + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
> > > + * this is called for each individual R5 core to
> > > + * set up mailbox, Xilinx platform manager unique ID,
> > > + * add to rproc core
> > > + *
> > > + * @r5_core: zynqmp_r5_core r5 core object to initialize
> > > + *
> > > + * Return: 0 for success, negative value for failure.
> > > + */
> > > +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
> > > +{
> > > + int ret;
> > > + struct rproc *r5_rproc;
> > > + struct device *dev;
> > > +
> > > + dev = r5_core->dev;
> > > +
> > > + /* Set up DMA mask */
> > > + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> > > + if (ret)
> > > + return ret;
> > > +
> > > + /* Allocate remoteproc instance */
> > > + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> > > + NULL, sizeof(struct zynqmp_r5_core));
> > > + if (IS_ERR_OR_NULL(r5_rproc))
> > > + return -ENOMEM;
> > > +
> > > + r5_rproc->auto_boot = false;
> > > + r5_rproc->priv = r5_core;
> > > +
> > > + /* Add R5 remoteproc */
> > > + ret = devm_rproc_add(dev, r5_rproc);
> > > + if (ret) {
> > > + pr_err("failed to add r5 remoteproc\n");
> > > + return ret;
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + int tcm_bank_count, tcm_node;
> > > + int i = 0, j;
> > > + struct zynqmp_r5_core *r5_core;
> > > + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
> > > + struct device *dev = cluster->dev;
> > > +
> > > + /* ToDo: Use predefined TCM address space values from driver until
> > > + * system-dt spec is not final fot TCM
> > > + */
> > > + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
> > > +
> > > + /* count per core tcm banks */
> > > + tcm_bank_count = tcm_bank_count / cluster->core_count;
> > > +
> > > + /* r5 core 0 will use all of TCM banks in lockstep mode.
> > > + * In split mode, r5 core0 will use 128k and r5 core1 will use another
> > > + * 128k. Assign TCM banks to each core accordingly
> > > + */
> > > + tcm_node = 0;
> > > + for (j = 0; j < cluster->core_count; j++) {
> > > + r5_core = &cluster->r5_cores[j];
> > > + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
> > > + tcm_bank_count, GFP_KERNEL);
> > > + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
> > > + return -ENOMEM;
> > > +
> > > + for (i = 0; i < tcm_bank_count; i++) {
> > > + /* Use pre-defined TCM reg values.
> > > + * Eventually this should be replaced by values
> > > + * parsed from dts.
> > > + */
> > > + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
> > > + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
> > > + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
> > > + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
> > > + tcm_node++;
> > > + }
> > > +
> > > + r5_core->tcm_bank_count = tcm_bank_count;
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
> > > +{
> > > + int res_mem_count, i;
> > > + struct device *dev;
> > > + struct device_node *np, *rmem_np;
> > > + struct reserved_mem *rmem;
> > > +
> > > + dev = r5_core->dev;
> > > +
> > > + np = r5_core->np;
> > > + if (IS_ERR_OR_NULL(np)) {
> > > + pr_err("invalid device node of r5 core\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
> > > + sizeof(phandle));
> > > + if (res_mem_count <= 0) {
> > > + dev_warn(dev, "failed to get memory-region property %d\n",
> > > + res_mem_count);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + r5_core->res_mem = devm_kzalloc(dev,
> > > + res_mem_count * sizeof(struct reserved_mem),
> > > + GFP_KERNEL);
> > > + if (!r5_core->res_mem) {
> > > + dev_err(dev, "failed to allocate mem region memory\n");
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + for (i = 0; i < res_mem_count; i++) {
> > > + rmem_np = of_parse_phandle(np, "memory-region", i);
> > > + if (!rmem_np)
> > > + return -EINVAL;
> > > +
> > > + rmem = of_reserved_mem_lookup(rmem_np);
> > > + if (!rmem) {
> > > + of_node_put(rmem_np);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + memcpy(&r5_core->res_mem[i], rmem,
> > > + sizeof(struct reserved_mem));
> > > + of_node_put(rmem_np);
> > > + }
> > > +
> > > + r5_core->res_mem_count = res_mem_count;
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + int ret, i;
> > > + struct zynqmp_r5_core *r5_core;
> > > + struct device *dev = cluster->dev;
> > > +
> > > + ret = zynqmp_r5_get_tcm_node(cluster);
> > > + if (ret < 0) {
> > > + dev_err(dev, "can't get tcm node, err %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + for (i = 0; i < cluster->core_count; i++) {
> > > + r5_core = &cluster->r5_cores[i];
> > > + if (!r5_core) {
> > > + pr_err("invalid r5 core\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + ret = zynqmp_r5_get_mem_region_node(r5_core);
> > > + if (ret)
> > > + dev_warn(dev, "memory-region prop failed %d\n", ret);
> > > +
> > > + ret = of_property_read_u32_index(r5_core->np, "power-domains",
> > > + 1, &r5_core->pm_domain_id);
> > > + if (ret) {
> > > + dev_err(dev, "failed to get power-domains property\n");
> > > + return ret;
> > > + }
> > > +
> > > + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
> > > + if (ret)
> > > + return ret;
> > > +
> > > + ret = zynqmp_r5_add_rproc_core(r5_core);
> > > + if (ret) {
> > > + dev_err(dev, "failed to init r5 core %d\n", i);
> > > + return ret;
> > > + }
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > + struct device *dev = cluster->dev;
> > > + struct device_node *dev_node = dev_of_node(dev);
> > > + struct device_node *child;
> > > + struct platform_device *child_pdev;
> > > + int core_count = 0, ret, i;
> > > + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
> > > + struct zynqmp_r5_core *r5_cores;
> > > +
> > > + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
> > > +
> > > + /* on success returns 0, if not defined then returns -EINVAL,
> > > + * In that case, default is LOCKSTEP mode
> > > + */
> > > + if (ret != -EINVAL && ret != 0) {
> > > + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + if (cluster_mode == SINGLE_CPU_MODE) {
> > > + dev_err(dev, "driver does not support single cpu mode\n");
> > > + return -EINVAL;
> > > + } else if ((cluster_mode != SPLIT_MODE &&
> > > + cluster_mode != LOCKSTEP_MODE)) {
> > > + dev_err(dev, "Invalid cluster mode\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + core_count = of_get_available_child_count(dev_node);
> > > + if (core_count <= 0) {
> > > + dev_err(dev, "Invalid number of r5 cores %d", core_count);
> > > + return -EINVAL;
> > > + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
> > > + dev_err(dev, "Invalid number of r5 cores for split mode\n");
> > > + return -EINVAL;
> > > + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
> > > + dev_warn(dev, "Only r5 core0 will be used\n");
> > > + core_count = 1;
> > > + }
> > > +
> > > + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
> > > + core_count, GFP_KERNEL);
> > > + if (IS_ERR_OR_NULL(r5_cores)) {
> > > + dev_err(dev, "can't allocate memory for cores\n");
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + i = 0;
> > > + for_each_available_child_of_node(dev_node, child) {
> > > + child_pdev = of_find_device_by_node(child);
> > > + if (!child_pdev)
> > > + return -ENODEV;
> > > +
> > > + r5_cores[i].dev = &child_pdev->dev;
> > > + if (!r5_cores[i].dev) {
> > > + pr_err("can't get device for r5 core %d\n", i);
> > > + return -ENODEV;
> > > + }
> > > +
> > > + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
> > > + if (!r5_cores[i].np) {
> > > + pr_err("can't get device node for r5 core %d\n", i);
> > > + return -ENODEV;
> > > + }
> > > +
> > > + i++;
> > > + if (i == core_count)
> > > + break;
> > > + }
> > > +
> > > + cluster->mode = cluster_mode;
> > > + cluster->core_count = core_count;
> > > + cluster->r5_cores = r5_cores;
> > > +
> > > + ret = zynqmp_r5_core_init(cluster);
> > > + if (ret < 0) {
> > > + dev_err(dev, "failed to init r5 core err %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + zynqmp_r5_print_dt_node_info(cluster);
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static void zynqmp_r5_cluster_exit(void *data)
> > > +{
> > > + struct platform_device *pdev = (struct platform_device *)data;
> > > +
> > > + platform_set_drvdata(pdev, NULL);
> > > +
> > > + pr_info("Exit r5f subsystem driver\n");
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_remoteproc_probe()
> > > + *
> > > + * @pdev: domain platform device for R5 cluster
> > > + *
> > > + * called when driver is probed, for each R5 core specified in DT,
> > > + * setup as needed to do remoteproc-related operations
> > > + *
> > > + * Return: 0 for success, negative value for failure.
> > > + */
> > > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > > +{
> > > + int ret;
> > > + struct zynqmp_r5_cluster *cluster;
> > > + struct device *dev = &pdev->dev;
> > > +
> > > + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > > + if (IS_ERR_OR_NULL(cluster))
> > > + return -ENOMEM;
> > > +
> > > + cluster->dev = dev;
> > > +
> > > + ret = devm_of_platform_populate(dev);
> > > + if (ret) {
> > > + dev_err(dev, "failed to populate platform dev %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + /* wire in so each core can be cleaned up at driver remove */
> > > + platform_set_drvdata(pdev, cluster);
> > > +
> > > + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
> > > + if (ret)
> > > + return ret;
> > > +
> > > + ret = zynqmp_r5_cluster_init(cluster);
> > > + if (ret) {
> > > + dev_err(dev, "Invalid r5f subsystem device tree\n");
> > > + return ret;
> > > + }
> > > +
> > > + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
> > > + return 0;
> > > +}
> > > +
> > > +/* Match table for OF platform binding */
> > > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > > + { .compatible = "xlnx,zynqmp-r5fss", },
> > > + { /* end of list */ },
> > > +};
> > > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > > +
> > > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > > + .probe = zynqmp_r5_remoteproc_probe,
> > > + .driver = {
> > > + .name = "zynqmp_r5_remoteproc",
> > > + .of_match_table = zynqmp_r5_remoteproc_match,
> > > + },
> > > +};
> > > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > > +
> > > +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
> > > +MODULE_AUTHOR("Xilinx Inc.");
> > > +MODULE_LICENSE("GPL v2");
> > > --
> > > 2.25.1
> > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
2022-02-08 16:54 ` Mathieu Poirier
@ 2022-02-09 10:23 ` Tanmay Shah
-1 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2022-02-09 10:23 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On 2/8/22 10:24 PM, Mathieu Poirier wrote:
> Hi Tanmay,
>
> [...]
>
>>> I am done reviewing this set - the overall structure is sound but there is a
>>> fair amount of work to be done. There is no need to rush another revision as I
>>> won't have time to review it before the holidays. On that front, change
>>> _only_ the things I have commented on. We can address other issues as we move
>>> forward. Last but not least, please run the sparse checker on your work.
>>>
>>> Thanks,
>>> Mathieu
>> Hi Mathieu,
>>
>> I am ready to post v3 in this patch series.
>>
>> I have developed v3 based on rproc-next. However, when I rebased my work on
>> tip of linux-next branch,
> Are the merge conflicts related to drivers/remoteproc or drivers/rpmsg? That
> would be surprising since the code that is in rproc-next and rpmsg-next is also
> in linux-next.
Merge conflict is in Xilinx firmware driver related code. i.e. file
"include/linux/firmware/xlnx-zynqmp.h"
>
> In any case I would suggest to base your v3 on _today's_ for-next branch [1] and
> explicitly mention any dependency in the cover letter. From there I might be
> able to see where the problems are and advise further.
>
> [1]. https://git.kernel.org/pub/scm/linux/kernel/git/remoteproc/linux.git/log/?h=for-next
Sure I will do the same.
I found that following two patches are available in linux-next and not
in rproc-next/for-next:
https://lore.kernel.org/all/20211025082350.10881-2-abhyuday.godhasara@xilinx.com/
https://lore.kernel.org/all/20211026042525.26612-4-abhyuday.godhasara@xilinx.com/
>> I faced merge-conflicts.
>>
>> What is preferred way to proceed in such case? Should I post my patch based
>> on rproc-next or linux-next?
>>
>> Or any other way to proceed further?
>>
>>
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + ret = add_tcm_banks(rproc);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + ret = add_mem_regions(rproc);
>>>> + if (ret)
>>>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>>>> +
>>>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>>>> + if (ret == -EINVAL) {
>>>> + /*
>>>> + * resource table only required for IPC.
>>>> + * if not present, this is not necessarily an error;
>>>> + * for example, loading r5 hello world application
>>>> + * so simply inform user and keep going.
>>>> + */
>>>> + dev_info(&rproc->dev, "no resource table found.\n");
>>>> + ret = 0;
>>>> + }
>>>> + return ret;
>>>> +}
>>>> +
>>>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>>>> + .start = zynqmp_r5_rproc_start,
>>>> + .stop = zynqmp_r5_rproc_stop,
>>>> + .load = rproc_elf_load_segments,
>>>> + .parse_fw = zynqmp_r5_parse_fw,
>>>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>>>> + .sanity_check = rproc_elf_sanity_check,
>>>> + .get_boot_addr = rproc_elf_get_boot_addr,
>>>> +};
>>>> +
>>>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + int i, j, k;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> +
>>>> + dev_dbg(cluster->dev, "Printing dt node info\n");
>>>> +
>>>> + pr_debug("cluster mode = %d\n", cluster->mode);
>>>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>>>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>>>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>>>> +
>>>> + for (i = 0; i < cluster->core_count; i++) {
>>>> + r5_core = &cluster->r5_cores[i];
>>>> + if (!r5_core) {
>>>> + pr_err("can't get r5_core\n");
>>>> + continue;
>>>> + }
>>>> +
>>>> + pr_debug("r5 core %d nodes\n", i);
>>>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>>>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>>>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>>>> + k, r5_core->tcm_banks[k].addr,
>>>> + r5_core->tcm_banks[k].size,
>>>> + r5_core->tcm_banks[k].pm_domain_id,
>>>> + r5_core->tcm_banks[k].bank_name);
>>>> + }
>>>> +
>>>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>>>> +
>>>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>>>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>>>> + j, r5_core->res_mem[j].base,
>>>> + r5_core->res_mem[j].size,
>>>> + r5_core->res_mem[j].name);
>>>> + }
>>>> + }
>>>> +}
>>>> +
>>>> +/**
>>>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>>>> + * this is called for each individual R5 core to
>>>> + * set up mailbox, Xilinx platform manager unique ID,
>>>> + * add to rproc core
>>>> + *
>>>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>>>> + *
>>>> + * Return: 0 for success, negative value for failure.
>>>> + */
>>>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>>>> +{
>>>> + int ret;
>>>> + struct rproc *r5_rproc;
>>>> + struct device *dev;
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + /* Set up DMA mask */
>>>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + /* Allocate remoteproc instance */
>>>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>>>> + NULL, sizeof(struct zynqmp_r5_core));
>>>> + if (IS_ERR_OR_NULL(r5_rproc))
>>>> + return -ENOMEM;
>>>> +
>>>> + r5_rproc->auto_boot = false;
>>>> + r5_rproc->priv = r5_core;
>>>> +
>>>> + /* Add R5 remoteproc */
>>>> + ret = devm_rproc_add(dev, r5_rproc);
>>>> + if (ret) {
>>>> + pr_err("failed to add r5 remoteproc\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + int tcm_bank_count, tcm_node;
>>>> + int i = 0, j;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>>>> + struct device *dev = cluster->dev;
>>>> +
>>>> + /* ToDo: Use predefined TCM address space values from driver until
>>>> + * system-dt spec is not final fot TCM
>>>> + */
>>>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>>>> +
>>>> + /* count per core tcm banks */
>>>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>>>> +
>>>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>>>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>>>> + * 128k. Assign TCM banks to each core accordingly
>>>> + */
>>>> + tcm_node = 0;
>>>> + for (j = 0; j < cluster->core_count; j++) {
>>>> + r5_core = &cluster->r5_cores[j];
>>>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>>>> + tcm_bank_count, GFP_KERNEL);
>>>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>>>> + return -ENOMEM;
>>>> +
>>>> + for (i = 0; i < tcm_bank_count; i++) {
>>>> + /* Use pre-defined TCM reg values.
>>>> + * Eventually this should be replaced by values
>>>> + * parsed from dts.
>>>> + */
>>>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>>>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>>>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>>>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>>>> + tcm_node++;
>>>> + }
>>>> +
>>>> + r5_core->tcm_bank_count = tcm_bank_count;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>>>> +{
>>>> + int res_mem_count, i;
>>>> + struct device *dev;
>>>> + struct device_node *np, *rmem_np;
>>>> + struct reserved_mem *rmem;
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + np = r5_core->np;
>>>> + if (IS_ERR_OR_NULL(np)) {
>>>> + pr_err("invalid device node of r5 core\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>>>> + sizeof(phandle));
>>>> + if (res_mem_count <= 0) {
>>>> + dev_warn(dev, "failed to get memory-region property %d\n",
>>>> + res_mem_count);
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + r5_core->res_mem = devm_kzalloc(dev,
>>>> + res_mem_count * sizeof(struct reserved_mem),
>>>> + GFP_KERNEL);
>>>> + if (!r5_core->res_mem) {
>>>> + dev_err(dev, "failed to allocate mem region memory\n");
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + for (i = 0; i < res_mem_count; i++) {
>>>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>>>> + if (!rmem_np)
>>>> + return -EINVAL;
>>>> +
>>>> + rmem = of_reserved_mem_lookup(rmem_np);
>>>> + if (!rmem) {
>>>> + of_node_put(rmem_np);
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + memcpy(&r5_core->res_mem[i], rmem,
>>>> + sizeof(struct reserved_mem));
>>>> + of_node_put(rmem_np);
>>>> + }
>>>> +
>>>> + r5_core->res_mem_count = res_mem_count;
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + int ret, i;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + struct device *dev = cluster->dev;
>>>> +
>>>> + ret = zynqmp_r5_get_tcm_node(cluster);
>>>> + if (ret < 0) {
>>>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + for (i = 0; i < cluster->core_count; i++) {
>>>> + r5_core = &cluster->r5_cores[i];
>>>> + if (!r5_core) {
>>>> + pr_err("invalid r5 core\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>>>> + if (ret)
>>>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>>>> +
>>>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>>>> + 1, &r5_core->pm_domain_id);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to get power-domains property\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to init r5 core %d\n", i);
>>>> + return ret;
>>>> + }
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + struct device *dev = cluster->dev;
>>>> + struct device_node *dev_node = dev_of_node(dev);
>>>> + struct device_node *child;
>>>> + struct platform_device *child_pdev;
>>>> + int core_count = 0, ret, i;
>>>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>>>> + struct zynqmp_r5_core *r5_cores;
>>>> +
>>>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>>>> +
>>>> + /* on success returns 0, if not defined then returns -EINVAL,
>>>> + * In that case, default is LOCKSTEP mode
>>>> + */
>>>> + if (ret != -EINVAL && ret != 0) {
>>>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + if (cluster_mode == SINGLE_CPU_MODE) {
>>>> + dev_err(dev, "driver does not support single cpu mode\n");
>>>> + return -EINVAL;
>>>> + } else if ((cluster_mode != SPLIT_MODE &&
>>>> + cluster_mode != LOCKSTEP_MODE)) {
>>>> + dev_err(dev, "Invalid cluster mode\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + core_count = of_get_available_child_count(dev_node);
>>>> + if (core_count <= 0) {
>>>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>>>> + return -EINVAL;
>>>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>>>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>>>> + return -EINVAL;
>>>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>>>> + dev_warn(dev, "Only r5 core0 will be used\n");
>>>> + core_count = 1;
>>>> + }
>>>> +
>>>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>>>> + core_count, GFP_KERNEL);
>>>> + if (IS_ERR_OR_NULL(r5_cores)) {
>>>> + dev_err(dev, "can't allocate memory for cores\n");
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + i = 0;
>>>> + for_each_available_child_of_node(dev_node, child) {
>>>> + child_pdev = of_find_device_by_node(child);
>>>> + if (!child_pdev)
>>>> + return -ENODEV;
>>>> +
>>>> + r5_cores[i].dev = &child_pdev->dev;
>>>> + if (!r5_cores[i].dev) {
>>>> + pr_err("can't get device for r5 core %d\n", i);
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>>>> + if (!r5_cores[i].np) {
>>>> + pr_err("can't get device node for r5 core %d\n", i);
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + i++;
>>>> + if (i == core_count)
>>>> + break;
>>>> + }
>>>> +
>>>> + cluster->mode = cluster_mode;
>>>> + cluster->core_count = core_count;
>>>> + cluster->r5_cores = r5_cores;
>>>> +
>>>> + ret = zynqmp_r5_core_init(cluster);
>>>> + if (ret < 0) {
>>>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + zynqmp_r5_print_dt_node_info(cluster);
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static void zynqmp_r5_cluster_exit(void *data)
>>>> +{
>>>> + struct platform_device *pdev = (struct platform_device *)data;
>>>> +
>>>> + platform_set_drvdata(pdev, NULL);
>>>> +
>>>> + pr_info("Exit r5f subsystem driver\n");
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_remoteproc_probe()
>>>> + *
>>>> + * @pdev: domain platform device for R5 cluster
>>>> + *
>>>> + * called when driver is probed, for each R5 core specified in DT,
>>>> + * setup as needed to do remoteproc-related operations
>>>> + *
>>>> + * Return: 0 for success, negative value for failure.
>>>> + */
>>>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>>>> +{
>>>> + int ret;
>>>> + struct zynqmp_r5_cluster *cluster;
>>>> + struct device *dev = &pdev->dev;
>>>> +
>>>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>>>> + if (IS_ERR_OR_NULL(cluster))
>>>> + return -ENOMEM;
>>>> +
>>>> + cluster->dev = dev;
>>>> +
>>>> + ret = devm_of_platform_populate(dev);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + /* wire in so each core can be cleaned up at driver remove */
>>>> + platform_set_drvdata(pdev, cluster);
>>>> +
>>>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + ret = zynqmp_r5_cluster_init(cluster);
>>>> + if (ret) {
>>>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/* Match table for OF platform binding */
>>>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>>>> + { .compatible = "xlnx,zynqmp-r5fss", },
>>>> + { /* end of list */ },
>>>> +};
>>>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>>>> +
>>>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>>>> + .probe = zynqmp_r5_remoteproc_probe,
>>>> + .driver = {
>>>> + .name = "zynqmp_r5_remoteproc",
>>>> + .of_match_table = zynqmp_r5_remoteproc_match,
>>>> + },
>>>> +};
>>>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>>>> +
>>>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>>>> +MODULE_AUTHOR("Xilinx Inc.");
>>>> +MODULE_LICENSE("GPL v2");
>>>> --
>>>> 2.25.1
>>>>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver
@ 2022-02-09 10:23 ` Tanmay Shah
0 siblings, 0 replies; 56+ messages in thread
From: Tanmay Shah @ 2022-02-09 10:23 UTC (permalink / raw)
To: Mathieu Poirier
Cc: Bjorn Andersson, Rob Herring, Michal Simek, Laurent Pinchart,
Ben Levinsky, Bill Mills, Sergei Korneichuk, linux-remoteproc,
linux-kernel, devicetree, linux-arm-kernel
On 2/8/22 10:24 PM, Mathieu Poirier wrote:
> Hi Tanmay,
>
> [...]
>
>>> I am done reviewing this set - the overall structure is sound but there is a
>>> fair amount of work to be done. There is no need to rush another revision as I
>>> won't have time to review it before the holidays. On that front, change
>>> _only_ the things I have commented on. We can address other issues as we move
>>> forward. Last but not least, please run the sparse checker on your work.
>>>
>>> Thanks,
>>> Mathieu
>> Hi Mathieu,
>>
>> I am ready to post v3 in this patch series.
>>
>> I have developed v3 based on rproc-next. However, when I rebased my work on
>> tip of linux-next branch,
> Are the merge conflicts related to drivers/remoteproc or drivers/rpmsg? That
> would be surprising since the code that is in rproc-next and rpmsg-next is also
> in linux-next.
Merge conflict is in Xilinx firmware driver related code. i.e. file
"include/linux/firmware/xlnx-zynqmp.h"
>
> In any case I would suggest to base your v3 on _today's_ for-next branch [1] and
> explicitly mention any dependency in the cover letter. From there I might be
> able to see where the problems are and advise further.
>
> [1]. https://git.kernel.org/pub/scm/linux/kernel/git/remoteproc/linux.git/log/?h=for-next
Sure I will do the same.
I found that following two patches are available in linux-next and not
in rproc-next/for-next:
https://lore.kernel.org/all/20211025082350.10881-2-abhyuday.godhasara@xilinx.com/
https://lore.kernel.org/all/20211026042525.26612-4-abhyuday.godhasara@xilinx.com/
>> I faced merge-conflicts.
>>
>> What is preferred way to proceed in such case? Should I post my patch based
>> on rproc-next or linux-next?
>>
>> Or any other way to proceed further?
>>
>>
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + ret = add_tcm_banks(rproc);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to get TCM banks, err %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + ret = add_mem_regions(rproc);
>>>> + if (ret)
>>>> + dev_warn(dev, "failed to get reserve mem regions %d\n", ret);
>>>> +
>>>> + ret = rproc_elf_load_rsc_table(rproc, fw);
>>>> + if (ret == -EINVAL) {
>>>> + /*
>>>> + * resource table only required for IPC.
>>>> + * if not present, this is not necessarily an error;
>>>> + * for example, loading r5 hello world application
>>>> + * so simply inform user and keep going.
>>>> + */
>>>> + dev_info(&rproc->dev, "no resource table found.\n");
>>>> + ret = 0;
>>>> + }
>>>> + return ret;
>>>> +}
>>>> +
>>>> +static struct rproc_ops zynqmp_r5_rproc_ops = {
>>>> + .start = zynqmp_r5_rproc_start,
>>>> + .stop = zynqmp_r5_rproc_stop,
>>>> + .load = rproc_elf_load_segments,
>>>> + .parse_fw = zynqmp_r5_parse_fw,
>>>> + .find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>>>> + .sanity_check = rproc_elf_sanity_check,
>>>> + .get_boot_addr = rproc_elf_get_boot_addr,
>>>> +};
>>>> +
>>>> +static void zynqmp_r5_print_dt_node_info(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + int i, j, k;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> +
>>>> + dev_dbg(cluster->dev, "Printing dt node info\n");
>>>> +
>>>> + pr_debug("cluster mode = %d\n", cluster->mode);
>>>> + pr_debug("r5f cluster in %s mode\n", (cluster->mode == 0) ? "SPLIT" :
>>>> + cluster->mode == 1 ? "LOCKSTEP" : "SINGLE_CPU");
>>>> + pr_debug("r5f num cores = %d\n", cluster->core_count);
>>>> +
>>>> + for (i = 0; i < cluster->core_count; i++) {
>>>> + r5_core = &cluster->r5_cores[i];
>>>> + if (!r5_core) {
>>>> + pr_err("can't get r5_core\n");
>>>> + continue;
>>>> + }
>>>> +
>>>> + pr_debug("r5 core %d nodes\n", i);
>>>> + pr_debug("TCM banks = %d\n", r5_core->tcm_bank_count);
>>>> + for (k = 0; k < r5_core->tcm_bank_count; k++) {
>>>> + pr_debug("tcm %d addr=0x%llx size=0x%lx, pm_id=%d, %s\n",
>>>> + k, r5_core->tcm_banks[k].addr,
>>>> + r5_core->tcm_banks[k].size,
>>>> + r5_core->tcm_banks[k].pm_domain_id,
>>>> + r5_core->tcm_banks[k].bank_name);
>>>> + }
>>>> +
>>>> + pr_debug("reserve mem regions = %d\n", r5_core->res_mem_count);
>>>> +
>>>> + for (j = 0; j < r5_core->res_mem_count; j++) {
>>>> + pr_debug("mem %d addr=0x%llx, size=0x%llx, name=%s\n",
>>>> + j, r5_core->res_mem[j].base,
>>>> + r5_core->res_mem[j].size,
>>>> + r5_core->res_mem[j].name);
>>>> + }
>>>> + }
>>>> +}
>>>> +
>>>> +/**
>>>> + * zynqmp_r5_add_rproc_core() - Probes ZynqMP R5 processor device node
>>>> + * this is called for each individual R5 core to
>>>> + * set up mailbox, Xilinx platform manager unique ID,
>>>> + * add to rproc core
>>>> + *
>>>> + * @r5_core: zynqmp_r5_core r5 core object to initialize
>>>> + *
>>>> + * Return: 0 for success, negative value for failure.
>>>> + */
>>>> +static int zynqmp_r5_add_rproc_core(struct zynqmp_r5_core *r5_core)
>>>> +{
>>>> + int ret;
>>>> + struct rproc *r5_rproc;
>>>> + struct device *dev;
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + /* Set up DMA mask */
>>>> + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + /* Allocate remoteproc instance */
>>>> + r5_rproc = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
>>>> + NULL, sizeof(struct zynqmp_r5_core));
>>>> + if (IS_ERR_OR_NULL(r5_rproc))
>>>> + return -ENOMEM;
>>>> +
>>>> + r5_rproc->auto_boot = false;
>>>> + r5_rproc->priv = r5_core;
>>>> +
>>>> + /* Add R5 remoteproc */
>>>> + ret = devm_rproc_add(dev, r5_rproc);
>>>> + if (ret) {
>>>> + pr_err("failed to add r5 remoteproc\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_get_tcm_node(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + int tcm_bank_count, tcm_node;
>>>> + int i = 0, j;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + const struct mem_bank_data *tcm = zynqmp_tcm_banks;
>>>> + struct device *dev = cluster->dev;
>>>> +
>>>> + /* ToDo: Use predefined TCM address space values from driver until
>>>> + * system-dt spec is not final fot TCM
>>>> + */
>>>> + tcm_bank_count = ARRAY_SIZE(zynqmp_tcm_banks);
>>>> +
>>>> + /* count per core tcm banks */
>>>> + tcm_bank_count = tcm_bank_count / cluster->core_count;
>>>> +
>>>> + /* r5 core 0 will use all of TCM banks in lockstep mode.
>>>> + * In split mode, r5 core0 will use 128k and r5 core1 will use another
>>>> + * 128k. Assign TCM banks to each core accordingly
>>>> + */
>>>> + tcm_node = 0;
>>>> + for (j = 0; j < cluster->core_count; j++) {
>>>> + r5_core = &cluster->r5_cores[j];
>>>> + r5_core->tcm_banks = devm_kzalloc(dev, sizeof(struct mem_bank_data) *
>>>> + tcm_bank_count, GFP_KERNEL);
>>>> + if (IS_ERR_OR_NULL(r5_core->tcm_banks))
>>>> + return -ENOMEM;
>>>> +
>>>> + for (i = 0; i < tcm_bank_count; i++) {
>>>> + /* Use pre-defined TCM reg values.
>>>> + * Eventually this should be replaced by values
>>>> + * parsed from dts.
>>>> + */
>>>> + r5_core->tcm_banks[i].addr = tcm[tcm_node].addr;
>>>> + r5_core->tcm_banks[i].size = tcm[tcm_node].size;
>>>> + r5_core->tcm_banks[i].pm_domain_id = tcm[tcm_node].pm_domain_id;
>>>> + r5_core->tcm_banks[i].bank_name = tcm[tcm_node].bank_name;
>>>> + tcm_node++;
>>>> + }
>>>> +
>>>> + r5_core->tcm_bank_count = tcm_bank_count;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_get_mem_region_node(struct zynqmp_r5_core *r5_core)
>>>> +{
>>>> + int res_mem_count, i;
>>>> + struct device *dev;
>>>> + struct device_node *np, *rmem_np;
>>>> + struct reserved_mem *rmem;
>>>> +
>>>> + dev = r5_core->dev;
>>>> +
>>>> + np = r5_core->np;
>>>> + if (IS_ERR_OR_NULL(np)) {
>>>> + pr_err("invalid device node of r5 core\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + res_mem_count = of_property_count_elems_of_size(np, "memory-region",
>>>> + sizeof(phandle));
>>>> + if (res_mem_count <= 0) {
>>>> + dev_warn(dev, "failed to get memory-region property %d\n",
>>>> + res_mem_count);
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + r5_core->res_mem = devm_kzalloc(dev,
>>>> + res_mem_count * sizeof(struct reserved_mem),
>>>> + GFP_KERNEL);
>>>> + if (!r5_core->res_mem) {
>>>> + dev_err(dev, "failed to allocate mem region memory\n");
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + for (i = 0; i < res_mem_count; i++) {
>>>> + rmem_np = of_parse_phandle(np, "memory-region", i);
>>>> + if (!rmem_np)
>>>> + return -EINVAL;
>>>> +
>>>> + rmem = of_reserved_mem_lookup(rmem_np);
>>>> + if (!rmem) {
>>>> + of_node_put(rmem_np);
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + memcpy(&r5_core->res_mem[i], rmem,
>>>> + sizeof(struct reserved_mem));
>>>> + of_node_put(rmem_np);
>>>> + }
>>>> +
>>>> + r5_core->res_mem_count = res_mem_count;
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + int ret, i;
>>>> + struct zynqmp_r5_core *r5_core;
>>>> + struct device *dev = cluster->dev;
>>>> +
>>>> + ret = zynqmp_r5_get_tcm_node(cluster);
>>>> + if (ret < 0) {
>>>> + dev_err(dev, "can't get tcm node, err %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + for (i = 0; i < cluster->core_count; i++) {
>>>> + r5_core = &cluster->r5_cores[i];
>>>> + if (!r5_core) {
>>>> + pr_err("invalid r5 core\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + ret = zynqmp_r5_get_mem_region_node(r5_core);
>>>> + if (ret)
>>>> + dev_warn(dev, "memory-region prop failed %d\n", ret);
>>>> +
>>>> + ret = of_property_read_u32_index(r5_core->np, "power-domains",
>>>> + 1, &r5_core->pm_domain_id);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to get power-domains property\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + ret = zynqmp_r5_set_mode(r5_core, cluster->mode);
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + ret = zynqmp_r5_add_rproc_core(r5_core);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to init r5 core %d\n", i);
>>>> + return ret;
>>>> + }
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>>>> +{
>>>> + struct device *dev = cluster->dev;
>>>> + struct device_node *dev_node = dev_of_node(dev);
>>>> + struct device_node *child;
>>>> + struct platform_device *child_pdev;
>>>> + int core_count = 0, ret, i;
>>>> + enum zynqmp_r5_cluster_mode cluster_mode = LOCKSTEP_MODE;
>>>> + struct zynqmp_r5_core *r5_cores;
>>>> +
>>>> + ret = of_property_read_u32(dev_node, "xlnx,cluster-mode", &cluster_mode);
>>>> +
>>>> + /* on success returns 0, if not defined then returns -EINVAL,
>>>> + * In that case, default is LOCKSTEP mode
>>>> + */
>>>> + if (ret != -EINVAL && ret != 0) {
>>>> + dev_err(dev, "Invalid xlnx,cluster-mode property\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + if (cluster_mode == SINGLE_CPU_MODE) {
>>>> + dev_err(dev, "driver does not support single cpu mode\n");
>>>> + return -EINVAL;
>>>> + } else if ((cluster_mode != SPLIT_MODE &&
>>>> + cluster_mode != LOCKSTEP_MODE)) {
>>>> + dev_err(dev, "Invalid cluster mode\n");
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + core_count = of_get_available_child_count(dev_node);
>>>> + if (core_count <= 0) {
>>>> + dev_err(dev, "Invalid number of r5 cores %d", core_count);
>>>> + return -EINVAL;
>>>> + } else if (cluster_mode == SPLIT_MODE && core_count != 2) {
>>>> + dev_err(dev, "Invalid number of r5 cores for split mode\n");
>>>> + return -EINVAL;
>>>> + } else if (cluster_mode == LOCKSTEP_MODE && core_count == 2) {
>>>> + dev_warn(dev, "Only r5 core0 will be used\n");
>>>> + core_count = 1;
>>>> + }
>>>> +
>>>> + r5_cores = devm_kzalloc(dev, sizeof(struct zynqmp_r5_core) *
>>>> + core_count, GFP_KERNEL);
>>>> + if (IS_ERR_OR_NULL(r5_cores)) {
>>>> + dev_err(dev, "can't allocate memory for cores\n");
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + i = 0;
>>>> + for_each_available_child_of_node(dev_node, child) {
>>>> + child_pdev = of_find_device_by_node(child);
>>>> + if (!child_pdev)
>>>> + return -ENODEV;
>>>> +
>>>> + r5_cores[i].dev = &child_pdev->dev;
>>>> + if (!r5_cores[i].dev) {
>>>> + pr_err("can't get device for r5 core %d\n", i);
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + r5_cores[i].np = dev_of_node(r5_cores[i].dev);
>>>> + if (!r5_cores[i].np) {
>>>> + pr_err("can't get device node for r5 core %d\n", i);
>>>> + return -ENODEV;
>>>> + }
>>>> +
>>>> + i++;
>>>> + if (i == core_count)
>>>> + break;
>>>> + }
>>>> +
>>>> + cluster->mode = cluster_mode;
>>>> + cluster->core_count = core_count;
>>>> + cluster->r5_cores = r5_cores;
>>>> +
>>>> + ret = zynqmp_r5_core_init(cluster);
>>>> + if (ret < 0) {
>>>> + dev_err(dev, "failed to init r5 core err %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + zynqmp_r5_print_dt_node_info(cluster);
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static void zynqmp_r5_cluster_exit(void *data)
>>>> +{
>>>> + struct platform_device *pdev = (struct platform_device *)data;
>>>> +
>>>> + platform_set_drvdata(pdev, NULL);
>>>> +
>>>> + pr_info("Exit r5f subsystem driver\n");
>>>> +}
>>>> +
>>>> +/*
>>>> + * zynqmp_r5_remoteproc_probe()
>>>> + *
>>>> + * @pdev: domain platform device for R5 cluster
>>>> + *
>>>> + * called when driver is probed, for each R5 core specified in DT,
>>>> + * setup as needed to do remoteproc-related operations
>>>> + *
>>>> + * Return: 0 for success, negative value for failure.
>>>> + */
>>>> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
>>>> +{
>>>> + int ret;
>>>> + struct zynqmp_r5_cluster *cluster;
>>>> + struct device *dev = &pdev->dev;
>>>> +
>>>> + cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
>>>> + if (IS_ERR_OR_NULL(cluster))
>>>> + return -ENOMEM;
>>>> +
>>>> + cluster->dev = dev;
>>>> +
>>>> + ret = devm_of_platform_populate(dev);
>>>> + if (ret) {
>>>> + dev_err(dev, "failed to populate platform dev %d\n", ret);
>>>> + return ret;
>>>> + }
>>>> +
>>>> + /* wire in so each core can be cleaned up at driver remove */
>>>> + platform_set_drvdata(pdev, cluster);
>>>> +
>>>> + ret = devm_add_action_or_reset(dev, zynqmp_r5_cluster_exit, pdev);
>>>> + if (ret)
>>>> + return ret;
>>>> +
>>>> + ret = zynqmp_r5_cluster_init(cluster);
>>>> + if (ret) {
>>>> + dev_err(dev, "Invalid r5f subsystem device tree\n");
>>>> + return ret;
>>>> + }
>>>> +
>>>> + dev_info(dev, "Xilinx r5f remoteproc driver probe success\n");
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/* Match table for OF platform binding */
>>>> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
>>>> + { .compatible = "xlnx,zynqmp-r5fss", },
>>>> + { /* end of list */ },
>>>> +};
>>>> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
>>>> +
>>>> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
>>>> + .probe = zynqmp_r5_remoteproc_probe,
>>>> + .driver = {
>>>> + .name = "zynqmp_r5_remoteproc",
>>>> + .of_match_table = zynqmp_r5_remoteproc_match,
>>>> + },
>>>> +};
>>>> +module_platform_driver(zynqmp_r5_remoteproc_driver);
>>>> +
>>>> +MODULE_DESCRIPTION("Xilinx R5F remote processor driver");
>>>> +MODULE_AUTHOR("Xilinx Inc.");
>>>> +MODULE_LICENSE("GPL v2");
>>>> --
>>>> 2.25.1
>>>>
^ permalink raw reply [flat|nested] 56+ messages in thread
end of thread, other threads:[~2022-02-09 11:31 UTC | newest]
Thread overview: 56+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-23 6:20 [PATCH v2 0/6] Add Xilinx RPU subsystem support Tanmay Shah
2021-11-23 6:20 ` Tanmay Shah
2021-11-23 6:20 ` [PATCH v2 1/6] dt-bindings: remoteproc: Add Xilinx RPU subsystem bindings Tanmay Shah
2021-11-23 6:20 ` Tanmay Shah
2021-11-23 6:20 ` [PATCH v2 2/6] arm64: dts: xilinx: zynqmp: Add RPU subsystem device node Tanmay Shah
2021-11-23 6:20 ` Tanmay Shah
2021-11-29 17:58 ` Mathieu Poirier
2021-11-29 17:58 ` Mathieu Poirier
2021-11-30 8:01 ` Tanmay Shah
2021-11-30 8:01 ` Tanmay Shah
2021-11-23 6:20 ` [PATCH v2 3/6] firmware: xilinx: Add ZynqMP firmware ioctl enums for RPU configuration Tanmay Shah
2021-11-23 6:20 ` Tanmay Shah
2021-11-23 6:20 ` [PATCH v2 4/6] firmware: xilinx: Add shutdown/wakeup APIs Tanmay Shah
2021-11-23 6:20 ` Tanmay Shah
2021-11-23 6:20 ` [PATCH v2 5/6] firmware: xilinx: Add RPU configuration APIs Tanmay Shah
2021-11-23 6:20 ` Tanmay Shah
2021-11-23 6:20 ` [PATCH v2 6/6] drivers: remoteproc: Add Xilinx r5 remoteproc driver Tanmay Shah
2021-11-23 6:20 ` Tanmay Shah
2021-11-29 18:42 ` Mathieu Poirier
2021-11-29 18:42 ` Mathieu Poirier
2021-11-30 8:01 ` Tanmay Shah
2021-11-30 8:01 ` Tanmay Shah
2021-11-30 20:14 ` Mathieu Poirier
2021-11-30 20:14 ` Mathieu Poirier
2021-12-01 18:37 ` Tanmay Shah
2021-12-01 18:37 ` Tanmay Shah
2021-12-03 18:55 ` Mathieu Poirier
2021-12-03 18:55 ` Mathieu Poirier
2021-12-06 11:03 ` Tanmay Shah
2021-12-06 11:03 ` Tanmay Shah
2021-12-13 8:10 ` Tanmay Shah
2021-12-13 8:10 ` Tanmay Shah
2021-12-13 18:39 ` Mathieu Poirier
2021-12-13 18:39 ` Mathieu Poirier
2021-12-06 19:04 ` Mathieu Poirier
2021-12-06 19:04 ` Mathieu Poirier
2021-12-13 19:11 ` Tanmay Shah
2021-12-13 19:11 ` Tanmay Shah
2021-12-07 18:31 ` Mathieu Poirier
2021-12-07 18:31 ` Mathieu Poirier
2021-12-13 19:17 ` Tanmay Shah
2021-12-13 19:17 ` Tanmay Shah
2022-02-07 18:57 ` Tanmay Shah
2022-02-07 18:57 ` Tanmay Shah
2022-02-08 16:54 ` Mathieu Poirier
2022-02-08 16:54 ` Mathieu Poirier
2022-02-09 10:23 ` Tanmay Shah
2022-02-09 10:23 ` Tanmay Shah
2021-12-13 11:08 ` Lars-Peter Clausen
2021-12-13 11:08 ` Lars-Peter Clausen
2021-12-13 19:31 ` Tanmay Shah
2021-12-13 19:31 ` Tanmay Shah
2021-12-13 19:43 ` Lars-Peter Clausen
2021-12-13 19:43 ` Lars-Peter Clausen
2021-12-13 19:46 ` Tanmay Shah
2021-12-13 19:46 ` Tanmay Shah
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.