[PATCH v25 0/5] Provide basic driver to control Arm R5 co-processor found on Xilinx ZynqMP

All of lore.kernel.org
 help / color / mirror / Atom feed

* [PATCH v25 0/5] Provide basic driver to control Arm R5 co-processor found on Xilinx ZynqMP
@ 2021-01-11  2:02 ` Ben Levinsky
  0 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-01-11  2:02 UTC (permalink / raw)
  To: mathieu.poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
remotproc driver, we can boot the R5 sub-system in two different
configurations -
	* Split
	* Lockstep

The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
Platform Management Unit that handles the R5 configuration, memory access
and R5 lifecycle management. The interface to this manager is done in this
driver via zynqmp_pm_* function calls.

v25:
- reword error message for rpu configuration
- change char name[15] to 16 chars to have null terminated string
  in parse_mem_regions()
- reword comments in tcm_mem_alloc
- grammar in device tree bindings
- call xilinx platform cleanup in parse_tcm_banks() case of failure
- check value of mbox_send_message in all calls within this driver
- update include/linux/firmware/xlnx-zynqmp.h style to match for enum
  pm_node_id
- indentation in zynqmp_r5_remoteproc driver
- update style of constructing carveouts to match convention in ST
  remoteproc drivers. No longer iterate through memory-region property
  using of_count_phandle_with_args. Instead use of_phandle_iterator
- use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per feedback
- rework loop in zynqmp_r5_pm_request_sram as per feedback
- fix comment and linebreak in tcm_mem_alloc description
- add comments and remove extraneous devm_ioremap_wc call in tcm_mem_alloc
- in parse_tcm_banks remove check for bank not being available
- rework order of locals and remove else in zynqmp_r5_rproc_kick
- document and update logic for zynqmp_r5_parse_fw
- add dev_dbg output in event_notified_idr_cb if
  rproc_vq_interrupt(rproc, id) == IRQ_NONE
- add comment for handle_event_notified mbox_send_message function call
- add comment for zynqmp_r5_mb_rx_cb
- update zynqmp_r5_setup_mbox to match convention of mbox setup in ST
  remoteproc drivers
- change return in zynqmp_r5_setup_mbox to use PTR_ERR
- add zynqmp_r5_cleanup_mbox
- in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as arg.
- in zynqmp_r5_probe use return of PTR_ERR and use zynqmp_r5_cleanup_mbox
- in zynqmp_r5_remoteproc_probe update use return value of zynqmp_r5_probe as
  either zynqmp_r5_rproc* or PTR_ERR
- update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_probe
- update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_remove

Previous version:
https://patchwork.kernel.org/project/linux-remoteproc/list/?series=393387


Ben Levinsky (5):
  firmware: xilinx: Add ZynqMP firmware ioctl enums for RPU
    configuration.
  firmware: xilinx: Add shutdown/wakeup APIs
  firmware: xilinx: Add RPU configuration APIs
  dt-bindings: remoteproc: Add documentation for ZynqMP R5 rproc
    bindings
  remoteproc: Add initial zynqmp R5 remoteproc driver

 .../xilinx,zynqmp-r5-remoteproc.yaml          | 223 +++++
 drivers/firmware/xilinx/zynqmp.c              |  96 ++
 drivers/remoteproc/Kconfig                    |   8 +
 drivers/remoteproc/Makefile                   |   1 +
 drivers/remoteproc/zynqmp_r5_remoteproc.c     | 897 ++++++++++++++++++
 include/linux/firmware/xlnx-zynqmp.h          |  66 +-
 6 files changed, 1290 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/xilinx,zynqmp-r5-remoteproc.yaml
 create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c

-- 
2.17.1


^ permalink raw reply	[flat|nested] 28+ messages in thread

* [PATCH v25 0/5] Provide basic driver to control Arm R5 co-processor found on Xilinx ZynqMP
@ 2021-01-11  2:02 ` Ben Levinsky
  0 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-01-11  2:02 UTC (permalink / raw)
  To: mathieu.poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
remotproc driver, we can boot the R5 sub-system in two different
configurations -
	* Split
	* Lockstep

The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
Platform Management Unit that handles the R5 configuration, memory access
and R5 lifecycle management. The interface to this manager is done in this
driver via zynqmp_pm_* function calls.

v25:
- reword error message for rpu configuration
- change char name[15] to 16 chars to have null terminated string
  in parse_mem_regions()
- reword comments in tcm_mem_alloc
- grammar in device tree bindings
- call xilinx platform cleanup in parse_tcm_banks() case of failure
- check value of mbox_send_message in all calls within this driver
- update include/linux/firmware/xlnx-zynqmp.h style to match for enum
  pm_node_id
- indentation in zynqmp_r5_remoteproc driver
- update style of constructing carveouts to match convention in ST
  remoteproc drivers. No longer iterate through memory-region property
  using of_count_phandle_with_args. Instead use of_phandle_iterator
- use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per feedback
- rework loop in zynqmp_r5_pm_request_sram as per feedback
- fix comment and linebreak in tcm_mem_alloc description
- add comments and remove extraneous devm_ioremap_wc call in tcm_mem_alloc
- in parse_tcm_banks remove check for bank not being available
- rework order of locals and remove else in zynqmp_r5_rproc_kick
- document and update logic for zynqmp_r5_parse_fw
- add dev_dbg output in event_notified_idr_cb if
  rproc_vq_interrupt(rproc, id) == IRQ_NONE
- add comment for handle_event_notified mbox_send_message function call
- add comment for zynqmp_r5_mb_rx_cb
- update zynqmp_r5_setup_mbox to match convention of mbox setup in ST
  remoteproc drivers
- change return in zynqmp_r5_setup_mbox to use PTR_ERR
- add zynqmp_r5_cleanup_mbox
- in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as arg.
- in zynqmp_r5_probe use return of PTR_ERR and use zynqmp_r5_cleanup_mbox
- in zynqmp_r5_remoteproc_probe update use return value of zynqmp_r5_probe as
  either zynqmp_r5_rproc* or PTR_ERR
- update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_probe
- update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_remove

Previous version:
https://patchwork.kernel.org/project/linux-remoteproc/list/?series=393387


Ben Levinsky (5):
  firmware: xilinx: Add ZynqMP firmware ioctl enums for RPU
    configuration.
  firmware: xilinx: Add shutdown/wakeup APIs
  firmware: xilinx: Add RPU configuration APIs
  dt-bindings: remoteproc: Add documentation for ZynqMP R5 rproc
    bindings
  remoteproc: Add initial zynqmp R5 remoteproc driver

 .../xilinx,zynqmp-r5-remoteproc.yaml          | 223 +++++
 drivers/firmware/xilinx/zynqmp.c              |  96 ++
 drivers/remoteproc/Kconfig                    |   8 +
 drivers/remoteproc/Makefile                   |   1 +
 drivers/remoteproc/zynqmp_r5_remoteproc.c     | 897 ++++++++++++++++++
 include/linux/firmware/xlnx-zynqmp.h          |  66 +-
 6 files changed, 1290 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/xilinx,zynqmp-r5-remoteproc.yaml
 create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c

-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 28+ messages in thread

* [PATCH v25 1/5] firmware: xilinx: Add ZynqMP firmware ioctl enums for RPU configuration.
  2021-01-11  2:02 ` Ben Levinsky
@ 2021-01-11  2:02   ` Ben Levinsky
  -1 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-01-11  2:02 UTC (permalink / raw)
  To: mathieu.poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

Add ZynqMP firmware ioctl enums for RPU configuration and TCM Nodes for
later use via request_node and release_node

Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
---
v25:
- update include/linux/firmware/xlnx-zynqmp.h style to match for enum
---
 include/linux/firmware/xlnx-zynqmp.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index 5968df82b991..e4044cae0713 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -104,6 +104,10 @@ enum pm_ret_status {
 };
 
 enum pm_ioctl_id {
+	IOCTL_GET_RPU_OPER_MODE = 0,
+	IOCTL_SET_RPU_OPER_MODE = 1,
+	IOCTL_RPU_BOOT_ADDR_CONFIG = 2,
+	IOCTL_TCM_COMB_CONFIG = 3,
 	IOCTL_SD_DLL_RESET = 6,
 	IOCTL_SET_SD_TAPDELAY,
 	IOCTL_SET_PLL_FRAC_MODE,
@@ -129,6 +133,21 @@ enum pm_query_id {
 	PM_QID_CLOCK_GET_MAX_DIVISOR,
 };
 
+enum rpu_oper_mode {
+	PM_RPU_MODE_LOCKSTEP = 0,
+	PM_RPU_MODE_SPLIT = 1,
+};
+
+enum rpu_boot_mem {
+	PM_RPU_BOOTMEM_LOVEC = 0,
+	PM_RPU_BOOTMEM_HIVEC = 1,
+};
+
+enum rpu_tcm_comb {
+	PM_RPU_TCM_SPLIT = 0,
+	PM_RPU_TCM_COMB = 1,
+};
+
 enum zynqmp_pm_reset_action {
 	PM_RESET_ACTION_RELEASE,
 	PM_RESET_ACTION_ASSERT,
@@ -273,6 +292,10 @@ enum zynqmp_pm_request_ack {
 };
 
 enum pm_node_id {
+	NODE_TCM_0_A = 15,
+	NODE_TCM_0_B,
+	NODE_TCM_1_A,
+	NODE_TCM_1_B,
 	NODE_SD_0 = 39,
 	NODE_SD_1,
 };
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [PATCH v25 1/5] firmware: xilinx: Add ZynqMP firmware ioctl enums for RPU configuration.
@ 2021-01-11  2:02   ` Ben Levinsky
  0 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-01-11  2:02 UTC (permalink / raw)
  To: mathieu.poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

Add ZynqMP firmware ioctl enums for RPU configuration and TCM Nodes for
later use via request_node and release_node

Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
---
v25:
- update include/linux/firmware/xlnx-zynqmp.h style to match for enum
---
 include/linux/firmware/xlnx-zynqmp.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index 5968df82b991..e4044cae0713 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -104,6 +104,10 @@ enum pm_ret_status {
 };
 
 enum pm_ioctl_id {
+	IOCTL_GET_RPU_OPER_MODE = 0,
+	IOCTL_SET_RPU_OPER_MODE = 1,
+	IOCTL_RPU_BOOT_ADDR_CONFIG = 2,
+	IOCTL_TCM_COMB_CONFIG = 3,
 	IOCTL_SD_DLL_RESET = 6,
 	IOCTL_SET_SD_TAPDELAY,
 	IOCTL_SET_PLL_FRAC_MODE,
@@ -129,6 +133,21 @@ enum pm_query_id {
 	PM_QID_CLOCK_GET_MAX_DIVISOR,
 };
 
+enum rpu_oper_mode {
+	PM_RPU_MODE_LOCKSTEP = 0,
+	PM_RPU_MODE_SPLIT = 1,
+};
+
+enum rpu_boot_mem {
+	PM_RPU_BOOTMEM_LOVEC = 0,
+	PM_RPU_BOOTMEM_HIVEC = 1,
+};
+
+enum rpu_tcm_comb {
+	PM_RPU_TCM_SPLIT = 0,
+	PM_RPU_TCM_COMB = 1,
+};
+
 enum zynqmp_pm_reset_action {
 	PM_RESET_ACTION_RELEASE,
 	PM_RESET_ACTION_ASSERT,
@@ -273,6 +292,10 @@ enum zynqmp_pm_request_ack {
 };
 
 enum pm_node_id {
+	NODE_TCM_0_A = 15,
+	NODE_TCM_0_B,
+	NODE_TCM_1_A,
+	NODE_TCM_1_B,
 	NODE_SD_0 = 39,
 	NODE_SD_1,
 };
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [PATCH v25 2/5] firmware: xilinx: Add shutdown/wakeup APIs
  2021-01-11  2:02 ` Ben Levinsky
@ 2021-01-11  2:02   ` Ben Levinsky
  -1 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-01-11  2:02 UTC (permalink / raw)
  To: mathieu.poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

Add shutdown/wakeup a resource eemi operations to shutdown
or bringup a resource.

Note alignment of args matches convention of other fn's in this file.
The reason being that the long fn name results in aligned args that
otherwise go over 80 chars so shift right to avoid this

Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
---
 drivers/firmware/xilinx/zynqmp.c     | 35 ++++++++++++++++++++++++++++
 include/linux/firmware/xlnx-zynqmp.h | 23 ++++++++++++++++++
 2 files changed, 58 insertions(+)

diff --git a/drivers/firmware/xilinx/zynqmp.c b/drivers/firmware/xilinx/zynqmp.c
index 8d1ff2454e2e..a966ee956573 100644
--- a/drivers/firmware/xilinx/zynqmp.c
+++ b/drivers/firmware/xilinx/zynqmp.c
@@ -846,6 +846,41 @@ int zynqmp_pm_release_node(const u32 node)
 }
 EXPORT_SYMBOL_GPL(zynqmp_pm_release_node);
 
+/**
+ * zynqmp_pm_force_pwrdwn - PM call to request for another PU or subsystem to
+ *             be powered down forcefully
+ * @node:  Node ID of the targeted PU or subsystem
+ * @ack:   Flag to specify whether acknowledge is requested
+ *
+ * Return: status, either success or error+reason
+ */
+int zynqmp_pm_force_pwrdwn(const u32 node,
+			   const enum zynqmp_pm_request_ack ack)
+{
+	return zynqmp_pm_invoke_fn(PM_FORCE_POWERDOWN, node, ack, 0, 0, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_force_pwrdwn);
+
+/**
+ * zynqmp_pm_request_wake - PM call to wake up selected master or subsystem
+ * @node:  Node ID of the master or subsystem
+ * @set_addr:  Specifies whether the address argument is relevant
+ * @address:   Address from which to resume when woken up
+ * @ack:   Flag to specify whether acknowledge requested
+ *
+ * Return: status, either success or error+reason
+ */
+int zynqmp_pm_request_wake(const u32 node,
+			   const bool set_addr,
+			   const u64 address,
+			   const enum zynqmp_pm_request_ack ack)
+{
+	/* set_addr flag is encoded into 1st bit of address */
+	return zynqmp_pm_invoke_fn(PM_REQUEST_WAKEUP, node, address | set_addr,
+				   address >> 32, ack, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_request_wake);
+
 /**
  * zynqmp_pm_set_requirement() - PM call to set requirement for PM slaves
  * @node:		Node ID of the slave
diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index dc43f005191c..baa3d84fdcce 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -12,6 +12,7 @@
 
 #ifndef __FIRMWARE_ZYNQMP_H__
 #define __FIRMWARE_ZYNQMP_H__
+#include <linux/types.h>
 
 #define ZYNQMP_PM_VERSION_MAJOR	1
 #define ZYNQMP_PM_VERSION_MINOR	0
@@ -64,6 +65,8 @@
 
 enum pm_api_id {
 	PM_GET_API_VERSION = 1,
+	PM_FORCE_POWERDOWN = 8,
+	PM_REQUEST_WAKEUP = 10,
 	PM_SYSTEM_SHUTDOWN = 12,
 	PM_REQUEST_NODE = 13,
 	PM_RELEASE_NODE,
@@ -380,6 +383,12 @@ int zynqmp_pm_write_pggs(u32 index, u32 value);
 int zynqmp_pm_read_pggs(u32 index, u32 *value);
 int zynqmp_pm_system_shutdown(const u32 type, const u32 subtype);
 int zynqmp_pm_set_boot_health_status(u32 value);
+int zynqmp_pm_force_pwrdwn(const u32 target,
+			   const enum zynqmp_pm_request_ack ack);
+int zynqmp_pm_request_wake(const u32 node,
+			   const bool set_addr,
+			   const u64 address,
+			   const enum zynqmp_pm_request_ack ack);
 #else
 static inline struct zynqmp_eemi_ops *zynqmp_pm_get_eemi_ops(void)
 {
@@ -530,6 +539,20 @@ static inline int zynqmp_pm_set_boot_health_status(u32 value)
 {
 	return -ENODEV;
 }
+
+static inline int zynqmp_pm_force_pwrdwn(const u32 target,
+					 const enum zynqmp_pm_request_ack ack)
+{
+	return -ENODEV;
+}
+
+static inline int zynqmp_pm_request_wake(const u32 node,
+					 const bool set_addr,
+					 const u64 address,
+					 const enum zynqmp_pm_request_ack ack)
+{
+	return -ENODEV;
+}
 #endif
 
 #endif /* __FIRMWARE_ZYNQMP_H__ */
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [PATCH v25 2/5] firmware: xilinx: Add shutdown/wakeup APIs
@ 2021-01-11  2:02   ` Ben Levinsky
  0 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-01-11  2:02 UTC (permalink / raw)
  To: mathieu.poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

Add shutdown/wakeup a resource eemi operations to shutdown
or bringup a resource.

Note alignment of args matches convention of other fn's in this file.
The reason being that the long fn name results in aligned args that
otherwise go over 80 chars so shift right to avoid this

Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
---
 drivers/firmware/xilinx/zynqmp.c     | 35 ++++++++++++++++++++++++++++
 include/linux/firmware/xlnx-zynqmp.h | 23 ++++++++++++++++++
 2 files changed, 58 insertions(+)

diff --git a/drivers/firmware/xilinx/zynqmp.c b/drivers/firmware/xilinx/zynqmp.c
index 8d1ff2454e2e..a966ee956573 100644
--- a/drivers/firmware/xilinx/zynqmp.c
+++ b/drivers/firmware/xilinx/zynqmp.c
@@ -846,6 +846,41 @@ int zynqmp_pm_release_node(const u32 node)
 }
 EXPORT_SYMBOL_GPL(zynqmp_pm_release_node);
 
+/**
+ * zynqmp_pm_force_pwrdwn - PM call to request for another PU or subsystem to
+ *             be powered down forcefully
+ * @node:  Node ID of the targeted PU or subsystem
+ * @ack:   Flag to specify whether acknowledge is requested
+ *
+ * Return: status, either success or error+reason
+ */
+int zynqmp_pm_force_pwrdwn(const u32 node,
+			   const enum zynqmp_pm_request_ack ack)
+{
+	return zynqmp_pm_invoke_fn(PM_FORCE_POWERDOWN, node, ack, 0, 0, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_force_pwrdwn);
+
+/**
+ * zynqmp_pm_request_wake - PM call to wake up selected master or subsystem
+ * @node:  Node ID of the master or subsystem
+ * @set_addr:  Specifies whether the address argument is relevant
+ * @address:   Address from which to resume when woken up
+ * @ack:   Flag to specify whether acknowledge requested
+ *
+ * Return: status, either success or error+reason
+ */
+int zynqmp_pm_request_wake(const u32 node,
+			   const bool set_addr,
+			   const u64 address,
+			   const enum zynqmp_pm_request_ack ack)
+{
+	/* set_addr flag is encoded into 1st bit of address */
+	return zynqmp_pm_invoke_fn(PM_REQUEST_WAKEUP, node, address | set_addr,
+				   address >> 32, ack, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_request_wake);
+
 /**
  * zynqmp_pm_set_requirement() - PM call to set requirement for PM slaves
  * @node:		Node ID of the slave
diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index dc43f005191c..baa3d84fdcce 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -12,6 +12,7 @@
 
 #ifndef __FIRMWARE_ZYNQMP_H__
 #define __FIRMWARE_ZYNQMP_H__
+#include <linux/types.h>
 
 #define ZYNQMP_PM_VERSION_MAJOR	1
 #define ZYNQMP_PM_VERSION_MINOR	0
@@ -64,6 +65,8 @@
 
 enum pm_api_id {
 	PM_GET_API_VERSION = 1,
+	PM_FORCE_POWERDOWN = 8,
+	PM_REQUEST_WAKEUP = 10,
 	PM_SYSTEM_SHUTDOWN = 12,
 	PM_REQUEST_NODE = 13,
 	PM_RELEASE_NODE,
@@ -380,6 +383,12 @@ int zynqmp_pm_write_pggs(u32 index, u32 value);
 int zynqmp_pm_read_pggs(u32 index, u32 *value);
 int zynqmp_pm_system_shutdown(const u32 type, const u32 subtype);
 int zynqmp_pm_set_boot_health_status(u32 value);
+int zynqmp_pm_force_pwrdwn(const u32 target,
+			   const enum zynqmp_pm_request_ack ack);
+int zynqmp_pm_request_wake(const u32 node,
+			   const bool set_addr,
+			   const u64 address,
+			   const enum zynqmp_pm_request_ack ack);
 #else
 static inline struct zynqmp_eemi_ops *zynqmp_pm_get_eemi_ops(void)
 {
@@ -530,6 +539,20 @@ static inline int zynqmp_pm_set_boot_health_status(u32 value)
 {
 	return -ENODEV;
 }
+
+static inline int zynqmp_pm_force_pwrdwn(const u32 target,
+					 const enum zynqmp_pm_request_ack ack)
+{
+	return -ENODEV;
+}
+
+static inline int zynqmp_pm_request_wake(const u32 node,
+					 const bool set_addr,
+					 const u64 address,
+					 const enum zynqmp_pm_request_ack ack)
+{
+	return -ENODEV;
+}
 #endif
 
 #endif /* __FIRMWARE_ZYNQMP_H__ */
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [PATCH v25 3/5] firmware: xilinx: Add RPU configuration APIs
  2021-01-11  2:02 ` Ben Levinsky
@ 2021-01-11  2:02   ` Ben Levinsky
  -1 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-01-11  2:02 UTC (permalink / raw)
  To: mathieu.poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

This patch adds APIs to access to configure RPU and its
processor-specific memory.

That is query the run-time mode of RPU as either split or lockstep as well
as API to set this mode. In addition add APIs to access configuration of
the RPUs' tightly coupled memory (TCM).

Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
---
 drivers/firmware/xilinx/zynqmp.c     | 61 ++++++++++++++++++++++++++++
 include/linux/firmware/xlnx-zynqmp.h | 18 ++++++++
 2 files changed, 79 insertions(+)

diff --git a/drivers/firmware/xilinx/zynqmp.c b/drivers/firmware/xilinx/zynqmp.c
index a966ee956573..b390a00338d0 100644
--- a/drivers/firmware/xilinx/zynqmp.c
+++ b/drivers/firmware/xilinx/zynqmp.c
@@ -846,6 +846,67 @@ int zynqmp_pm_release_node(const u32 node)
 }
 EXPORT_SYMBOL_GPL(zynqmp_pm_release_node);
 
+/**
+ * zynqmp_pm_get_rpu_mode() - Get RPU mode
+ * @node_id:	Node ID of the device
+ * @rpu_mode:	return by reference value
+ *		either split or lockstep
+ *
+ * Return:	return 0 on success or error+reason.
+ *		if success, then  rpu_mode will be set
+ *		to current rpu mode.
+ */
+int zynqmp_pm_get_rpu_mode(u32 node_id, enum rpu_oper_mode *rpu_mode)
+{
+	u32 ret_payload[PAYLOAD_ARG_CNT];
+	int ret;
+
+	ret = zynqmp_pm_invoke_fn(PM_IOCTL, node_id,
+				  IOCTL_GET_RPU_OPER_MODE, 0, 0, ret_payload);
+
+	/* only set rpu_mode if no error */
+	if (ret == XST_PM_SUCCESS)
+		*rpu_mode = ret_payload[0];
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_get_rpu_mode);
+
+/**
+ * zynqmp_pm_set_rpu_mode() - Set RPU mode
+ * @node_id:	Node ID of the device
+ * @rpu_mode:	Argument 1 to requested IOCTL call. either split or lockstep
+ *
+ *		This function is used to set RPU mode to split or
+ *		lockstep
+ *
+ * Return:	Returns status, either success or error+reason
+ */
+int zynqmp_pm_set_rpu_mode(u32 node_id, enum rpu_oper_mode rpu_mode)
+{
+	return zynqmp_pm_invoke_fn(PM_IOCTL, node_id,
+				   IOCTL_SET_RPU_OPER_MODE, (u32)rpu_mode,
+				   0, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_set_rpu_mode);
+
+/**
+ * zynqmp_pm_set_tcm_config - configure TCM
+ * @tcm_mode:	Argument 1 to requested IOCTL call
+ *              either PM_RPU_TCM_COMB or PM_RPU_TCM_SPLIT
+ *
+ * This function is used to set RPU mode to split or combined
+ *
+ * Return: status: 0 for success, else failure
+ */
+int zynqmp_pm_set_tcm_config(u32 node_id, enum rpu_tcm_comb tcm_mode)
+{
+	return zynqmp_pm_invoke_fn(PM_IOCTL, node_id,
+				   IOCTL_TCM_COMB_CONFIG, (u32)tcm_mode, 0,
+				   NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_set_tcm_config);
+
 /**
  * zynqmp_pm_force_pwrdwn - PM call to request for another PU or subsystem to
  *             be powered down forcefully
diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index baa3d84fdcce..0ffd62d6740c 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -389,6 +389,9 @@ int zynqmp_pm_request_wake(const u32 node,
 			   const bool set_addr,
 			   const u64 address,
 			   const enum zynqmp_pm_request_ack ack);
+int zynqmp_pm_get_rpu_mode(u32 node_id, enum rpu_oper_mode *rpu_mode);
+int zynqmp_pm_set_rpu_mode(u32 node_id, u32 arg1);
+int zynqmp_pm_set_tcm_config(u32 node_id, u32 arg1);
 #else
 static inline struct zynqmp_eemi_ops *zynqmp_pm_get_eemi_ops(void)
 {
@@ -553,6 +556,21 @@ static inline int zynqmp_pm_request_wake(const u32 node,
 {
 	return -ENODEV;
 }
+
+static inline int zynqmp_pm_get_rpu_mode(u32 node_id, enum rpu_oper_mode *rpu_mode)
+{
+	return -ENODEV;
+}
+
+static inline int zynqmp_pm_set_rpu_mode(u32 node_id, u32 arg1)
+{
+	return -ENODEV;
+}
+
+static inline int zynqmp_pm_set_tcm_config(u32 node_id, u32 arg1)
+{
+	return -ENODEV;
+}
 #endif
 
 #endif /* __FIRMWARE_ZYNQMP_H__ */
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [PATCH v25 3/5] firmware: xilinx: Add RPU configuration APIs
@ 2021-01-11  2:02   ` Ben Levinsky
  0 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-01-11  2:02 UTC (permalink / raw)
  To: mathieu.poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

This patch adds APIs to access to configure RPU and its
processor-specific memory.

That is query the run-time mode of RPU as either split or lockstep as well
as API to set this mode. In addition add APIs to access configuration of
the RPUs' tightly coupled memory (TCM).

Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
---
 drivers/firmware/xilinx/zynqmp.c     | 61 ++++++++++++++++++++++++++++
 include/linux/firmware/xlnx-zynqmp.h | 18 ++++++++
 2 files changed, 79 insertions(+)

diff --git a/drivers/firmware/xilinx/zynqmp.c b/drivers/firmware/xilinx/zynqmp.c
index a966ee956573..b390a00338d0 100644
--- a/drivers/firmware/xilinx/zynqmp.c
+++ b/drivers/firmware/xilinx/zynqmp.c
@@ -846,6 +846,67 @@ int zynqmp_pm_release_node(const u32 node)
 }
 EXPORT_SYMBOL_GPL(zynqmp_pm_release_node);
 
+/**
+ * zynqmp_pm_get_rpu_mode() - Get RPU mode
+ * @node_id:	Node ID of the device
+ * @rpu_mode:	return by reference value
+ *		either split or lockstep
+ *
+ * Return:	return 0 on success or error+reason.
+ *		if success, then  rpu_mode will be set
+ *		to current rpu mode.
+ */
+int zynqmp_pm_get_rpu_mode(u32 node_id, enum rpu_oper_mode *rpu_mode)
+{
+	u32 ret_payload[PAYLOAD_ARG_CNT];
+	int ret;
+
+	ret = zynqmp_pm_invoke_fn(PM_IOCTL, node_id,
+				  IOCTL_GET_RPU_OPER_MODE, 0, 0, ret_payload);
+
+	/* only set rpu_mode if no error */
+	if (ret == XST_PM_SUCCESS)
+		*rpu_mode = ret_payload[0];
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_get_rpu_mode);
+
+/**
+ * zynqmp_pm_set_rpu_mode() - Set RPU mode
+ * @node_id:	Node ID of the device
+ * @rpu_mode:	Argument 1 to requested IOCTL call. either split or lockstep
+ *
+ *		This function is used to set RPU mode to split or
+ *		lockstep
+ *
+ * Return:	Returns status, either success or error+reason
+ */
+int zynqmp_pm_set_rpu_mode(u32 node_id, enum rpu_oper_mode rpu_mode)
+{
+	return zynqmp_pm_invoke_fn(PM_IOCTL, node_id,
+				   IOCTL_SET_RPU_OPER_MODE, (u32)rpu_mode,
+				   0, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_set_rpu_mode);
+
+/**
+ * zynqmp_pm_set_tcm_config - configure TCM
+ * @tcm_mode:	Argument 1 to requested IOCTL call
+ *              either PM_RPU_TCM_COMB or PM_RPU_TCM_SPLIT
+ *
+ * This function is used to set RPU mode to split or combined
+ *
+ * Return: status: 0 for success, else failure
+ */
+int zynqmp_pm_set_tcm_config(u32 node_id, enum rpu_tcm_comb tcm_mode)
+{
+	return zynqmp_pm_invoke_fn(PM_IOCTL, node_id,
+				   IOCTL_TCM_COMB_CONFIG, (u32)tcm_mode, 0,
+				   NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_set_tcm_config);
+
 /**
  * zynqmp_pm_force_pwrdwn - PM call to request for another PU or subsystem to
  *             be powered down forcefully
diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index baa3d84fdcce..0ffd62d6740c 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -389,6 +389,9 @@ int zynqmp_pm_request_wake(const u32 node,
 			   const bool set_addr,
 			   const u64 address,
 			   const enum zynqmp_pm_request_ack ack);
+int zynqmp_pm_get_rpu_mode(u32 node_id, enum rpu_oper_mode *rpu_mode);
+int zynqmp_pm_set_rpu_mode(u32 node_id, u32 arg1);
+int zynqmp_pm_set_tcm_config(u32 node_id, u32 arg1);
 #else
 static inline struct zynqmp_eemi_ops *zynqmp_pm_get_eemi_ops(void)
 {
@@ -553,6 +556,21 @@ static inline int zynqmp_pm_request_wake(const u32 node,
 {
 	return -ENODEV;
 }
+
+static inline int zynqmp_pm_get_rpu_mode(u32 node_id, enum rpu_oper_mode *rpu_mode)
+{
+	return -ENODEV;
+}
+
+static inline int zynqmp_pm_set_rpu_mode(u32 node_id, u32 arg1)
+{
+	return -ENODEV;
+}
+
+static inline int zynqmp_pm_set_tcm_config(u32 node_id, u32 arg1)
+{
+	return -ENODEV;
+}
 #endif
 
 #endif /* __FIRMWARE_ZYNQMP_H__ */
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [PATCH v25 4/5] dt-bindings: remoteproc: Add documentation for ZynqMP R5 rproc bindings
  2021-01-11  2:02 ` Ben Levinsky
@ 2021-01-11  2:02   ` Ben Levinsky
  -1 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-01-11  2:02 UTC (permalink / raw)
  To: mathieu.poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

Add binding for ZynqMP R5 OpenAMP.

Represent the RPU domain resources in one device node. Each RPU
processor is a subnode of the top RPU domain node.

Signed-off-by: Jason Wu <j.wu@xilinx.com>
Signed-off-by: Wendy Liang <jliang@xilinx.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
---
v25:
- grammar in device tree binding's comments
---
 .../xilinx,zynqmp-r5-remoteproc.yaml          | 223 ++++++++++++++++++
 1 file changed, 223 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/xilinx,zynqmp-r5-remoteproc.yaml

diff --git a/Documentation/devicetree/bindings/remoteproc/xilinx,zynqmp-r5-remoteproc.yaml b/Documentation/devicetree/bindings/remoteproc/xilinx,zynqmp-r5-remoteproc.yaml
new file mode 100644
index 000000000000..b9412c2def7b
--- /dev/null
+++ b/Documentation/devicetree/bindings/remoteproc/xilinx,zynqmp-r5-remoteproc.yaml
@@ -0,0 +1,223 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: "http://devicetree.org/schemas/remoteproc/xilinx,zynqmp-r5-remoteproc.yaml#"
+$schema: "http://devicetree.org/meta-schemas/core.yaml#"
+
+title: Xilinx R5 remote processor controller bindings
+
+description:
+  This document defines the binding for the remoteproc component that loads and
+  boots firmwares on the Xilinx Zynqmp and Versal family chipsets.
+
+  Note that the Linux has global addressing view of the R5-related memory (TCM)
+  so the absolute address ranges are provided in TCM reg's.
+
+maintainers:
+  - Ed Mooring <ed.mooring@xilinx.com>
+  - Ben Levinsky <ben.levinsky@xilinx.com>
+
+properties:
+  $nodename:
+    pattern: "^r5fss(@.*)?"
+
+  compatible:
+    enum:
+      - xlnx,zynqmp-r5-remoteproc
+
+  reg:
+    items:
+      - description: Address and Size of Xilinx RPU Configuration register
+
+  "#address-cells":
+    const: 2
+
+  "#size-cells":
+    const: 2
+
+  ranges: true
+
+# Optional properties:
+# --------------------
+  xlnx,cluster-mode:
+    $ref: /schemas/types.yaml#/definitions/uint32
+    enum: [0, 1]
+    description: |
+      Configuration Mode for the Dual R5F cores within the R5F cluster.
+      Should be either a value of 1 (LockStep mode) or 0 (Split mode),
+      default is LockStep mode if omitted.
+
+
+#R5F Processor Child Nodes:
+# ==========================
+
+patternProperties:
+  "^r5f_[a-f0-9]+$":
+    type: object
+    description: |
+      The R5F Sub-System device node should define one or two R5F child nodes,
+      each node representing a Xilinx instantiation of the Arm Cortex R5F core.
+      There should be one or two child nodes if the R5F is in Split mode and
+      one child node if the R5F is in Lockstep mode.
+
+      In Split mode each R5F core has two associated TCM Banks. R5_0 has
+      TCM Banks 0A and 0B and R5_1 has TCM Banks 1A and 1B.
+
+      In Lockstep mode only one R5F child node should be defined. This one
+      child has access to TCM Banks 0A, 0B, 1A and 1B and any of the four can
+      be included in the child R5F's sram property.
+
+      The example below shows Split mode with two child nodes.
+
+    properties:
+      compatible:
+        enum:
+          - xilinx,r5f
+
+# The following properties are mandatory for R5F Core0 in both LockStep and Split
+# modes, and are mandatory for R5F Core1 _only_ in Split mode.
+
+      memory-region:
+        description: |
+          Phandles to the memory nodes to be associated with the
+          The reserved memory nodes should be carveout nodes, and
+          should be defined with a "no-map" property as per the bindings in
+          Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
+        minItems: 1
+        maxItems: 6
+        items:
+          - description: Region used for dynamic DMA allocations like vrings and
+                         vring buffers
+          - description: Region reserved for firmware image sections
+        additionalItems: true
+
+      power-domain:
+        description: |
+          Power node ID that is used to uniquely identify the RPU for Xilinx
+          Power Management.
+        maxItems: 1
+
+# Optional properties:
+# --------------------
+# The following properties are optional properties for each of the R5F cores:
+
+      mboxes:
+        description: |
+          Standard property to specify a mailbox
+          This property is required only if the rpmsg/virtio functionality
+          is used
+
+           Refer to the zynqmp-ipi-mailbox documentation for client usage of this
+           property
+        maxItems: 1
+
+      mbox-names:
+        description: |
+          Refer to the zynqmp-ipi-mailbox documentation for client usage of this
+          property
+        items:
+          - const: tx
+          - const: rx
+
+      sram:
+        $ref: /schemas/types.yaml#/definitions/phandle-array
+        minItems: 1
+        maxItems: 4
+        description: |
+          Phandles to one or more reserved on-chip SRAM regions. The regions
+          should be defined as child nodes of the respective SRAM node, and
+          should be defined as per the generic bindings in
+          Documentation/devicetree/bindings/sram/sram.yaml
+
+    required:
+      - compatible
+      - power-domain
+
+    unevaluatedProperties: false
+
+required:
+  - reg
+  - compatible
+  - "#address-cells"
+  - "#size-cells"
+  - ranges
+
+additionalProperties: false
+
+examples:
+  - |
+    / {
+        compatible = "xlnx,zynqmp-zcu102-rev1.0", "xlnx,zynqmp-zcu102", "xlnx,zynqmp";
+        #address-cells = <2>;
+        #size-cells = <2>;
+        model = "ZynqMP ZCU102 ";
+
+        zynqmp_ipi1 {
+          compatible = "xlnx,zynqmp-ipi-mailbox";
+          interrupt-parent = <&gic>;
+          interrupts = <0 33 4>;
+          xlnx,ipi-id = <5>;
+          #address-cells = <1>;
+          #size-cells = <0>;
+
+          ipi_mailbox_rpu0: mailbox@ff990600 {
+            reg = <0xff990600 0x20>,
+                  <0xff990620 0x20>,
+                  <0xff9900c0 0x20>,
+                  <0xff9900e0 0x20>;
+            reg-names = "local_request_region",
+            "local_response_region",
+            "remote_request_region",
+            "remote_response_region";
+            #mbox-cells = <1>;
+            xlnx,ipi-id = <3>;
+          };
+          ipi_mailbox_rpu1: mailbox@ff990780 {
+            reg = <0xff990780 0x20>,
+                  <0xff9907a0 0x20>,
+                  <0xff9907c0 0x20>,
+                  <0xff9905a0 0x20>;
+            reg-names = "local_request_region",
+            "local_response_region",
+            "remote_request_region",
+            "remote_response_region";
+            #mbox-cells = <1>;
+            xlnx,ipi-id = <3>;
+          };
+        };
+
+        r5fss@ff9a0000 {
+          compatible = "xlnx,zynqmp-r5-remoteproc";
+          #address-cells = <2>;
+          #size-cells = <2>;
+          ranges;
+          reg = <0x0 0xff9a0000 0x0 0x10000>;
+          xlnx,cluster-mode = <0>;
+
+          r5f_0 {
+               compatible = "xilinx,r5f";
+               memory-region = <&elf_load0>,
+                               <&rpu0vdev0vring0>,
+                               <&rpu0vdev0vring1>,
+                               <&rpu0vdev0buffer>;
+               sram = <&tcm_0a>, <&tcm_0b>;
+               mboxes = <&ipi_mailbox_rpu0 0x0 &ipi_mailbox_rpu0 0x1>;
+               mbox-names = "tx", "rx";
+               power-domain = <0x7>;
+          };
+          r5f_1 {
+               compatible = "xilinx,r5f";
+               memory-region = <&elf_load1>,
+                               <&rpu1vdev0vring0>,
+                               <&rpu1vdev0vring1>,
+                               <&rpu1vdev0buffer>;
+               sram = <&tcm_1a>, <&tcm_1b>;
+               mboxes = <&ipi_mailbox_rpu1 0x0 &ipi_mailbox_rpu1 0x1>;
+               mbox-names = "tx", "rx";
+               power-domain = <0x8>;
+          };
+
+        };
+    };
+
+...
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [PATCH v25 4/5] dt-bindings: remoteproc: Add documentation for ZynqMP R5 rproc bindings
@ 2021-01-11  2:02   ` Ben Levinsky
  0 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-01-11  2:02 UTC (permalink / raw)
  To: mathieu.poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

Add binding for ZynqMP R5 OpenAMP.

Represent the RPU domain resources in one device node. Each RPU
processor is a subnode of the top RPU domain node.

Signed-off-by: Jason Wu <j.wu@xilinx.com>
Signed-off-by: Wendy Liang <jliang@xilinx.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
---
v25:
- grammar in device tree binding's comments
---
 .../xilinx,zynqmp-r5-remoteproc.yaml          | 223 ++++++++++++++++++
 1 file changed, 223 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/xilinx,zynqmp-r5-remoteproc.yaml

diff --git a/Documentation/devicetree/bindings/remoteproc/xilinx,zynqmp-r5-remoteproc.yaml b/Documentation/devicetree/bindings/remoteproc/xilinx,zynqmp-r5-remoteproc.yaml
new file mode 100644
index 000000000000..b9412c2def7b
--- /dev/null
+++ b/Documentation/devicetree/bindings/remoteproc/xilinx,zynqmp-r5-remoteproc.yaml
@@ -0,0 +1,223 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: "http://devicetree.org/schemas/remoteproc/xilinx,zynqmp-r5-remoteproc.yaml#"
+$schema: "http://devicetree.org/meta-schemas/core.yaml#"
+
+title: Xilinx R5 remote processor controller bindings
+
+description:
+  This document defines the binding for the remoteproc component that loads and
+  boots firmwares on the Xilinx Zynqmp and Versal family chipsets.
+
+  Note that the Linux has global addressing view of the R5-related memory (TCM)
+  so the absolute address ranges are provided in TCM reg's.
+
+maintainers:
+  - Ed Mooring <ed.mooring@xilinx.com>
+  - Ben Levinsky <ben.levinsky@xilinx.com>
+
+properties:
+  $nodename:
+    pattern: "^r5fss(@.*)?"
+
+  compatible:
+    enum:
+      - xlnx,zynqmp-r5-remoteproc
+
+  reg:
+    items:
+      - description: Address and Size of Xilinx RPU Configuration register
+
+  "#address-cells":
+    const: 2
+
+  "#size-cells":
+    const: 2
+
+  ranges: true
+
+# Optional properties:
+# --------------------
+  xlnx,cluster-mode:
+    $ref: /schemas/types.yaml#/definitions/uint32
+    enum: [0, 1]
+    description: |
+      Configuration Mode for the Dual R5F cores within the R5F cluster.
+      Should be either a value of 1 (LockStep mode) or 0 (Split mode),
+      default is LockStep mode if omitted.
+
+
+#R5F Processor Child Nodes:
+# ==========================
+
+patternProperties:
+  "^r5f_[a-f0-9]+$":
+    type: object
+    description: |
+      The R5F Sub-System device node should define one or two R5F child nodes,
+      each node representing a Xilinx instantiation of the Arm Cortex R5F core.
+      There should be one or two child nodes if the R5F is in Split mode and
+      one child node if the R5F is in Lockstep mode.
+
+      In Split mode each R5F core has two associated TCM Banks. R5_0 has
+      TCM Banks 0A and 0B and R5_1 has TCM Banks 1A and 1B.
+
+      In Lockstep mode only one R5F child node should be defined. This one
+      child has access to TCM Banks 0A, 0B, 1A and 1B and any of the four can
+      be included in the child R5F's sram property.
+
+      The example below shows Split mode with two child nodes.
+
+    properties:
+      compatible:
+        enum:
+          - xilinx,r5f
+
+# The following properties are mandatory for R5F Core0 in both LockStep and Split
+# modes, and are mandatory for R5F Core1 _only_ in Split mode.
+
+      memory-region:
+        description: |
+          Phandles to the memory nodes to be associated with the
+          The reserved memory nodes should be carveout nodes, and
+          should be defined with a "no-map" property as per the bindings in
+          Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
+        minItems: 1
+        maxItems: 6
+        items:
+          - description: Region used for dynamic DMA allocations like vrings and
+                         vring buffers
+          - description: Region reserved for firmware image sections
+        additionalItems: true
+
+      power-domain:
+        description: |
+          Power node ID that is used to uniquely identify the RPU for Xilinx
+          Power Management.
+        maxItems: 1
+
+# Optional properties:
+# --------------------
+# The following properties are optional properties for each of the R5F cores:
+
+      mboxes:
+        description: |
+          Standard property to specify a mailbox
+          This property is required only if the rpmsg/virtio functionality
+          is used
+
+           Refer to the zynqmp-ipi-mailbox documentation for client usage of this
+           property
+        maxItems: 1
+
+      mbox-names:
+        description: |
+          Refer to the zynqmp-ipi-mailbox documentation for client usage of this
+          property
+        items:
+          - const: tx
+          - const: rx
+
+      sram:
+        $ref: /schemas/types.yaml#/definitions/phandle-array
+        minItems: 1
+        maxItems: 4
+        description: |
+          Phandles to one or more reserved on-chip SRAM regions. The regions
+          should be defined as child nodes of the respective SRAM node, and
+          should be defined as per the generic bindings in
+          Documentation/devicetree/bindings/sram/sram.yaml
+
+    required:
+      - compatible
+      - power-domain
+
+    unevaluatedProperties: false
+
+required:
+  - reg
+  - compatible
+  - "#address-cells"
+  - "#size-cells"
+  - ranges
+
+additionalProperties: false
+
+examples:
+  - |
+    / {
+        compatible = "xlnx,zynqmp-zcu102-rev1.0", "xlnx,zynqmp-zcu102", "xlnx,zynqmp";
+        #address-cells = <2>;
+        #size-cells = <2>;
+        model = "ZynqMP ZCU102 ";
+
+        zynqmp_ipi1 {
+          compatible = "xlnx,zynqmp-ipi-mailbox";
+          interrupt-parent = <&gic>;
+          interrupts = <0 33 4>;
+          xlnx,ipi-id = <5>;
+          #address-cells = <1>;
+          #size-cells = <0>;
+
+          ipi_mailbox_rpu0: mailbox@ff990600 {
+            reg = <0xff990600 0x20>,
+                  <0xff990620 0x20>,
+                  <0xff9900c0 0x20>,
+                  <0xff9900e0 0x20>;
+            reg-names = "local_request_region",
+            "local_response_region",
+            "remote_request_region",
+            "remote_response_region";
+            #mbox-cells = <1>;
+            xlnx,ipi-id = <3>;
+          };
+          ipi_mailbox_rpu1: mailbox@ff990780 {
+            reg = <0xff990780 0x20>,
+                  <0xff9907a0 0x20>,
+                  <0xff9907c0 0x20>,
+                  <0xff9905a0 0x20>;
+            reg-names = "local_request_region",
+            "local_response_region",
+            "remote_request_region",
+            "remote_response_region";
+            #mbox-cells = <1>;
+            xlnx,ipi-id = <3>;
+          };
+        };
+
+        r5fss@ff9a0000 {
+          compatible = "xlnx,zynqmp-r5-remoteproc";
+          #address-cells = <2>;
+          #size-cells = <2>;
+          ranges;
+          reg = <0x0 0xff9a0000 0x0 0x10000>;
+          xlnx,cluster-mode = <0>;
+
+          r5f_0 {
+               compatible = "xilinx,r5f";
+               memory-region = <&elf_load0>,
+                               <&rpu0vdev0vring0>,
+                               <&rpu0vdev0vring1>,
+                               <&rpu0vdev0buffer>;
+               sram = <&tcm_0a>, <&tcm_0b>;
+               mboxes = <&ipi_mailbox_rpu0 0x0 &ipi_mailbox_rpu0 0x1>;
+               mbox-names = "tx", "rx";
+               power-domain = <0x7>;
+          };
+          r5f_1 {
+               compatible = "xilinx,r5f";
+               memory-region = <&elf_load1>,
+                               <&rpu1vdev0vring0>,
+                               <&rpu1vdev0vring1>,
+                               <&rpu1vdev0buffer>;
+               sram = <&tcm_1a>, <&tcm_1b>;
+               mboxes = <&ipi_mailbox_rpu1 0x0 &ipi_mailbox_rpu1 0x1>;
+               mbox-names = "tx", "rx";
+               power-domain = <0x8>;
+          };
+
+        };
+    };
+
+...
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
  2021-01-11  2:02 ` Ben Levinsky
@ 2021-01-11  2:02   ` Ben Levinsky
  -1 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-01-11  2:02 UTC (permalink / raw)
  To: mathieu.poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
remotproc driver, we can boot the R5 sub-system in two different
configurations -
	* Split
	* Lockstep

The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
Platform Management Unit that handles the R5 configuration, memory access
and R5 lifecycle management. The interface to this manager is done in this
driver via zynqmp_pm_* function calls.

Signed-off-by: Wendy Liang <wendy.liang@xilinx.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Ed Mooring <ed.mooring@xilinx.com>
Signed-off-by: Jason Wu <j.wu@xilinx.com>
Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
---
v25:
- reword error message for rpu configuration
- change char name[15] to 16 chars to have null terminated string
  in parse_mem_regions()
- reword comments in tcm_mem_alloc
- call xilinx platform cleanup in parse_tcm_banks() case of failure
- check value of mbox_send_message in all instances within this driver 
- indentation in zynqmp_r5_remoteproc driver
- update style of constructing carveouts to match convention in ST
  remoteproc drivers. No longer iterate through memory-region property
  using of_count_phandle_with_args. Instead use of_phandle_iterator
- use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per feedback
- rework loop in zynqmp_r5_pm_request_sram as per feedback
- fix comment and linebreak in tcm_mem_alloc description
- add comments and remove extraneous devm_ioremap_wc call in tcm_mem_alloc
- in parse_tcm_banks remove check for bank not being available
- rework order of locals and remove else in zynqmp_r5_rproc_kick
- document and update logic for zynqmp_r5_parse_fw
- add dev_dbg output in event_notified_idr_cb if
  rproc_vq_interrupt(rproc, id) == IRQ_NONE
- add comment for handle_event_notified mbox_send_message function call
- add comment for zynqmp_r5_mb_rx_cb
- update zynqmp_r5_setup_mbox to match convention of mbox setup in ST
  remoteproc drivers
- change return in zynqmp_r5_setup_mbox to use PTR_ERR
- add zynqmp_r5_cleanup_mbox
- in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as arg.
- in zynqmp_r5_probe use return of PTR_ERR and use zynqmp_r5_cleanup_mbox
- in zynqmp_r5_remoteproc_probe update use return value of zynqmp_r5_probe as
  either zynqmp_r5_rproc* or PTR_ERR
- update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_probe
- update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_remove
---
 drivers/remoteproc/Kconfig                |   8 +
 drivers/remoteproc/Makefile               |   1 +
 drivers/remoteproc/zynqmp_r5_remoteproc.c | 898 ++++++++++++++++++++++
 3 files changed, 907 insertions(+)
 create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c

diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
index c6659dfea7c7..c2fe54b1d94f 100644
--- a/drivers/remoteproc/Kconfig
+++ b/drivers/remoteproc/Kconfig
@@ -275,6 +275,14 @@ config TI_K3_DSP_REMOTEPROC
 	  It's safe to say N here if you're not interested in utilizing
 	  the DSP slave processors.
 
+config ZYNQMP_R5_REMOTEPROC
+	tristate "ZynqMP R5 remoteproc support"
+	depends on PM && ARCH_ZYNQMP
+	select RPMSG_VIRTIO
+	select ZYNQMP_IPI_MBOX
+	help
+	  Say y or m here to support ZynqMP R5 remote processors via the remote
+	  processor framework.
 endif # REMOTEPROC
 
 endmenu
diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
index 3dfa28e6c701..ef1abff654c2 100644
--- a/drivers/remoteproc/Makefile
+++ b/drivers/remoteproc/Makefile
@@ -33,3 +33,4 @@ obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
 obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
 obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
 obj-$(CONFIG_TI_K3_DSP_REMOTEPROC)	+= ti_k3_dsp_remoteproc.o
+obj-$(CONFIG_ZYNQMP_R5_REMOTEPROC)	+= zynqmp_r5_remoteproc.o
diff --git a/drivers/remoteproc/zynqmp_r5_remoteproc.c b/drivers/remoteproc/zynqmp_r5_remoteproc.c
new file mode 100644
index 000000000000..9e228c9ecd43
--- /dev/null
+++ b/drivers/remoteproc/zynqmp_r5_remoteproc.c
@@ -0,0 +1,898 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Zynq R5 Remote Processor driver
+ *
+ * Based on origin OMAP and Zynq Remote Processor driver
+ *
+ */
+
+#include <linux/firmware/xlnx-zynqmp.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/mailbox_client.h>
+#include <linux/mailbox/zynqmp-ipi-message.h>
+#include <linux/module.h>
+#include <linux/of_address.h>
+#include <linux/of_platform.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/remoteproc.h>
+#include <linux/skbuff.h>
+#include <linux/sysfs.h>
+
+#include "remoteproc_internal.h"
+
+#define MAX_RPROCS	2 /* Support up to 2 RPU */
+#define MAX_MEM_PNODES	4 /* Max power nodes for one RPU memory instance */
+
+#define BANK_LIST_PROP	"sram"
+#define DDR_LIST_PROP	"memory-region"
+
+/* IPI buffer MAX length */
+#define IPI_BUF_LEN_MAX	32U
+/* RX mailbox client buffer max length */
+#define RX_MBOX_CLIENT_BUF_MAX	(IPI_BUF_LEN_MAX + \
+				 sizeof(struct zynqmp_ipi_message))
+
+/*
+ * Map each Xilinx on-chip SRAM  Bank address to their own respective
+ * pm_node_id.
+ */
+struct sram_addr_data {
+	phys_addr_t addr;
+	enum pm_node_id id;
+};
+
+#define NUM_SRAMS 4U
+static const struct sram_addr_data zynqmp_banks[NUM_SRAMS] = {
+	{0xffe00000UL, NODE_TCM_0_A},
+	{0xffe20000UL, NODE_TCM_0_B},
+	{0xffe90000UL, NODE_TCM_1_A},
+	{0xffeb0000UL, NODE_TCM_1_B},
+};
+
+/**
+ * struct zynqmp_r5_rproc - ZynqMP R5 core structure
+ *
+ * @rx_mc_buf: rx mailbox client buffer to save the rx message
+ * @tx_mc: tx mailbox client
+ * @rx_mc: rx mailbox client
+ * @mbox_work: mbox_work for the RPU remoteproc
+ * @tx_mc_skbs: socket buffers for tx mailbox client
+ * @dev: device of RPU instance
+ * @rproc: rproc handle
+ * @tx_chan: tx mailbox channel
+ * @rx_chan: rx mailbox channel
+ * @pnode_id: RPU CPU power domain id
+ * @elem: linked list item
+ */
+struct zynqmp_r5_rproc {
+	unsigned char rx_mc_buf[RX_MBOX_CLIENT_BUF_MAX];
+	struct mbox_client tx_mc;
+	struct mbox_client rx_mc;
+	struct work_struct mbox_work;
+	struct sk_buff_head tx_mc_skbs;
+	struct device *dev;
+	struct rproc *rproc;
+	struct mbox_chan *tx_chan;
+	struct mbox_chan *rx_chan;
+	u32 pnode_id;
+	struct list_head elem;
+};
+
+/*
+ * r5_set_mode - set RPU operation mode
+ * @z_rproc: Remote processor private data
+ * @rpu_mode: mode specified by device tree to configure the RPU to
+ *
+ * set RPU operation mode
+ *
+ * Return: 0 for success, negative value for failure
+ */
+static int r5_set_mode(struct zynqmp_r5_rproc *z_rproc,
+		       enum rpu_oper_mode rpu_mode)
+{
+	enum rpu_tcm_comb tcm_mode;
+	enum rpu_oper_mode cur_rpu_mode;
+	int ret;
+
+	ret = zynqmp_pm_get_rpu_mode(z_rproc->pnode_id, &cur_rpu_mode);
+	if (ret < 0)
+		return ret;
+
+	if (rpu_mode != cur_rpu_mode) {
+		ret = zynqmp_pm_set_rpu_mode(z_rproc->pnode_id, rpu_mode);
+		if (ret < 0)
+			return ret;
+	}
+
+	tcm_mode = (rpu_mode == PM_RPU_MODE_LOCKSTEP) ?
+		    PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
+	return zynqmp_pm_set_tcm_config(z_rproc->pnode_id, tcm_mode);
+}
+
+/*
+ * zynqmp_r5_rproc_mem_release
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to unmap
+ *
+ * Unmap TCM banks when powering down R5 core.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int tcm_mem_release(struct rproc *rproc, struct rproc_mem_entry *mem)
+{
+	u32 pnode_id = (u64)mem->priv;
+
+	iounmap(mem->va);
+	return zynqmp_pm_release_node(pnode_id);
+}
+
+/*
+ * zynqmp_r5_rproc_start
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Start R5 Core from designated boot address.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_start(struct rproc *rproc)
+{
+	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
+	enum rpu_boot_mem bootmem;
+
+	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
+		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
+
+	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
+		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
+
+	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
+				      bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
+}
+
+/*
+ * zynqmp_r5_rproc_stop
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Power down  R5 Core.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_stop(struct rproc *rproc)
+{
+	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
+
+	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
+				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+}
+
+/*
+ * zynqmp_r5_rproc_mem_alloc
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to map
+ *
+ * Callback to map va for memory-region's carveout.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
+				     struct rproc_mem_entry *mem)
+{
+	void *va;
+
+	va = ioremap_wc(mem->dma, mem->len);
+	if (IS_ERR_OR_NULL(va))
+		return -ENOMEM;
+
+	mem->va = va;
+
+	return 0;
+}
+
+/*
+ * zynqmp_r5_rproc_mem_release
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to unmap
+ *
+ * Unmap memory-region carveout
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
+				       struct rproc_mem_entry *mem)
+{
+	iounmap(mem->va);
+	return 0;
+}
+
+/*
+ * parse_mem_regions
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Construct rproc mem carveouts from carveout provided in
+ * memory-region property
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int parse_mem_regions(struct rproc *rproc)
+{
+	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
+	struct device_node *np = z_rproc->dev->of_node;
+	struct device *dev = &rproc->dev;
+	struct of_phandle_iterator it;
+	struct rproc_mem_entry *mem;
+	struct reserved_mem *rmem;
+	int index = 0;
+
+	/* Register associated reserved memory regions */
+	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
+	while (of_phandle_iterator_next(&it) == 0) {
+		rmem = of_reserved_mem_lookup(it.node);
+		if (!rmem) {
+			dev_err(dev, "unable to acquire %s\n", DDR_LIST_PROP);
+			return -EINVAL;
+		}
+
+		if (strstr(it.node->name, "vdev0buffer")) {
+			mem = rproc_of_resm_mem_entry_init(dev, index,
+							   rmem->size,
+							   rmem->base,
+							   "vdev0buffer");
+		} else {
+			/*
+			 * The ensuing carveout is either for vring, which has
+			 * device tree node name of length 15 characters, or
+			 * firmware mem, where name is not used by remoteproc
+			 *  core later on. So default to vring length of 15.
+			 *
+			 * Extra char for null-terminated string.
+			 */
+			char name[16];
+
+			if (strstr(it.node->name, "vdev0vring")) {
+				int vring_id;
+
+				/*
+				 * Expect "rpuXvdev0vringX" as documented
+				 * in xilinx remoteproc device tree binding
+				 */
+				if (strlen(it.node->name) < 15) {
+					dev_err(dev, "%pOF is less than 15 chars",
+						it.node);
+					return -EINVAL;
+				}
+
+				/*
+				 * can be 1 of multiple vring IDs per IPC channel
+				 * e.g. 'vdev0vring0' and 'vdev0vring1'
+				 *
+				 * Here get vring ID from 15th char
+				 */
+				vring_id = it.node->name[14] - '0';
+				snprintf(name, sizeof(name), "vdev0vring%d", vring_id);
+			} else {
+				strncpy(name, it.node->name, 16);
+			}
+
+			mem = rproc_mem_entry_init(dev, NULL,
+						   (dma_addr_t)rmem->base,
+						   rmem->size, rmem->base,
+						   zynqmp_r5_rproc_mem_alloc,
+						   zynqmp_r5_rproc_mem_release,
+						   name);
+		}
+
+		if (!mem)
+			return -ENOMEM;
+
+		rproc_add_carveout(rproc, mem);
+		index++;
+	}
+
+	return 0;
+}
+
+/*
+ * zynqmp_r5_pm_request_tcm
+ * @addr: base address of mem provided in R5 core's sram property.
+ *
+ * Given sram base address, determine its corresponding Xilinx
+ * Platform Management ID and then request access to this node
+ * so that it can be power up.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
+{
+	int i;
+
+	for (i = 0; i < NUM_SRAMS; i++) {
+		if (zynqmp_banks[i].addr != addr)
+			continue;
+
+		return zynqmp_pm_request_node(zynqmp_banks[i].id,
+					      ZYNQMP_PM_CAPABILITY_ACCESS, 0,
+					      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+	}
+
+	return -EINVAL;
+}
+
+/*
+ * tcm_mem_alloc
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to initialize the va and da fields of
+ *
+ * Given TCM bank entry, this callback will set device address for R5
+ * running on TCM and also setup virtual address for TCM bank
+ * remoteproc carveout.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int tcm_mem_alloc(struct rproc *rproc,
+			 struct rproc_mem_entry *mem)
+{
+	void *va;
+	struct device *dev = rproc->dev.parent;
+
+	va = ioremap_wc(mem->dma, mem->len);
+	if (IS_ERR_OR_NULL(va))
+		return -ENOMEM;
+
+	/* Update memory entry va */
+	mem->va = va;
+
+	/*
+	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
+	 * while on the Linux side they are at 0xffexxxxx.
+	 *
+	 * Zero out the high 12 bits of the address. This will give
+	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
+	 */
+	mem->da &= 0x000fffff;
+
+	/*
+	 * TCM Banks 1A and 1B still have to be translated.
+	 *
+	 * Below handle these two banks' absolute addresses (0xffe90000 and
+	 * 0xffeb0000) and convert to the expected relative addresses
+	 * (0x0 and 0x20000).
+	 */
+	if (mem->da == 0x90000 || mem->da == 0xB0000)
+		mem->da -= 0x90000;
+
+	/* if translated TCM bank address is not valid report error */
+	if (mem->da != 0x0 && mem->da != 0x20000) {
+		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/*
+ * parse_tcm_banks()
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Given R5 node in remoteproc instance
+ * allocate remoteproc carveout for TCM memory
+ * needed for firmware to be loaded
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int parse_tcm_banks(struct rproc *rproc)
+{
+	int i, num_banks;
+	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
+	struct device *dev = &rproc->dev;
+	struct device_node *r5_node = z_rproc->dev->of_node;
+
+	/* go through TCM banks for r5 node */
+	num_banks = of_count_phandle_with_args(r5_node, BANK_LIST_PROP, NULL);
+	if (num_banks <= 0) {
+		dev_err(dev, "need to specify TCM banks\n");
+		return -EINVAL;
+	}
+	for (i = 0; i < num_banks; i++) {
+		struct resource rsc;
+		resource_size_t size;
+		struct device_node *dt_node;
+		struct rproc_mem_entry *mem;
+		int ret;
+		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
+
+		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
+		if (!dt_node)
+			return -EINVAL;
+
+		ret = of_address_to_resource(dt_node, 0, &rsc);
+		if (ret < 0) {
+			of_node_put(dt_node);
+			return ret;
+		}
+		of_node_put(dt_node);
+
+		ret = zynqmp_r5_pm_request_sram(rsc.start);
+		if (ret < 0)
+			return ret;
+
+		/* add carveout */
+		size = resource_size(&rsc);
+		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
+					   (size_t)size, rsc.start,
+					   tcm_mem_alloc,
+					   tcm_mem_release,
+					   rsc.name);
+		if (!mem) {
+			ret = zynqmp_pm_release_node(pnode_id);
+			if (ret)
+				dev_warn(dev,
+					 "fail to release node: %x ret: %x\n",
+					 pnode_id, ret);
+			return -ENOMEM;
+		}
+
+		mem->priv = (void *)(u64)pnode_id;
+		rproc_add_carveout(rproc, mem);
+	}
+
+	return 0;
+}
+
+/*
+ * zynqmp_r5_parse_fw()
+ * @rproc: single R5 core's corresponding rproc instance
+ * @fw: ptr to firmware to be loaded onto r5 core
+ *
+ * When loading firmware, ensure the necessary carveouts are in remoteproc
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
+{
+	int ret;
+
+	ret = parse_tcm_banks(rproc);
+	if (ret)
+		return ret;
+
+	ret = parse_mem_regions(rproc);
+	if (ret)
+		return ret;
+
+	ret = rproc_elf_load_rsc_table(rproc, fw);
+	if (ret == -EINVAL) {
+		/*
+		 * resource table only required for IPC.
+		 * if not present, this is not necessarily an error;
+		 * for example, loading r5 hello world application
+		 * so simply inform user and keep going.
+		 */
+		dev_info(&rproc->dev, "no resource table found.\n");
+		ret = 0;
+	}
+	return ret;
+}
+
+/*
+ * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
+ * @rproc: r5 core's corresponding rproc structure
+ * @vqid: virtqueue ID
+ */
+static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
+{
+	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
+	struct device *dev = rproc->dev.parent;
+	struct zynqmp_ipi_message *mb_msg = NULL;
+	unsigned int skb_len;
+	struct sk_buff *skb;
+	int ret;
+
+	/* If vqid is negative, do not pass the vqid to
+	 * mailbox as vqid is supposed to be 0 or positive.
+	 * It also gives a way to just kick instead but
+	 * not use the IPI buffer.
+	 *
+	 * For now use negative vqid to assume no message will be
+	 * passed with IPI buffer, but just raise interrupt.
+	 * This will be faster as it doesn't need to copy the
+	 * message to the ZynqMP/Versal IPI message buffer.
+	 */
+	if (vqid >= 0) {
+		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
+		skb = alloc_skb(skb_len, GFP_KERNEL);
+		if (!skb)
+			return;
+
+		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb, skb_len);
+		mb_msg->len = sizeof(vqid);
+		memcpy(mb_msg->data, &vqid, sizeof(vqid));
+
+		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
+	}
+
+	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
+	if (ret < 0) {
+		dev_warn(dev, "Failed to kick remote.\n");
+		if (mb_msg) {
+			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
+			kfree_skb(skb);
+		}
+	}
+}
+
+static struct rproc_ops zynqmp_r5_rproc_ops = {
+	.start		= zynqmp_r5_rproc_start,
+	.stop		= zynqmp_r5_rproc_stop,
+	.load		= rproc_elf_load_segments,
+	.parse_fw	= zynqmp_r5_parse_fw,
+	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
+	.sanity_check	= rproc_elf_sanity_check,
+	.get_boot_addr	= rproc_elf_get_boot_addr,
+	.kick		= zynqmp_r5_rproc_kick,
+};
+
+/**
+ * event_notified_idr_cb() - event notified idr callback
+ * @id: idr id
+ * @ptr: pointer to idr private data
+ * @data: data passed to idr_for_each callback
+ *
+ * Pass notification to remoteproc virtio
+ *
+ * Return: 0. having return is to satisfy the idr_for_each() function
+ *          pointer input argument requirement.
+ **/
+static int event_notified_idr_cb(int id, void *ptr, void *data)
+{
+	struct rproc *rproc = data;
+
+	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
+		dev_dbg(&rproc->dev, "no message was found in idr with id: %d\n", id);
+	return 0;
+}
+
+/**
+ * handle_event_notified() - remoteproc notification work function
+ * @work: pointer to the work structure
+ *
+ * It checks each registered remoteproc notify IDs.
+ */
+static void handle_event_notified(struct work_struct *work)
+{
+	struct zynqmp_r5_rproc *z_rproc;
+	struct rproc *rproc;
+
+	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
+	rproc = z_rproc->rproc;
+
+	/*
+	 * This is to simply raise interrupt as ACK for remote.
+	 *
+	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
+	 * mapping to remote processor.
+	 */
+	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
+		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
+
+	/*
+	 * We only use IPI for interrupt. The firmware side may or may
+	 * not write the notifyid when it trigger IPI.
+	 * And thus, we scan through all the registered notifyids.
+	 */
+	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
+}
+
+/**
+ * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
+ * @cl: mailbox client
+ * @msg: message pointer
+ *
+ * It will schedule the R5 notification work.
+ */
+static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
+{
+	struct zynqmp_r5_rproc *z_rproc;
+
+	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
+	/*
+	 * Notification can be received with message payload.
+	 * To handle this, check for message contents.
+	 * If there are contents, copy to ipi message buffer payload location.
+	 *
+	 * If message is empty, then that means that was simple raising
+	 * of interrupt. no payload to process.
+	 *
+	 * Note: enqueue work regardless of msg as kick may imply that remote
+	 * is waiting for ack.
+	 */
+	if (msg) {
+		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
+		size_t len;
+
+		ipi_msg = (struct zynqmp_ipi_message *)msg;
+		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
+		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
+		      IPI_BUF_LEN_MAX : ipi_msg->len;
+		buf_msg->len = len;
+		memcpy(buf_msg->data, ipi_msg->data, len);
+	}
+	schedule_work(&z_rproc->mbox_work);
+}
+
+/**
+ * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
+ * @cl: mailbox client
+ * @msg: pointer to the message which has been sent
+ * @r: status of last TX - OK or error
+ *
+ * It will be called by the mailbox framework when the last TX has done.
+ */
+static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg, int r)
+{
+	struct zynqmp_r5_rproc *z_rproc;
+	struct sk_buff *skb;
+
+	if (!msg)
+		return;
+	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
+	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
+	kfree_skb(skb);
+}
+
+/**
+ * zynqmp_r5_setup_mbox() - Setup mailboxes
+ *			    this is used for each individual R5 core
+ *
+ * @z_rproc: pointer to the ZynqMP R5 processor platform data
+ * @node: pointer of the device node
+ *
+ * Function to setup mailboxes to talk to RPU.
+ *
+ * Return: 0 for success, negative value for failure.
+ */
+static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
+				struct device_node *node)
+{
+	struct mbox_client *mclient;
+
+	/* Setup TX mailbox channel client */
+	mclient			= &z_rproc->tx_mc;
+	mclient->dev		= z_rproc->dev;
+	mclient->tx_done	= zynqmp_r5_mb_tx_done;
+	mclient->tx_block	= false;
+	mclient->knows_txdone	= false;
+	mclient->rx_callback	= NULL;
+
+	/* Setup RX mailbox channel client */
+	mclient			= &z_rproc->rx_mc;
+	mclient->dev		= z_rproc->dev;
+	mclient->tx_done	= NULL;
+	mclient->tx_block	= false;
+	mclient->knows_txdone	= false;
+	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
+
+	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
+
+	/* Request TX and RX channels */
+	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc->tx_mc, "tx");
+	if (IS_ERR(z_rproc->tx_chan)) {
+		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
+		return PTR_ERR(z_rproc->tx_chan);
+	}
+
+	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc->rx_mc, "rx");
+	if (IS_ERR(z_rproc->rx_chan)) {
+		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
+		return PTR_ERR(z_rproc->rx_chan);
+	}
+	skb_queue_head_init(&z_rproc->tx_mc_skbs);
+
+	return 0;
+}
+
+static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc *z_rproc)
+{
+	mbox_free_channel(z_rproc->tx_chan);
+	mbox_free_channel(z_rproc->rx_chan);
+}
+
+/**
+ * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
+ *		       this is called for each individual R5 core to
+ *		       set up mailbox, Xilinx platform manager unique ID,
+ *		       add to rproc core
+ *
+ * @pdev: domain platform device for current R5 core
+ * @node: pointer of the device node for current R5 core
+ * @rpu_mode: mode to configure RPU, split or lockstep
+ *
+ * Return: 0 for success, negative value for failure.
+ */
+static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device *pdev,
+					       struct device_node *node,
+					       enum rpu_oper_mode rpu_mode)
+{
+	int ret;
+	struct device *dev = &pdev->dev;
+	struct rproc *rproc_ptr;
+	struct zynqmp_r5_rproc *z_rproc;
+
+	/* Allocate remoteproc instance */
+	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
+				     NULL, sizeof(struct zynqmp_r5_rproc));
+	if (!rproc_ptr) {
+		ret = -ENOMEM;
+		goto error;
+	}
+
+	rproc_ptr->auto_boot = false;
+	z_rproc = rproc_ptr->priv;
+	z_rproc->rproc = rproc_ptr;
+	z_rproc->dev = dev;
+
+	/* Set up DMA mask */
+	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
+	if (ret)
+		goto error;
+
+	/* Get R5 power domain node */
+	ret = of_property_read_u32(node, "power-domain", &z_rproc->pnode_id);
+	if (ret)
+		goto error;
+
+	ret = r5_set_mode(z_rproc, rpu_mode);
+	if (ret)
+		goto error;
+
+	if (of_property_read_bool(node, "mboxes")) {
+		ret = zynqmp_r5_setup_mbox(z_rproc, node);
+		if (ret)
+			goto error;
+	}
+
+	/* Add R5 remoteproc */
+	ret = devm_rproc_add(dev, rproc_ptr);
+	if (ret) {
+		zynqmp_r5_cleanup_mbox(z_rproc);
+		goto error;
+	}
+
+	return z_rproc;
+error:
+	return ERR_PTR(ret);
+}
+
+/*
+ * zynqmp_r5_remoteproc_probe()
+ *
+ * @pdev: domain platform device for R5 cluster
+ *
+ * called when driver is probed, for each R5 core specified in DT,
+ * setup as needed to do remoteproc-related operations
+ *
+ * Return: 0 for success, negative value for failure.
+ */
+static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
+{
+	int ret, core_count;
+	struct device *dev = &pdev->dev;
+	struct device_node *nc;
+	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
+	struct list_head *cluster; /* list to track each core's rproc */
+	struct zynqmp_r5_rproc *z_rproc;
+	struct platform_device *child_pdev;
+	struct list_head *pos;
+
+	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode", &rpu_mode);
+	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
+			rpu_mode != PM_RPU_MODE_SPLIT)) {
+		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
+			ret, rpu_mode);
+		return ret;
+	}
+
+	dev_dbg(dev, "RPU configuration: %s\n",
+		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" : "split");
+
+	/*
+	 * if 2 RPUs provided but one is lockstep, then we have an
+	 * invalid configuration.
+	 */
+
+	core_count = of_get_available_child_count(dev->of_node);
+	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
+	    core_count > MAX_RPROCS)
+		return -EINVAL;
+
+	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
+	if (!cluster)
+		return -ENOMEM;
+	INIT_LIST_HEAD(cluster);
+
+	ret = devm_of_platform_populate(dev);
+	if (ret) {
+		dev_err(dev, "devm_of_platform_populate failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	/* probe each individual r5 core's remoteproc-related info */
+	for_each_available_child_of_node(dev->of_node, nc) {
+		child_pdev = of_find_device_by_node(nc);
+		if (!child_pdev) {
+			dev_err(dev, "could not get R5 core platform device\n");
+			ret = -ENODEV;
+			goto out;
+		}
+
+		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
+		dev_dbg(dev, "%s to probe rpu %pOF\n",
+			ret ? "Failed" : "Able",
+			nc);
+		if (IS_ERR(z_rproc)) {
+			ret = PTR_ERR(z_rproc);
+			goto out;
+		}
+		list_add_tail(&z_rproc->elem, cluster);
+	}
+	/* wire in so each core can be cleaned up at driver remove */
+	platform_set_drvdata(pdev, cluster);
+	return 0;
+out:
+	/*
+	 * undo core0 upon any failures on core1 in split-mode
+	 *
+	 * in zynqmp_r5_probe z_rproc is set to null
+	 * and ret to non-zero value if error
+	 */
+	list_for_each(pos, cluster) {
+		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
+		zynqmp_r5_cleanup_mbox(z_rproc);
+	}
+	return ret;
+}
+
+/*
+ * zynqmp_r5_remoteproc_remove()
+ *
+ * @pdev: domain platform device for R5 cluster
+ *
+ * When the driver is unloaded, clean up the mailboxes for each
+ * remoteproc that was initially probed.
+ */
+static int zynqmp_r5_remoteproc_remove(struct platform_device *pdev)
+{
+	struct list_head *pos, *temp, *cluster = (struct list_head *)
+						 platform_get_drvdata(pdev);
+	struct zynqmp_r5_rproc *z_rproc = NULL;
+
+	list_for_each_safe(pos, temp, cluster) {
+		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
+		zynqmp_r5_cleanup_mbox(z_rproc);
+		list_del(pos);
+	}
+	return 0;
+}
+
+/* Match table for OF platform binding */
+static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
+	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
+	{ /* end of list */ },
+};
+MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
+
+static struct platform_driver zynqmp_r5_remoteproc_driver = {
+	.probe = zynqmp_r5_remoteproc_probe,
+	.remove = zynqmp_r5_remoteproc_remove,
+	.driver = {
+		.name = "zynqmp_r5_remoteproc",
+		.of_match_table = zynqmp_r5_remoteproc_match,
+	},
+};
+module_platform_driver(zynqmp_r5_remoteproc_driver);
+
+MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
+MODULE_LICENSE("GPL v2");
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 28+ messages in thread

* [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
@ 2021-01-11  2:02   ` Ben Levinsky
  0 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-01-11  2:02 UTC (permalink / raw)
  To: mathieu.poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
remotproc driver, we can boot the R5 sub-system in two different
configurations -
	* Split
	* Lockstep

The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
Platform Management Unit that handles the R5 configuration, memory access
and R5 lifecycle management. The interface to this manager is done in this
driver via zynqmp_pm_* function calls.

Signed-off-by: Wendy Liang <wendy.liang@xilinx.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Ed Mooring <ed.mooring@xilinx.com>
Signed-off-by: Jason Wu <j.wu@xilinx.com>
Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
---
v25:
- reword error message for rpu configuration
- change char name[15] to 16 chars to have null terminated string
  in parse_mem_regions()
- reword comments in tcm_mem_alloc
- call xilinx platform cleanup in parse_tcm_banks() case of failure
- check value of mbox_send_message in all instances within this driver 
- indentation in zynqmp_r5_remoteproc driver
- update style of constructing carveouts to match convention in ST
  remoteproc drivers. No longer iterate through memory-region property
  using of_count_phandle_with_args. Instead use of_phandle_iterator
- use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per feedback
- rework loop in zynqmp_r5_pm_request_sram as per feedback
- fix comment and linebreak in tcm_mem_alloc description
- add comments and remove extraneous devm_ioremap_wc call in tcm_mem_alloc
- in parse_tcm_banks remove check for bank not being available
- rework order of locals and remove else in zynqmp_r5_rproc_kick
- document and update logic for zynqmp_r5_parse_fw
- add dev_dbg output in event_notified_idr_cb if
  rproc_vq_interrupt(rproc, id) == IRQ_NONE
- add comment for handle_event_notified mbox_send_message function call
- add comment for zynqmp_r5_mb_rx_cb
- update zynqmp_r5_setup_mbox to match convention of mbox setup in ST
  remoteproc drivers
- change return in zynqmp_r5_setup_mbox to use PTR_ERR
- add zynqmp_r5_cleanup_mbox
- in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as arg.
- in zynqmp_r5_probe use return of PTR_ERR and use zynqmp_r5_cleanup_mbox
- in zynqmp_r5_remoteproc_probe update use return value of zynqmp_r5_probe as
  either zynqmp_r5_rproc* or PTR_ERR
- update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_probe
- update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_remove
---
 drivers/remoteproc/Kconfig                |   8 +
 drivers/remoteproc/Makefile               |   1 +
 drivers/remoteproc/zynqmp_r5_remoteproc.c | 898 ++++++++++++++++++++++
 3 files changed, 907 insertions(+)
 create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c

diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
index c6659dfea7c7..c2fe54b1d94f 100644
--- a/drivers/remoteproc/Kconfig
+++ b/drivers/remoteproc/Kconfig
@@ -275,6 +275,14 @@ config TI_K3_DSP_REMOTEPROC
 	  It's safe to say N here if you're not interested in utilizing
 	  the DSP slave processors.
 
+config ZYNQMP_R5_REMOTEPROC
+	tristate "ZynqMP R5 remoteproc support"
+	depends on PM && ARCH_ZYNQMP
+	select RPMSG_VIRTIO
+	select ZYNQMP_IPI_MBOX
+	help
+	  Say y or m here to support ZynqMP R5 remote processors via the remote
+	  processor framework.
 endif # REMOTEPROC
 
 endmenu
diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
index 3dfa28e6c701..ef1abff654c2 100644
--- a/drivers/remoteproc/Makefile
+++ b/drivers/remoteproc/Makefile
@@ -33,3 +33,4 @@ obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
 obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
 obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
 obj-$(CONFIG_TI_K3_DSP_REMOTEPROC)	+= ti_k3_dsp_remoteproc.o
+obj-$(CONFIG_ZYNQMP_R5_REMOTEPROC)	+= zynqmp_r5_remoteproc.o
diff --git a/drivers/remoteproc/zynqmp_r5_remoteproc.c b/drivers/remoteproc/zynqmp_r5_remoteproc.c
new file mode 100644
index 000000000000..9e228c9ecd43
--- /dev/null
+++ b/drivers/remoteproc/zynqmp_r5_remoteproc.c
@@ -0,0 +1,898 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Zynq R5 Remote Processor driver
+ *
+ * Based on origin OMAP and Zynq Remote Processor driver
+ *
+ */
+
+#include <linux/firmware/xlnx-zynqmp.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/mailbox_client.h>
+#include <linux/mailbox/zynqmp-ipi-message.h>
+#include <linux/module.h>
+#include <linux/of_address.h>
+#include <linux/of_platform.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/remoteproc.h>
+#include <linux/skbuff.h>
+#include <linux/sysfs.h>
+
+#include "remoteproc_internal.h"
+
+#define MAX_RPROCS	2 /* Support up to 2 RPU */
+#define MAX_MEM_PNODES	4 /* Max power nodes for one RPU memory instance */
+
+#define BANK_LIST_PROP	"sram"
+#define DDR_LIST_PROP	"memory-region"
+
+/* IPI buffer MAX length */
+#define IPI_BUF_LEN_MAX	32U
+/* RX mailbox client buffer max length */
+#define RX_MBOX_CLIENT_BUF_MAX	(IPI_BUF_LEN_MAX + \
+				 sizeof(struct zynqmp_ipi_message))
+
+/*
+ * Map each Xilinx on-chip SRAM  Bank address to their own respective
+ * pm_node_id.
+ */
+struct sram_addr_data {
+	phys_addr_t addr;
+	enum pm_node_id id;
+};
+
+#define NUM_SRAMS 4U
+static const struct sram_addr_data zynqmp_banks[NUM_SRAMS] = {
+	{0xffe00000UL, NODE_TCM_0_A},
+	{0xffe20000UL, NODE_TCM_0_B},
+	{0xffe90000UL, NODE_TCM_1_A},
+	{0xffeb0000UL, NODE_TCM_1_B},
+};
+
+/**
+ * struct zynqmp_r5_rproc - ZynqMP R5 core structure
+ *
+ * @rx_mc_buf: rx mailbox client buffer to save the rx message
+ * @tx_mc: tx mailbox client
+ * @rx_mc: rx mailbox client
+ * @mbox_work: mbox_work for the RPU remoteproc
+ * @tx_mc_skbs: socket buffers for tx mailbox client
+ * @dev: device of RPU instance
+ * @rproc: rproc handle
+ * @tx_chan: tx mailbox channel
+ * @rx_chan: rx mailbox channel
+ * @pnode_id: RPU CPU power domain id
+ * @elem: linked list item
+ */
+struct zynqmp_r5_rproc {
+	unsigned char rx_mc_buf[RX_MBOX_CLIENT_BUF_MAX];
+	struct mbox_client tx_mc;
+	struct mbox_client rx_mc;
+	struct work_struct mbox_work;
+	struct sk_buff_head tx_mc_skbs;
+	struct device *dev;
+	struct rproc *rproc;
+	struct mbox_chan *tx_chan;
+	struct mbox_chan *rx_chan;
+	u32 pnode_id;
+	struct list_head elem;
+};
+
+/*
+ * r5_set_mode - set RPU operation mode
+ * @z_rproc: Remote processor private data
+ * @rpu_mode: mode specified by device tree to configure the RPU to
+ *
+ * set RPU operation mode
+ *
+ * Return: 0 for success, negative value for failure
+ */
+static int r5_set_mode(struct zynqmp_r5_rproc *z_rproc,
+		       enum rpu_oper_mode rpu_mode)
+{
+	enum rpu_tcm_comb tcm_mode;
+	enum rpu_oper_mode cur_rpu_mode;
+	int ret;
+
+	ret = zynqmp_pm_get_rpu_mode(z_rproc->pnode_id, &cur_rpu_mode);
+	if (ret < 0)
+		return ret;
+
+	if (rpu_mode != cur_rpu_mode) {
+		ret = zynqmp_pm_set_rpu_mode(z_rproc->pnode_id, rpu_mode);
+		if (ret < 0)
+			return ret;
+	}
+
+	tcm_mode = (rpu_mode == PM_RPU_MODE_LOCKSTEP) ?
+		    PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
+	return zynqmp_pm_set_tcm_config(z_rproc->pnode_id, tcm_mode);
+}
+
+/*
+ * zynqmp_r5_rproc_mem_release
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to unmap
+ *
+ * Unmap TCM banks when powering down R5 core.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int tcm_mem_release(struct rproc *rproc, struct rproc_mem_entry *mem)
+{
+	u32 pnode_id = (u64)mem->priv;
+
+	iounmap(mem->va);
+	return zynqmp_pm_release_node(pnode_id);
+}
+
+/*
+ * zynqmp_r5_rproc_start
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Start R5 Core from designated boot address.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_start(struct rproc *rproc)
+{
+	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
+	enum rpu_boot_mem bootmem;
+
+	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
+		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
+
+	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
+		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
+
+	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
+				      bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
+}
+
+/*
+ * zynqmp_r5_rproc_stop
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Power down  R5 Core.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_stop(struct rproc *rproc)
+{
+	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
+
+	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
+				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+}
+
+/*
+ * zynqmp_r5_rproc_mem_alloc
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to map
+ *
+ * Callback to map va for memory-region's carveout.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
+				     struct rproc_mem_entry *mem)
+{
+	void *va;
+
+	va = ioremap_wc(mem->dma, mem->len);
+	if (IS_ERR_OR_NULL(va))
+		return -ENOMEM;
+
+	mem->va = va;
+
+	return 0;
+}
+
+/*
+ * zynqmp_r5_rproc_mem_release
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to unmap
+ *
+ * Unmap memory-region carveout
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
+				       struct rproc_mem_entry *mem)
+{
+	iounmap(mem->va);
+	return 0;
+}
+
+/*
+ * parse_mem_regions
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Construct rproc mem carveouts from carveout provided in
+ * memory-region property
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int parse_mem_regions(struct rproc *rproc)
+{
+	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
+	struct device_node *np = z_rproc->dev->of_node;
+	struct device *dev = &rproc->dev;
+	struct of_phandle_iterator it;
+	struct rproc_mem_entry *mem;
+	struct reserved_mem *rmem;
+	int index = 0;
+
+	/* Register associated reserved memory regions */
+	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
+	while (of_phandle_iterator_next(&it) == 0) {
+		rmem = of_reserved_mem_lookup(it.node);
+		if (!rmem) {
+			dev_err(dev, "unable to acquire %s\n", DDR_LIST_PROP);
+			return -EINVAL;
+		}
+
+		if (strstr(it.node->name, "vdev0buffer")) {
+			mem = rproc_of_resm_mem_entry_init(dev, index,
+							   rmem->size,
+							   rmem->base,
+							   "vdev0buffer");
+		} else {
+			/*
+			 * The ensuing carveout is either for vring, which has
+			 * device tree node name of length 15 characters, or
+			 * firmware mem, where name is not used by remoteproc
+			 *  core later on. So default to vring length of 15.
+			 *
+			 * Extra char for null-terminated string.
+			 */
+			char name[16];
+
+			if (strstr(it.node->name, "vdev0vring")) {
+				int vring_id;
+
+				/*
+				 * Expect "rpuXvdev0vringX" as documented
+				 * in xilinx remoteproc device tree binding
+				 */
+				if (strlen(it.node->name) < 15) {
+					dev_err(dev, "%pOF is less than 15 chars",
+						it.node);
+					return -EINVAL;
+				}
+
+				/*
+				 * can be 1 of multiple vring IDs per IPC channel
+				 * e.g. 'vdev0vring0' and 'vdev0vring1'
+				 *
+				 * Here get vring ID from 15th char
+				 */
+				vring_id = it.node->name[14] - '0';
+				snprintf(name, sizeof(name), "vdev0vring%d", vring_id);
+			} else {
+				strncpy(name, it.node->name, 16);
+			}
+
+			mem = rproc_mem_entry_init(dev, NULL,
+						   (dma_addr_t)rmem->base,
+						   rmem->size, rmem->base,
+						   zynqmp_r5_rproc_mem_alloc,
+						   zynqmp_r5_rproc_mem_release,
+						   name);
+		}
+
+		if (!mem)
+			return -ENOMEM;
+
+		rproc_add_carveout(rproc, mem);
+		index++;
+	}
+
+	return 0;
+}
+
+/*
+ * zynqmp_r5_pm_request_tcm
+ * @addr: base address of mem provided in R5 core's sram property.
+ *
+ * Given sram base address, determine its corresponding Xilinx
+ * Platform Management ID and then request access to this node
+ * so that it can be power up.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
+{
+	int i;
+
+	for (i = 0; i < NUM_SRAMS; i++) {
+		if (zynqmp_banks[i].addr != addr)
+			continue;
+
+		return zynqmp_pm_request_node(zynqmp_banks[i].id,
+					      ZYNQMP_PM_CAPABILITY_ACCESS, 0,
+					      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+	}
+
+	return -EINVAL;
+}
+
+/*
+ * tcm_mem_alloc
+ * @rproc: single R5 core's corresponding rproc instance
+ * @mem: mem entry to initialize the va and da fields of
+ *
+ * Given TCM bank entry, this callback will set device address for R5
+ * running on TCM and also setup virtual address for TCM bank
+ * remoteproc carveout.
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int tcm_mem_alloc(struct rproc *rproc,
+			 struct rproc_mem_entry *mem)
+{
+	void *va;
+	struct device *dev = rproc->dev.parent;
+
+	va = ioremap_wc(mem->dma, mem->len);
+	if (IS_ERR_OR_NULL(va))
+		return -ENOMEM;
+
+	/* Update memory entry va */
+	mem->va = va;
+
+	/*
+	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
+	 * while on the Linux side they are at 0xffexxxxx.
+	 *
+	 * Zero out the high 12 bits of the address. This will give
+	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
+	 */
+	mem->da &= 0x000fffff;
+
+	/*
+	 * TCM Banks 1A and 1B still have to be translated.
+	 *
+	 * Below handle these two banks' absolute addresses (0xffe90000 and
+	 * 0xffeb0000) and convert to the expected relative addresses
+	 * (0x0 and 0x20000).
+	 */
+	if (mem->da == 0x90000 || mem->da == 0xB0000)
+		mem->da -= 0x90000;
+
+	/* if translated TCM bank address is not valid report error */
+	if (mem->da != 0x0 && mem->da != 0x20000) {
+		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/*
+ * parse_tcm_banks()
+ * @rproc: single R5 core's corresponding rproc instance
+ *
+ * Given R5 node in remoteproc instance
+ * allocate remoteproc carveout for TCM memory
+ * needed for firmware to be loaded
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int parse_tcm_banks(struct rproc *rproc)
+{
+	int i, num_banks;
+	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
+	struct device *dev = &rproc->dev;
+	struct device_node *r5_node = z_rproc->dev->of_node;
+
+	/* go through TCM banks for r5 node */
+	num_banks = of_count_phandle_with_args(r5_node, BANK_LIST_PROP, NULL);
+	if (num_banks <= 0) {
+		dev_err(dev, "need to specify TCM banks\n");
+		return -EINVAL;
+	}
+	for (i = 0; i < num_banks; i++) {
+		struct resource rsc;
+		resource_size_t size;
+		struct device_node *dt_node;
+		struct rproc_mem_entry *mem;
+		int ret;
+		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
+
+		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
+		if (!dt_node)
+			return -EINVAL;
+
+		ret = of_address_to_resource(dt_node, 0, &rsc);
+		if (ret < 0) {
+			of_node_put(dt_node);
+			return ret;
+		}
+		of_node_put(dt_node);
+
+		ret = zynqmp_r5_pm_request_sram(rsc.start);
+		if (ret < 0)
+			return ret;
+
+		/* add carveout */
+		size = resource_size(&rsc);
+		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
+					   (size_t)size, rsc.start,
+					   tcm_mem_alloc,
+					   tcm_mem_release,
+					   rsc.name);
+		if (!mem) {
+			ret = zynqmp_pm_release_node(pnode_id);
+			if (ret)
+				dev_warn(dev,
+					 "fail to release node: %x ret: %x\n",
+					 pnode_id, ret);
+			return -ENOMEM;
+		}
+
+		mem->priv = (void *)(u64)pnode_id;
+		rproc_add_carveout(rproc, mem);
+	}
+
+	return 0;
+}
+
+/*
+ * zynqmp_r5_parse_fw()
+ * @rproc: single R5 core's corresponding rproc instance
+ * @fw: ptr to firmware to be loaded onto r5 core
+ *
+ * When loading firmware, ensure the necessary carveouts are in remoteproc
+ *
+ * return 0 on success, otherwise non-zero value on failure
+ */
+static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
+{
+	int ret;
+
+	ret = parse_tcm_banks(rproc);
+	if (ret)
+		return ret;
+
+	ret = parse_mem_regions(rproc);
+	if (ret)
+		return ret;
+
+	ret = rproc_elf_load_rsc_table(rproc, fw);
+	if (ret == -EINVAL) {
+		/*
+		 * resource table only required for IPC.
+		 * if not present, this is not necessarily an error;
+		 * for example, loading r5 hello world application
+		 * so simply inform user and keep going.
+		 */
+		dev_info(&rproc->dev, "no resource table found.\n");
+		ret = 0;
+	}
+	return ret;
+}
+
+/*
+ * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
+ * @rproc: r5 core's corresponding rproc structure
+ * @vqid: virtqueue ID
+ */
+static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
+{
+	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
+	struct device *dev = rproc->dev.parent;
+	struct zynqmp_ipi_message *mb_msg = NULL;
+	unsigned int skb_len;
+	struct sk_buff *skb;
+	int ret;
+
+	/* If vqid is negative, do not pass the vqid to
+	 * mailbox as vqid is supposed to be 0 or positive.
+	 * It also gives a way to just kick instead but
+	 * not use the IPI buffer.
+	 *
+	 * For now use negative vqid to assume no message will be
+	 * passed with IPI buffer, but just raise interrupt.
+	 * This will be faster as it doesn't need to copy the
+	 * message to the ZynqMP/Versal IPI message buffer.
+	 */
+	if (vqid >= 0) {
+		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
+		skb = alloc_skb(skb_len, GFP_KERNEL);
+		if (!skb)
+			return;
+
+		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb, skb_len);
+		mb_msg->len = sizeof(vqid);
+		memcpy(mb_msg->data, &vqid, sizeof(vqid));
+
+		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
+	}
+
+	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
+	if (ret < 0) {
+		dev_warn(dev, "Failed to kick remote.\n");
+		if (mb_msg) {
+			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
+			kfree_skb(skb);
+		}
+	}
+}
+
+static struct rproc_ops zynqmp_r5_rproc_ops = {
+	.start		= zynqmp_r5_rproc_start,
+	.stop		= zynqmp_r5_rproc_stop,
+	.load		= rproc_elf_load_segments,
+	.parse_fw	= zynqmp_r5_parse_fw,
+	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
+	.sanity_check	= rproc_elf_sanity_check,
+	.get_boot_addr	= rproc_elf_get_boot_addr,
+	.kick		= zynqmp_r5_rproc_kick,
+};
+
+/**
+ * event_notified_idr_cb() - event notified idr callback
+ * @id: idr id
+ * @ptr: pointer to idr private data
+ * @data: data passed to idr_for_each callback
+ *
+ * Pass notification to remoteproc virtio
+ *
+ * Return: 0. having return is to satisfy the idr_for_each() function
+ *          pointer input argument requirement.
+ **/
+static int event_notified_idr_cb(int id, void *ptr, void *data)
+{
+	struct rproc *rproc = data;
+
+	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
+		dev_dbg(&rproc->dev, "no message was found in idr with id: %d\n", id);
+	return 0;
+}
+
+/**
+ * handle_event_notified() - remoteproc notification work function
+ * @work: pointer to the work structure
+ *
+ * It checks each registered remoteproc notify IDs.
+ */
+static void handle_event_notified(struct work_struct *work)
+{
+	struct zynqmp_r5_rproc *z_rproc;
+	struct rproc *rproc;
+
+	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
+	rproc = z_rproc->rproc;
+
+	/*
+	 * This is to simply raise interrupt as ACK for remote.
+	 *
+	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
+	 * mapping to remote processor.
+	 */
+	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
+		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
+
+	/*
+	 * We only use IPI for interrupt. The firmware side may or may
+	 * not write the notifyid when it trigger IPI.
+	 * And thus, we scan through all the registered notifyids.
+	 */
+	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
+}
+
+/**
+ * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
+ * @cl: mailbox client
+ * @msg: message pointer
+ *
+ * It will schedule the R5 notification work.
+ */
+static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
+{
+	struct zynqmp_r5_rproc *z_rproc;
+
+	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
+	/*
+	 * Notification can be received with message payload.
+	 * To handle this, check for message contents.
+	 * If there are contents, copy to ipi message buffer payload location.
+	 *
+	 * If message is empty, then that means that was simple raising
+	 * of interrupt. no payload to process.
+	 *
+	 * Note: enqueue work regardless of msg as kick may imply that remote
+	 * is waiting for ack.
+	 */
+	if (msg) {
+		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
+		size_t len;
+
+		ipi_msg = (struct zynqmp_ipi_message *)msg;
+		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
+		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
+		      IPI_BUF_LEN_MAX : ipi_msg->len;
+		buf_msg->len = len;
+		memcpy(buf_msg->data, ipi_msg->data, len);
+	}
+	schedule_work(&z_rproc->mbox_work);
+}
+
+/**
+ * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
+ * @cl: mailbox client
+ * @msg: pointer to the message which has been sent
+ * @r: status of last TX - OK or error
+ *
+ * It will be called by the mailbox framework when the last TX has done.
+ */
+static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg, int r)
+{
+	struct zynqmp_r5_rproc *z_rproc;
+	struct sk_buff *skb;
+
+	if (!msg)
+		return;
+	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
+	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
+	kfree_skb(skb);
+}
+
+/**
+ * zynqmp_r5_setup_mbox() - Setup mailboxes
+ *			    this is used for each individual R5 core
+ *
+ * @z_rproc: pointer to the ZynqMP R5 processor platform data
+ * @node: pointer of the device node
+ *
+ * Function to setup mailboxes to talk to RPU.
+ *
+ * Return: 0 for success, negative value for failure.
+ */
+static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
+				struct device_node *node)
+{
+	struct mbox_client *mclient;
+
+	/* Setup TX mailbox channel client */
+	mclient			= &z_rproc->tx_mc;
+	mclient->dev		= z_rproc->dev;
+	mclient->tx_done	= zynqmp_r5_mb_tx_done;
+	mclient->tx_block	= false;
+	mclient->knows_txdone	= false;
+	mclient->rx_callback	= NULL;
+
+	/* Setup RX mailbox channel client */
+	mclient			= &z_rproc->rx_mc;
+	mclient->dev		= z_rproc->dev;
+	mclient->tx_done	= NULL;
+	mclient->tx_block	= false;
+	mclient->knows_txdone	= false;
+	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
+
+	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
+
+	/* Request TX and RX channels */
+	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc->tx_mc, "tx");
+	if (IS_ERR(z_rproc->tx_chan)) {
+		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
+		return PTR_ERR(z_rproc->tx_chan);
+	}
+
+	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc->rx_mc, "rx");
+	if (IS_ERR(z_rproc->rx_chan)) {
+		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
+		return PTR_ERR(z_rproc->rx_chan);
+	}
+	skb_queue_head_init(&z_rproc->tx_mc_skbs);
+
+	return 0;
+}
+
+static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc *z_rproc)
+{
+	mbox_free_channel(z_rproc->tx_chan);
+	mbox_free_channel(z_rproc->rx_chan);
+}
+
+/**
+ * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
+ *		       this is called for each individual R5 core to
+ *		       set up mailbox, Xilinx platform manager unique ID,
+ *		       add to rproc core
+ *
+ * @pdev: domain platform device for current R5 core
+ * @node: pointer of the device node for current R5 core
+ * @rpu_mode: mode to configure RPU, split or lockstep
+ *
+ * Return: 0 for success, negative value for failure.
+ */
+static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device *pdev,
+					       struct device_node *node,
+					       enum rpu_oper_mode rpu_mode)
+{
+	int ret;
+	struct device *dev = &pdev->dev;
+	struct rproc *rproc_ptr;
+	struct zynqmp_r5_rproc *z_rproc;
+
+	/* Allocate remoteproc instance */
+	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
+				     NULL, sizeof(struct zynqmp_r5_rproc));
+	if (!rproc_ptr) {
+		ret = -ENOMEM;
+		goto error;
+	}
+
+	rproc_ptr->auto_boot = false;
+	z_rproc = rproc_ptr->priv;
+	z_rproc->rproc = rproc_ptr;
+	z_rproc->dev = dev;
+
+	/* Set up DMA mask */
+	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
+	if (ret)
+		goto error;
+
+	/* Get R5 power domain node */
+	ret = of_property_read_u32(node, "power-domain", &z_rproc->pnode_id);
+	if (ret)
+		goto error;
+
+	ret = r5_set_mode(z_rproc, rpu_mode);
+	if (ret)
+		goto error;
+
+	if (of_property_read_bool(node, "mboxes")) {
+		ret = zynqmp_r5_setup_mbox(z_rproc, node);
+		if (ret)
+			goto error;
+	}
+
+	/* Add R5 remoteproc */
+	ret = devm_rproc_add(dev, rproc_ptr);
+	if (ret) {
+		zynqmp_r5_cleanup_mbox(z_rproc);
+		goto error;
+	}
+
+	return z_rproc;
+error:
+	return ERR_PTR(ret);
+}
+
+/*
+ * zynqmp_r5_remoteproc_probe()
+ *
+ * @pdev: domain platform device for R5 cluster
+ *
+ * called when driver is probed, for each R5 core specified in DT,
+ * setup as needed to do remoteproc-related operations
+ *
+ * Return: 0 for success, negative value for failure.
+ */
+static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
+{
+	int ret, core_count;
+	struct device *dev = &pdev->dev;
+	struct device_node *nc;
+	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
+	struct list_head *cluster; /* list to track each core's rproc */
+	struct zynqmp_r5_rproc *z_rproc;
+	struct platform_device *child_pdev;
+	struct list_head *pos;
+
+	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode", &rpu_mode);
+	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
+			rpu_mode != PM_RPU_MODE_SPLIT)) {
+		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
+			ret, rpu_mode);
+		return ret;
+	}
+
+	dev_dbg(dev, "RPU configuration: %s\n",
+		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" : "split");
+
+	/*
+	 * if 2 RPUs provided but one is lockstep, then we have an
+	 * invalid configuration.
+	 */
+
+	core_count = of_get_available_child_count(dev->of_node);
+	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
+	    core_count > MAX_RPROCS)
+		return -EINVAL;
+
+	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
+	if (!cluster)
+		return -ENOMEM;
+	INIT_LIST_HEAD(cluster);
+
+	ret = devm_of_platform_populate(dev);
+	if (ret) {
+		dev_err(dev, "devm_of_platform_populate failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	/* probe each individual r5 core's remoteproc-related info */
+	for_each_available_child_of_node(dev->of_node, nc) {
+		child_pdev = of_find_device_by_node(nc);
+		if (!child_pdev) {
+			dev_err(dev, "could not get R5 core platform device\n");
+			ret = -ENODEV;
+			goto out;
+		}
+
+		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
+		dev_dbg(dev, "%s to probe rpu %pOF\n",
+			ret ? "Failed" : "Able",
+			nc);
+		if (IS_ERR(z_rproc)) {
+			ret = PTR_ERR(z_rproc);
+			goto out;
+		}
+		list_add_tail(&z_rproc->elem, cluster);
+	}
+	/* wire in so each core can be cleaned up at driver remove */
+	platform_set_drvdata(pdev, cluster);
+	return 0;
+out:
+	/*
+	 * undo core0 upon any failures on core1 in split-mode
+	 *
+	 * in zynqmp_r5_probe z_rproc is set to null
+	 * and ret to non-zero value if error
+	 */
+	list_for_each(pos, cluster) {
+		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
+		zynqmp_r5_cleanup_mbox(z_rproc);
+	}
+	return ret;
+}
+
+/*
+ * zynqmp_r5_remoteproc_remove()
+ *
+ * @pdev: domain platform device for R5 cluster
+ *
+ * When the driver is unloaded, clean up the mailboxes for each
+ * remoteproc that was initially probed.
+ */
+static int zynqmp_r5_remoteproc_remove(struct platform_device *pdev)
+{
+	struct list_head *pos, *temp, *cluster = (struct list_head *)
+						 platform_get_drvdata(pdev);
+	struct zynqmp_r5_rproc *z_rproc = NULL;
+
+	list_for_each_safe(pos, temp, cluster) {
+		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
+		zynqmp_r5_cleanup_mbox(z_rproc);
+		list_del(pos);
+	}
+	return 0;
+}
+
+/* Match table for OF platform binding */
+static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
+	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
+	{ /* end of list */ },
+};
+MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
+
+static struct platform_driver zynqmp_r5_remoteproc_driver = {
+	.probe = zynqmp_r5_remoteproc_probe,
+	.remove = zynqmp_r5_remoteproc_remove,
+	.driver = {
+		.name = "zynqmp_r5_remoteproc",
+		.of_match_table = zynqmp_r5_remoteproc_match,
+	},
+};
+module_platform_driver(zynqmp_r5_remoteproc_driver);
+
+MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
+MODULE_LICENSE("GPL v2");
-- 
2.17.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 28+ messages in thread

* RE: [PATCH v25 0/5] Provide basic driver to control Arm R5 co-processor found on Xilinx ZynqMP
  2021-01-11  2:02 ` Ben Levinsky
                   ` (5 preceding siblings ...)
  (?)
@ 2021-02-04 15:20 ` Ben Levinsky
  2021-02-04 17:04   ` Mathieu Poirier
  -1 siblings, 1 reply; 28+ messages in thread
From: Ben Levinsky @ 2021-02-04 15:20 UTC (permalink / raw)
  To: linux-remoteproc

Ping for comments

> -----Original Message-----
> From: Ben Levinsky <ben.levinsky@xilinx.com>
> Sent: Sunday, January 10, 2021 6:03 PM
> To: mathieu.poirier@linaro.org
> Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal Simek
> <michals@xilinx.com>
> Subject: [PATCH v25 0/5] Provide basic driver to control Arm R5 co-processor
> found on Xilinx ZynqMP
> 
> R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
> remotproc driver, we can boot the R5 sub-system in two different
> configurations -
> 	* Split
> 	* Lockstep
> 
> The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
> 
> v25:
> - reword error message for rpu configuration
> - change char name[15] to 16 chars to have null terminated string
>   in parse_mem_regions()
> - reword comments in tcm_mem_alloc
> - grammar in device tree bindings
> - call xilinx platform cleanup in parse_tcm_banks() case of failure
> - check value of mbox_send_message in all calls within this driver
> - update include/linux/firmware/xlnx-zynqmp.h style to match for enum
>   pm_node_id
> - indentation in zynqmp_r5_remoteproc driver
> - update style of constructing carveouts to match convention in ST
>   remoteproc drivers. No longer iterate through memory-region property
>   using of_count_phandle_with_args. Instead use of_phandle_iterator
> - use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per
> feedback
> - rework loop in zynqmp_r5_pm_request_sram as per feedback
> - fix comment and linebreak in tcm_mem_alloc description
> - add comments and remove extraneous devm_ioremap_wc call in
> tcm_mem_alloc
> - in parse_tcm_banks remove check for bank not being available
> - rework order of locals and remove else in zynqmp_r5_rproc_kick
> - document and update logic for zynqmp_r5_parse_fw
> - add dev_dbg output in event_notified_idr_cb if
>   rproc_vq_interrupt(rproc, id) == IRQ_NONE
> - add comment for handle_event_notified mbox_send_message function call
> - add comment for zynqmp_r5_mb_rx_cb
> - update zynqmp_r5_setup_mbox to match convention of mbox setup in ST
>   remoteproc drivers
> - change return in zynqmp_r5_setup_mbox to use PTR_ERR
> - add zynqmp_r5_cleanup_mbox
> - in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as arg.
> - in zynqmp_r5_probe use return of PTR_ERR and use
> zynqmp_r5_cleanup_mbox
> - in zynqmp_r5_remoteproc_probe update use return value of
> zynqmp_r5_probe as
>   either zynqmp_r5_rproc* or PTR_ERR
> - update loop that cleans up cluster and mboxes in
> zynqmp_r5_remoteproc_probe
> - update loop that cleans up cluster and mboxes in
> zynqmp_r5_remoteproc_remove
> 
> Previous version:
> https://patchwork.kernel.org/project/linux-remoteproc/list/?series=393387
> 
> 
> Ben Levinsky (5):
>   firmware: xilinx: Add ZynqMP firmware ioctl enums for RPU
>     configuration.
>   firmware: xilinx: Add shutdown/wakeup APIs
>   firmware: xilinx: Add RPU configuration APIs
>   dt-bindings: remoteproc: Add documentation for ZynqMP R5 rproc
>     bindings
>   remoteproc: Add initial zynqmp R5 remoteproc driver
> 
>  .../xilinx,zynqmp-r5-remoteproc.yaml          | 223 +++++
>  drivers/firmware/xilinx/zynqmp.c              |  96 ++
>  drivers/remoteproc/Kconfig                    |   8 +
>  drivers/remoteproc/Makefile                   |   1 +
>  drivers/remoteproc/zynqmp_r5_remoteproc.c     | 897 ++++++++++++++++++
>  include/linux/firmware/xlnx-zynqmp.h          |  66 +-
>  6 files changed, 1290 insertions(+), 1 deletion(-)
>  create mode 100644
> Documentation/devicetree/bindings/remoteproc/xilinx,zynqmp-r5-
> remoteproc.yaml
>  create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c
> 
> --
> 2.17.1


^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [PATCH v25 0/5] Provide basic driver to control Arm R5 co-processor found on Xilinx ZynqMP
  2021-02-04 15:20 ` [PATCH v25 0/5] Provide basic driver to control Arm R5 co-processor found on Xilinx ZynqMP Ben Levinsky
@ 2021-02-04 17:04   ` Mathieu Poirier
  0 siblings, 0 replies; 28+ messages in thread
From: Mathieu Poirier @ 2021-02-04 17:04 UTC (permalink / raw)
  To: Ben Levinsky; +Cc: linux-remoteproc

On Thu, 4 Feb 2021 at 09:03, Ben Levinsky <BLEVINSK@xilinx.com> wrote:
>
> Ping for comments
>

This patchset has been in my queue for a while now but was preempted
by other work.  I should be able to get to it within the next two
weeks.

> > -----Original Message-----
> > From: Ben Levinsky <ben.levinsky@xilinx.com>
> > Sent: Sunday, January 10, 2021 6:03 PM
> > To: mathieu.poirier@linaro.org
> > Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal Simek
> > <michals@xilinx.com>
> > Subject: [PATCH v25 0/5] Provide basic driver to control Arm R5 co-processor
> > found on Xilinx ZynqMP
> >
> > R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
> > remotproc driver, we can boot the R5 sub-system in two different
> > configurations -
> >       * Split
> >       * Lockstep
> >
> > The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
> > Platform Management Unit that handles the R5 configuration, memory access
> > and R5 lifecycle management. The interface to this manager is done in this
> > driver via zynqmp_pm_* function calls.
> >
> > v25:
> > - reword error message for rpu configuration
> > - change char name[15] to 16 chars to have null terminated string
> >   in parse_mem_regions()
> > - reword comments in tcm_mem_alloc
> > - grammar in device tree bindings
> > - call xilinx platform cleanup in parse_tcm_banks() case of failure
> > - check value of mbox_send_message in all calls within this driver
> > - update include/linux/firmware/xlnx-zynqmp.h style to match for enum
> >   pm_node_id
> > - indentation in zynqmp_r5_remoteproc driver
> > - update style of constructing carveouts to match convention in ST
> >   remoteproc drivers. No longer iterate through memory-region property
> >   using of_count_phandle_with_args. Instead use of_phandle_iterator
> > - use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per
> > feedback
> > - rework loop in zynqmp_r5_pm_request_sram as per feedback
> > - fix comment and linebreak in tcm_mem_alloc description
> > - add comments and remove extraneous devm_ioremap_wc call in
> > tcm_mem_alloc
> > - in parse_tcm_banks remove check for bank not being available
> > - rework order of locals and remove else in zynqmp_r5_rproc_kick
> > - document and update logic for zynqmp_r5_parse_fw
> > - add dev_dbg output in event_notified_idr_cb if
> >   rproc_vq_interrupt(rproc, id) == IRQ_NONE
> > - add comment for handle_event_notified mbox_send_message function call
> > - add comment for zynqmp_r5_mb_rx_cb
> > - update zynqmp_r5_setup_mbox to match convention of mbox setup in ST
> >   remoteproc drivers
> > - change return in zynqmp_r5_setup_mbox to use PTR_ERR
> > - add zynqmp_r5_cleanup_mbox
> > - in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as arg.
> > - in zynqmp_r5_probe use return of PTR_ERR and use
> > zynqmp_r5_cleanup_mbox
> > - in zynqmp_r5_remoteproc_probe update use return value of
> > zynqmp_r5_probe as
> >   either zynqmp_r5_rproc* or PTR_ERR
> > - update loop that cleans up cluster and mboxes in
> > zynqmp_r5_remoteproc_probe
> > - update loop that cleans up cluster and mboxes in
> > zynqmp_r5_remoteproc_remove
> >
> > Previous version:
> > https://patchwork.kernel.org/project/linux-remoteproc/list/?series=393387
> >
> >
> > Ben Levinsky (5):
> >   firmware: xilinx: Add ZynqMP firmware ioctl enums for RPU
> >     configuration.
> >   firmware: xilinx: Add shutdown/wakeup APIs
> >   firmware: xilinx: Add RPU configuration APIs
> >   dt-bindings: remoteproc: Add documentation for ZynqMP R5 rproc
> >     bindings
> >   remoteproc: Add initial zynqmp R5 remoteproc driver
> >
> >  .../xilinx,zynqmp-r5-remoteproc.yaml          | 223 +++++
> >  drivers/firmware/xilinx/zynqmp.c              |  96 ++
> >  drivers/remoteproc/Kconfig                    |   8 +
> >  drivers/remoteproc/Makefile                   |   1 +
> >  drivers/remoteproc/zynqmp_r5_remoteproc.c     | 897 ++++++++++++++++++
> >  include/linux/firmware/xlnx-zynqmp.h          |  66 +-
> >  6 files changed, 1290 insertions(+), 1 deletion(-)
> >  create mode 100644
> > Documentation/devicetree/bindings/remoteproc/xilinx,zynqmp-r5-
> > remoteproc.yaml
> >  create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c
> >
> > --
> > 2.17.1
>

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
  2021-01-11  2:02   ` Ben Levinsky
@ 2021-02-15 18:51     ` Mathieu Poirier
  -1 siblings, 0 replies; 28+ messages in thread
From: Mathieu Poirier @ 2021-02-15 18:51 UTC (permalink / raw)
  To: Ben Levinsky
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

Good day,

I have started to review this set - as with previous revisions comments will
come over several days.

On Sun, Jan 10, 2021 at 06:02:50PM -0800, Ben Levinsky wrote:
> R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
> remotproc driver, we can boot the R5 sub-system in two different
> configurations -
> 	* Split
> 	* Lockstep
> 
> The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
> 
> Signed-off-by: Wendy Liang <wendy.liang@xilinx.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> Signed-off-by: Ed Mooring <ed.mooring@xilinx.com>
> Signed-off-by: Jason Wu <j.wu@xilinx.com>
> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> ---
> v25:
> - reword error message for rpu configuration
> - change char name[15] to 16 chars to have null terminated string
>   in parse_mem_regions()
> - reword comments in tcm_mem_alloc
> - call xilinx platform cleanup in parse_tcm_banks() case of failure
> - check value of mbox_send_message in all instances within this driver 
> - indentation in zynqmp_r5_remoteproc driver
> - update style of constructing carveouts to match convention in ST
>   remoteproc drivers. No longer iterate through memory-region property
>   using of_count_phandle_with_args. Instead use of_phandle_iterator
> - use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per feedback
> - rework loop in zynqmp_r5_pm_request_sram as per feedback
> - fix comment and linebreak in tcm_mem_alloc description
> - add comments and remove extraneous devm_ioremap_wc call in tcm_mem_alloc
> - in parse_tcm_banks remove check for bank not being available
> - rework order of locals and remove else in zynqmp_r5_rproc_kick
> - document and update logic for zynqmp_r5_parse_fw
> - add dev_dbg output in event_notified_idr_cb if
>   rproc_vq_interrupt(rproc, id) == IRQ_NONE
> - add comment for handle_event_notified mbox_send_message function call
> - add comment for zynqmp_r5_mb_rx_cb
> - update zynqmp_r5_setup_mbox to match convention of mbox setup in ST
>   remoteproc drivers
> - change return in zynqmp_r5_setup_mbox to use PTR_ERR
> - add zynqmp_r5_cleanup_mbox
> - in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as arg.
> - in zynqmp_r5_probe use return of PTR_ERR and use zynqmp_r5_cleanup_mbox
> - in zynqmp_r5_remoteproc_probe update use return value of zynqmp_r5_probe as
>   either zynqmp_r5_rproc* or PTR_ERR
> - update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_probe
> - update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_remove
> ---
>  drivers/remoteproc/Kconfig                |   8 +
>  drivers/remoteproc/Makefile               |   1 +
>  drivers/remoteproc/zynqmp_r5_remoteproc.c | 898 ++++++++++++++++++++++
>  3 files changed, 907 insertions(+)
>  create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c
> 
> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> index c6659dfea7c7..c2fe54b1d94f 100644
> --- a/drivers/remoteproc/Kconfig
> +++ b/drivers/remoteproc/Kconfig
> @@ -275,6 +275,14 @@ config TI_K3_DSP_REMOTEPROC
>  	  It's safe to say N here if you're not interested in utilizing
>  	  the DSP slave processors.
>  
> +config ZYNQMP_R5_REMOTEPROC
> +	tristate "ZynqMP R5 remoteproc support"
> +	depends on PM && ARCH_ZYNQMP
> +	select RPMSG_VIRTIO
> +	select ZYNQMP_IPI_MBOX
> +	help
> +	  Say y or m here to support ZynqMP R5 remote processors via the remote
> +	  processor framework.
>  endif # REMOTEPROC
>  
>  endmenu
> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> index 3dfa28e6c701..ef1abff654c2 100644
> --- a/drivers/remoteproc/Makefile
> +++ b/drivers/remoteproc/Makefile
> @@ -33,3 +33,4 @@ obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
>  obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
>  obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
>  obj-$(CONFIG_TI_K3_DSP_REMOTEPROC)	+= ti_k3_dsp_remoteproc.o
> +obj-$(CONFIG_ZYNQMP_R5_REMOTEPROC)	+= zynqmp_r5_remoteproc.o
> diff --git a/drivers/remoteproc/zynqmp_r5_remoteproc.c b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> new file mode 100644
> index 000000000000..9e228c9ecd43
> --- /dev/null
> +++ b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> @@ -0,0 +1,898 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Zynq R5 Remote Processor driver
> + *
> + * Based on origin OMAP and Zynq Remote Processor driver
> + *
> + */
> +
> +#include <linux/firmware/xlnx-zynqmp.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/mailbox_client.h>
> +#include <linux/mailbox/zynqmp-ipi-message.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/remoteproc.h>
> +#include <linux/skbuff.h>
> +#include <linux/sysfs.h>
> +
> +#include "remoteproc_internal.h"
> +
> +#define MAX_RPROCS	2 /* Support up to 2 RPU */
> +#define MAX_MEM_PNODES	4 /* Max power nodes for one RPU memory instance */
> +
> +#define BANK_LIST_PROP	"sram"
> +#define DDR_LIST_PROP	"memory-region"
> +
> +/* IPI buffer MAX length */
> +#define IPI_BUF_LEN_MAX	32U
> +/* RX mailbox client buffer max length */
> +#define RX_MBOX_CLIENT_BUF_MAX	(IPI_BUF_LEN_MAX + \
> +				 sizeof(struct zynqmp_ipi_message))
> +
> +/*
> + * Map each Xilinx on-chip SRAM  Bank address to their own respective
> + * pm_node_id.
> + */
> +struct sram_addr_data {
> +	phys_addr_t addr;
> +	enum pm_node_id id;
> +};
> +
> +#define NUM_SRAMS 4U
> +static const struct sram_addr_data zynqmp_banks[NUM_SRAMS] = {
> +	{0xffe00000UL, NODE_TCM_0_A},
> +	{0xffe20000UL, NODE_TCM_0_B},
> +	{0xffe90000UL, NODE_TCM_1_A},
> +	{0xffeb0000UL, NODE_TCM_1_B},
> +};
> +
> +/**
> + * struct zynqmp_r5_rproc - ZynqMP R5 core structure
> + *
> + * @rx_mc_buf: rx mailbox client buffer to save the rx message
> + * @tx_mc: tx mailbox client
> + * @rx_mc: rx mailbox client
> + * @mbox_work: mbox_work for the RPU remoteproc
> + * @tx_mc_skbs: socket buffers for tx mailbox client
> + * @dev: device of RPU instance
> + * @rproc: rproc handle
> + * @tx_chan: tx mailbox channel
> + * @rx_chan: rx mailbox channel
> + * @pnode_id: RPU CPU power domain id
> + * @elem: linked list item
> + */
> +struct zynqmp_r5_rproc {
> +	unsigned char rx_mc_buf[RX_MBOX_CLIENT_BUF_MAX];
> +	struct mbox_client tx_mc;
> +	struct mbox_client rx_mc;
> +	struct work_struct mbox_work;
> +	struct sk_buff_head tx_mc_skbs;
> +	struct device *dev;
> +	struct rproc *rproc;
> +	struct mbox_chan *tx_chan;
> +	struct mbox_chan *rx_chan;
> +	u32 pnode_id;
> +	struct list_head elem;
> +};
> +
> +/*
> + * r5_set_mode - set RPU operation mode
> + * @z_rproc: Remote processor private data
> + * @rpu_mode: mode specified by device tree to configure the RPU to
> + *
> + * set RPU operation mode
> + *
> + * Return: 0 for success, negative value for failure
> + */
> +static int r5_set_mode(struct zynqmp_r5_rproc *z_rproc,
> +		       enum rpu_oper_mode rpu_mode)
> +{
> +	enum rpu_tcm_comb tcm_mode;
> +	enum rpu_oper_mode cur_rpu_mode;
> +	int ret;
> +
> +	ret = zynqmp_pm_get_rpu_mode(z_rproc->pnode_id, &cur_rpu_mode);
> +	if (ret < 0)
> +		return ret;
> +
> +	if (rpu_mode != cur_rpu_mode) {
> +		ret = zynqmp_pm_set_rpu_mode(z_rproc->pnode_id, rpu_mode);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	tcm_mode = (rpu_mode == PM_RPU_MODE_LOCKSTEP) ?
> +		    PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> +	return zynqmp_pm_set_tcm_config(z_rproc->pnode_id, tcm_mode);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_release
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap TCM banks when powering down R5 core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_release(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> +	u32 pnode_id = (u64)mem->priv;
> +
> +	iounmap(mem->va);
> +	return zynqmp_pm_release_node(pnode_id);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_start
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Start R5 Core from designated boot address.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	enum rpu_boot_mem bootmem;
> +
> +	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
> +		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> +
> +	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
> +		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> +
> +	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
> +				      bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_stop
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Power down  R5 Core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +
> +	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
> +				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_alloc
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
> +				     struct rproc_mem_entry *mem)
> +{
> +	void *va;
> +
> +	va = ioremap_wc(mem->dma, mem->len);
> +	if (IS_ERR_OR_NULL(va))
> +		return -ENOMEM;
> +
> +	mem->va = va;
> +
> +	return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_release
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap memory-region carveout
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
> +				       struct rproc_mem_entry *mem)
> +{
> +	iounmap(mem->va);
> +	return 0;
> +}
> +
> +/*
> + * parse_mem_regions
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Construct rproc mem carveouts from carveout provided in
> + * memory-region property
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int parse_mem_regions(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	struct device_node *np = z_rproc->dev->of_node;
> +	struct device *dev = &rproc->dev;
> +	struct of_phandle_iterator it;
> +	struct rproc_mem_entry *mem;
> +	struct reserved_mem *rmem;
> +	int index = 0;
> +
> +	/* Register associated reserved memory regions */
> +	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
> +	while (of_phandle_iterator_next(&it) == 0) {
> +		rmem = of_reserved_mem_lookup(it.node);
> +		if (!rmem) {
> +			dev_err(dev, "unable to acquire %s\n", DDR_LIST_PROP);
> +			return -EINVAL;
> +		}
> +
> +		if (strstr(it.node->name, "vdev0buffer")) {
> +			mem = rproc_of_resm_mem_entry_init(dev, index,
> +							   rmem->size,
> +							   rmem->base,
> +							   "vdev0buffer");
> +		} else {
> +			/*
> +			 * The ensuing carveout is either for vring, which has
> +			 * device tree node name of length 15 characters, or
> +			 * firmware mem, where name is not used by remoteproc
> +			 *  core later on. So default to vring length of 15.
> +			 *
> +			 * Extra char for null-terminated string.
> +			 */
> +			char name[16];
> +
> +			if (strstr(it.node->name, "vdev0vring")) {
> +				int vring_id;
> +
> +				/*
> +				 * Expect "rpuXvdev0vringX" as documented
> +				 * in xilinx remoteproc device tree binding
> +				 */
> +				if (strlen(it.node->name) < 15) {
> +					dev_err(dev, "%pOF is less than 15 chars",
> +						it.node);
> +					return -EINVAL;
> +				}
> +
> +				/*
> +				 * can be 1 of multiple vring IDs per IPC channel
> +				 * e.g. 'vdev0vring0' and 'vdev0vring1'
> +				 *
> +				 * Here get vring ID from 15th char
> +				 */
> +				vring_id = it.node->name[14] - '0';
> +				snprintf(name, sizeof(name), "vdev0vring%d", vring_id);
> +			} else {
> +				strncpy(name, it.node->name, 16);
> +			}
> +
> +			mem = rproc_mem_entry_init(dev, NULL,
> +						   (dma_addr_t)rmem->base,
> +						   rmem->size, rmem->base,
> +						   zynqmp_r5_rproc_mem_alloc,
> +						   zynqmp_r5_rproc_mem_release,
> +						   name);
> +		}
> +
> +		if (!mem)
> +			return -ENOMEM;
> +
> +		rproc_add_carveout(rproc, mem);
> +		index++;
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * zynqmp_r5_pm_request_tcm
> + * @addr: base address of mem provided in R5 core's sram property.
> + *
> + * Given sram base address, determine its corresponding Xilinx
> + * Platform Management ID and then request access to this node
> + * so that it can be power up.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
> +{
> +	int i;
> +
> +	for (i = 0; i < NUM_SRAMS; i++) {
> +		if (zynqmp_banks[i].addr != addr)
> +			continue;
> +
> +		return zynqmp_pm_request_node(zynqmp_banks[i].id,
> +					      ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> +					      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> +	}
> +
> +	return -EINVAL;
> +}
> +
> +/*
> + * tcm_mem_alloc
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to initialize the va and da fields of
> + *
> + * Given TCM bank entry, this callback will set device address for R5
> + * running on TCM and also setup virtual address for TCM bank
> + * remoteproc carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_alloc(struct rproc *rproc,
> +			 struct rproc_mem_entry *mem)
> +{
> +	void *va;
> +	struct device *dev = rproc->dev.parent;
> +
> +	va = ioremap_wc(mem->dma, mem->len);
> +	if (IS_ERR_OR_NULL(va))
> +		return -ENOMEM;
> +
> +	/* Update memory entry va */
> +	mem->va = va;
> +
> +	/*
> +	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> +	 * while on the Linux side they are at 0xffexxxxx.
> +	 *
> +	 * Zero out the high 12 bits of the address. This will give
> +	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> +	 */
> +	mem->da &= 0x000fffff;
> +
> +	/*
> +	 * TCM Banks 1A and 1B still have to be translated.
> +	 *
> +	 * Below handle these two banks' absolute addresses (0xffe90000 and
> +	 * 0xffeb0000) and convert to the expected relative addresses
> +	 * (0x0 and 0x20000).
> +	 */
> +	if (mem->da == 0x90000 || mem->da == 0xB0000)
> +		mem->da -= 0x90000;
> +
> +	/* if translated TCM bank address is not valid report error */
> +	if (mem->da != 0x0 && mem->da != 0x20000) {
> +		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * parse_tcm_banks()
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Given R5 node in remoteproc instance
> + * allocate remoteproc carveout for TCM memory
> + * needed for firmware to be loaded
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int parse_tcm_banks(struct rproc *rproc)
> +{
> +	int i, num_banks;
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	struct device *dev = &rproc->dev;
> +	struct device_node *r5_node = z_rproc->dev->of_node;
> +
> +	/* go through TCM banks for r5 node */
> +	num_banks = of_count_phandle_with_args(r5_node, BANK_LIST_PROP, NULL);
> +	if (num_banks <= 0) {
> +		dev_err(dev, "need to specify TCM banks\n");
> +		return -EINVAL;
> +	}
> +	for (i = 0; i < num_banks; i++) {
> +		struct resource rsc;
> +		resource_size_t size;
> +		struct device_node *dt_node;
> +		struct rproc_mem_entry *mem;
> +		int ret;
> +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> +
> +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> +		if (!dt_node)
> +			return -EINVAL;
> +
> +		ret = of_address_to_resource(dt_node, 0, &rsc);
> +		if (ret < 0) {
> +			of_node_put(dt_node);
> +			return ret;
> +		}
> +		of_node_put(dt_node);
> +
> +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> +		if (ret < 0)
> +			return ret;
> +
> +		/* add carveout */
> +		size = resource_size(&rsc);
> +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> +					   (size_t)size, rsc.start,
> +					   tcm_mem_alloc,
> +					   tcm_mem_release,
> +					   rsc.name);
> +		if (!mem) {
> +			ret = zynqmp_pm_release_node(pnode_id);
> +			if (ret)
> +				dev_warn(dev,
> +					 "fail to release node: %x ret: %x\n",
> +					 pnode_id, ret);
> +			return -ENOMEM;
> +		}
> +
> +		mem->priv = (void *)(u64)pnode_id;
> +		rproc_add_carveout(rproc, mem);
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * zynqmp_r5_parse_fw()
> + * @rproc: single R5 core's corresponding rproc instance
> + * @fw: ptr to firmware to be loaded onto r5 core
> + *
> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> +	int ret;
> +
> +	ret = parse_tcm_banks(rproc);
> +	if (ret)
> +		return ret;
> +
> +	ret = parse_mem_regions(rproc);
> +	if (ret)
> +		return ret;
> +
> +	ret = rproc_elf_load_rsc_table(rproc, fw);
> +	if (ret == -EINVAL) {
> +		/*
> +		 * resource table only required for IPC.
> +		 * if not present, this is not necessarily an error;
> +		 * for example, loading r5 hello world application
> +		 * so simply inform user and keep going.
> +		 */
> +		dev_info(&rproc->dev, "no resource table found.\n");
> +		ret = 0;
> +	}
> +	return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> + * @rproc: r5 core's corresponding rproc structure
> + * @vqid: virtqueue ID
> + */
> +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	struct device *dev = rproc->dev.parent;
> +	struct zynqmp_ipi_message *mb_msg = NULL;
> +	unsigned int skb_len;
> +	struct sk_buff *skb;
> +	int ret;
> +
> +	/* If vqid is negative, do not pass the vqid to
> +	 * mailbox as vqid is supposed to be 0 or positive.
> +	 * It also gives a way to just kick instead but
> +	 * not use the IPI buffer.
> +	 *
> +	 * For now use negative vqid to assume no message will be
> +	 * passed with IPI buffer, but just raise interrupt.
> +	 * This will be faster as it doesn't need to copy the
> +	 * message to the ZynqMP/Versal IPI message buffer.
> +	 */
> +	if (vqid >= 0) {
> +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> +		skb = alloc_skb(skb_len, GFP_KERNEL);
> +		if (!skb)
> +			return;
> +
> +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb, skb_len);
> +		mb_msg->len = sizeof(vqid);
> +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> +
> +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> +	}
> +
> +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> +	if (ret < 0) {
> +		dev_warn(dev, "Failed to kick remote.\n");
> +		if (mb_msg) {
> +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> +			kfree_skb(skb);
> +		}
> +	}
> +}
> +
> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> +	.start		= zynqmp_r5_rproc_start,
> +	.stop		= zynqmp_r5_rproc_stop,
> +	.load		= rproc_elf_load_segments,
> +	.parse_fw	= zynqmp_r5_parse_fw,
> +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> +	.sanity_check	= rproc_elf_sanity_check,
> +	.get_boot_addr	= rproc_elf_get_boot_addr,
> +	.kick		= zynqmp_r5_rproc_kick,
> +};
> +

I have reviewed from the end of the file to here (see comments below) and aside
from minor comments, things look good.  I will review from here to the top
tomorrow.

Thanks,
Mathieu

> +/**
> + * event_notified_idr_cb() - event notified idr callback
> + * @id: idr id
> + * @ptr: pointer to idr private data
> + * @data: data passed to idr_for_each callback
> + *
> + * Pass notification to remoteproc virtio
> + *
> + * Return: 0. having return is to satisfy the idr_for_each() function
> + *          pointer input argument requirement.
> + **/
> +static int event_notified_idr_cb(int id, void *ptr, void *data)
> +{
> +	struct rproc *rproc = data;
> +
> +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> +		dev_dbg(&rproc->dev, "no message was found in idr with id: %d\n", id);
> +	return 0;
> +}
> +
> +/**
> + * handle_event_notified() - remoteproc notification work function
> + * @work: pointer to the work structure
> + *
> + * It checks each registered remoteproc notify IDs.
> + */
> +static void handle_event_notified(struct work_struct *work)
> +{
> +	struct zynqmp_r5_rproc *z_rproc;
> +	struct rproc *rproc;
> +
> +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> +	rproc = z_rproc->rproc;
> +
> +	/*
> +	 * This is to simply raise interrupt as ACK for remote.
> +	 *
> +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> +	 * mapping to remote processor.
> +	 */
> +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> +
> +	/*
> +	 * We only use IPI for interrupt. The firmware side may or may
> +	 * not write the notifyid when it trigger IPI.
> +	 * And thus, we scan through all the registered notifyids.
> +	 */
> +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> +}
> +
> +/**
> + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> + * @cl: mailbox client
> + * @msg: message pointer
> + *
> + * It will schedule the R5 notification work.
> + */
> +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> +{
> +	struct zynqmp_r5_rproc *z_rproc;
> +
> +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> +	/*
> +	 * Notification can be received with message payload.
> +	 * To handle this, check for message contents.
> +	 * If there are contents, copy to ipi message buffer payload location.
> +	 *
> +	 * If message is empty, then that means that was simple raising
> +	 * of interrupt. no payload to process.
> +	 *
> +	 * Note: enqueue work regardless of msg as kick may imply that remote
> +	 * is waiting for ack.
> +	 */
> +	if (msg) {
> +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> +		size_t len;
> +
> +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> +		buf_msg->len = len;
> +		memcpy(buf_msg->data, ipi_msg->data, len);
> +	}
> +	schedule_work(&z_rproc->mbox_work);
> +}
> +
> +/**
> + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> + * @cl: mailbox client
> + * @msg: pointer to the message which has been sent
> + * @r: status of last TX - OK or error
> + *
> + * It will be called by the mailbox framework when the last TX has done.
> + */
> +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg, int r)
> +{
> +	struct zynqmp_r5_rproc *z_rproc;
> +	struct sk_buff *skb;
> +
> +	if (!msg)
> +		return;
> +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> +	kfree_skb(skb);
> +}
> +
> +/**
> + * zynqmp_r5_setup_mbox() - Setup mailboxes
> + *			    this is used for each individual R5 core
> + *
> + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> + * @node: pointer of the device node
> + *
> + * Function to setup mailboxes to talk to RPU.
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> +				struct device_node *node)
> +{
> +	struct mbox_client *mclient;
> +
> +	/* Setup TX mailbox channel client */
> +	mclient			= &z_rproc->tx_mc;
> +	mclient->dev		= z_rproc->dev;
> +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> +	mclient->tx_block	= false;
> +	mclient->knows_txdone	= false;
> +	mclient->rx_callback	= NULL;
> +
> +	/* Setup RX mailbox channel client */
> +	mclient			= &z_rproc->rx_mc;
> +	mclient->dev		= z_rproc->dev;
> +	mclient->tx_done	= NULL;
> +	mclient->tx_block	= false;
> +	mclient->knows_txdone	= false;
> +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> +
> +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> +
> +	/* Request TX and RX channels */
> +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc->tx_mc, "tx");
> +	if (IS_ERR(z_rproc->tx_chan)) {
> +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> +		return PTR_ERR(z_rproc->tx_chan);
> +	}
> +
> +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc->rx_mc, "rx");
> +	if (IS_ERR(z_rproc->rx_chan)) {
> +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> +		return PTR_ERR(z_rproc->rx_chan);
> +	}
> +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> +
> +	return 0;
> +}
> +
> +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc *z_rproc)
> +{
> +	mbox_free_channel(z_rproc->tx_chan);
> +	mbox_free_channel(z_rproc->rx_chan);
> +}
> +
> +/**
> + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> + *		       this is called for each individual R5 core to
> + *		       set up mailbox, Xilinx platform manager unique ID,
> + *		       add to rproc core
> + *
> + * @pdev: domain platform device for current R5 core
> + * @node: pointer of the device node for current R5 core
> + * @rpu_mode: mode to configure RPU, split or lockstep
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device *pdev,
> +					       struct device_node *node,
> +					       enum rpu_oper_mode rpu_mode)
> +{
> +	int ret;
> +	struct device *dev = &pdev->dev;
> +	struct rproc *rproc_ptr;
> +	struct zynqmp_r5_rproc *z_rproc;
> +
> +	/* Allocate remoteproc instance */
> +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> +				     NULL, sizeof(struct zynqmp_r5_rproc));
> +	if (!rproc_ptr) {
> +		ret = -ENOMEM;
> +		goto error;
> +	}
> +
> +	rproc_ptr->auto_boot = false;
> +	z_rproc = rproc_ptr->priv;
> +	z_rproc->rproc = rproc_ptr;
> +	z_rproc->dev = dev;

Here z_rproc->dev is the same as z_rproc->rproc->dev->parent and as far as I can
tell z_rproc->rproc is available where z_rproc->dev is used.  As such
zynqmpq_r5_rproc::dev can likely be removed.

> +
> +	/* Set up DMA mask */
> +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> +	if (ret)
> +		goto error;
> +
> +	/* Get R5 power domain node */
> +	ret = of_property_read_u32(node, "power-domain", &z_rproc->pnode_id);
> +	if (ret)
> +		goto error;
> +
> +	ret = r5_set_mode(z_rproc, rpu_mode);
> +	if (ret)
> +		goto error;
> +
> +	if (of_property_read_bool(node, "mboxes")) {
> +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> +		if (ret)
> +			goto error;
> +	}
> +
> +	/* Add R5 remoteproc */
> +	ret = devm_rproc_add(dev, rproc_ptr);
> +	if (ret) {
> +		zynqmp_r5_cleanup_mbox(z_rproc);
> +		goto error;
> +	}
> +
> +	return z_rproc;
> +error:
> +	return ERR_PTR(ret);
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_probe()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * called when driver is probed, for each R5 core specified in DT,
> + * setup as needed to do remoteproc-related operations
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> +{
> +	int ret, core_count;
> +	struct device *dev = &pdev->dev;
> +	struct device_node *nc;
> +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> +	struct list_head *cluster; /* list to track each core's rproc */
> +	struct zynqmp_r5_rproc *z_rproc;
> +	struct platform_device *child_pdev;
> +	struct list_head *pos;
> +
> +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode", &rpu_mode);
> +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> +			ret, rpu_mode);
> +		return ret;
> +	}
> +
> +	dev_dbg(dev, "RPU configuration: %s\n",
> +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" : "split");
> +
> +	/*
> +	 * if 2 RPUs provided but one is lockstep, then we have an
> +	 * invalid configuration.
> +	 */
> +
> +	core_count = of_get_available_child_count(dev->of_node);
> +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> +	    core_count > MAX_RPROCS)
> +		return -EINVAL;
> +
> +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> +	if (!cluster)
> +		return -ENOMEM;
> +	INIT_LIST_HEAD(cluster);
> +
> +	ret = devm_of_platform_populate(dev);
> +	if (ret) {
> +		dev_err(dev, "devm_of_platform_populate failed, ret = %d\n",
> +			ret);

You can move the above "ret);" to the above line to avoid breaking it.  The 80
character per line rule was relaxed for cases like these.

> +		return ret;
> +	}
> +
> +	/* probe each individual r5 core's remoteproc-related info */
> +	for_each_available_child_of_node(dev->of_node, nc) {
> +		child_pdev = of_find_device_by_node(nc);
> +		if (!child_pdev) {
> +			dev_err(dev, "could not get R5 core platform device\n");
> +			ret = -ENODEV;
> +			goto out;
> +		}
> +
> +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> +			ret ? "Failed" : "Able",
> +			nc);

Staking - "nc);" could easily be on the previous line.

> +		if (IS_ERR(z_rproc)) {
> +			ret = PTR_ERR(z_rproc);
> +			goto out;
> +		}
> +		list_add_tail(&z_rproc->elem, cluster);
> +	}
> +	/* wire in so each core can be cleaned up at driver remove */
> +	platform_set_drvdata(pdev, cluster);
> +	return 0;
> +out:
> +	/*
> +	 * undo core0 upon any failures on core1 in split-mode
> +	 *
> +	 * in zynqmp_r5_probe z_rproc is set to null
> +	 * and ret to non-zero value if error

Not sure this comment applies anymore.

> +	 */
> +	list_for_each(pos, cluster) {
> +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> +		zynqmp_r5_cleanup_mbox(z_rproc);
> +	}
> +	return ret;
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_remove()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * When the driver is unloaded, clean up the mailboxes for each
> + * remoteproc that was initially probed.
> + */
> +static int zynqmp_r5_remoteproc_remove(struct platform_device *pdev)
> +{
> +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> +						 platform_get_drvdata(pdev);
> +	struct zynqmp_r5_rproc *z_rproc = NULL;
> +
> +	list_for_each_safe(pos, temp, cluster) {
> +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> +		zynqmp_r5_cleanup_mbox(z_rproc);
> +		list_del(pos);

In zynqmp_r5_remoteproc_probe() the element is not removed.  It doesn't make a
difference since the list isn't used for anything after this.  Regardless, pick
a heuristic and stick with it.

> +	}
> +	return 0;
> +}
> +
> +/* Match table for OF platform binding */
> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> +	{ /* end of list */ },
> +};
> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> +
> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> +	.probe = zynqmp_r5_remoteproc_probe,
> +	.remove = zynqmp_r5_remoteproc_remove,
> +	.driver = {
> +		.name = "zynqmp_r5_remoteproc",
> +		.of_match_table = zynqmp_r5_remoteproc_match,
> +	},
> +};
> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> +
> +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> +MODULE_LICENSE("GPL v2");
> -- 
> 2.17.1
> 

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
@ 2021-02-15 18:51     ` Mathieu Poirier
  0 siblings, 0 replies; 28+ messages in thread
From: Mathieu Poirier @ 2021-02-15 18:51 UTC (permalink / raw)
  To: Ben Levinsky
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

Good day,

I have started to review this set - as with previous revisions comments will
come over several days.

On Sun, Jan 10, 2021 at 06:02:50PM -0800, Ben Levinsky wrote:
> R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
> remotproc driver, we can boot the R5 sub-system in two different
> configurations -
> 	* Split
> 	* Lockstep
> 
> The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
> 
> Signed-off-by: Wendy Liang <wendy.liang@xilinx.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> Signed-off-by: Ed Mooring <ed.mooring@xilinx.com>
> Signed-off-by: Jason Wu <j.wu@xilinx.com>
> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> ---
> v25:
> - reword error message for rpu configuration
> - change char name[15] to 16 chars to have null terminated string
>   in parse_mem_regions()
> - reword comments in tcm_mem_alloc
> - call xilinx platform cleanup in parse_tcm_banks() case of failure
> - check value of mbox_send_message in all instances within this driver 
> - indentation in zynqmp_r5_remoteproc driver
> - update style of constructing carveouts to match convention in ST
>   remoteproc drivers. No longer iterate through memory-region property
>   using of_count_phandle_with_args. Instead use of_phandle_iterator
> - use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per feedback
> - rework loop in zynqmp_r5_pm_request_sram as per feedback
> - fix comment and linebreak in tcm_mem_alloc description
> - add comments and remove extraneous devm_ioremap_wc call in tcm_mem_alloc
> - in parse_tcm_banks remove check for bank not being available
> - rework order of locals and remove else in zynqmp_r5_rproc_kick
> - document and update logic for zynqmp_r5_parse_fw
> - add dev_dbg output in event_notified_idr_cb if
>   rproc_vq_interrupt(rproc, id) == IRQ_NONE
> - add comment for handle_event_notified mbox_send_message function call
> - add comment for zynqmp_r5_mb_rx_cb
> - update zynqmp_r5_setup_mbox to match convention of mbox setup in ST
>   remoteproc drivers
> - change return in zynqmp_r5_setup_mbox to use PTR_ERR
> - add zynqmp_r5_cleanup_mbox
> - in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as arg.
> - in zynqmp_r5_probe use return of PTR_ERR and use zynqmp_r5_cleanup_mbox
> - in zynqmp_r5_remoteproc_probe update use return value of zynqmp_r5_probe as
>   either zynqmp_r5_rproc* or PTR_ERR
> - update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_probe
> - update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_remove
> ---
>  drivers/remoteproc/Kconfig                |   8 +
>  drivers/remoteproc/Makefile               |   1 +
>  drivers/remoteproc/zynqmp_r5_remoteproc.c | 898 ++++++++++++++++++++++
>  3 files changed, 907 insertions(+)
>  create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c
> 
> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> index c6659dfea7c7..c2fe54b1d94f 100644
> --- a/drivers/remoteproc/Kconfig
> +++ b/drivers/remoteproc/Kconfig
> @@ -275,6 +275,14 @@ config TI_K3_DSP_REMOTEPROC
>  	  It's safe to say N here if you're not interested in utilizing
>  	  the DSP slave processors.
>  
> +config ZYNQMP_R5_REMOTEPROC
> +	tristate "ZynqMP R5 remoteproc support"
> +	depends on PM && ARCH_ZYNQMP
> +	select RPMSG_VIRTIO
> +	select ZYNQMP_IPI_MBOX
> +	help
> +	  Say y or m here to support ZynqMP R5 remote processors via the remote
> +	  processor framework.
>  endif # REMOTEPROC
>  
>  endmenu
> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> index 3dfa28e6c701..ef1abff654c2 100644
> --- a/drivers/remoteproc/Makefile
> +++ b/drivers/remoteproc/Makefile
> @@ -33,3 +33,4 @@ obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
>  obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
>  obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
>  obj-$(CONFIG_TI_K3_DSP_REMOTEPROC)	+= ti_k3_dsp_remoteproc.o
> +obj-$(CONFIG_ZYNQMP_R5_REMOTEPROC)	+= zynqmp_r5_remoteproc.o
> diff --git a/drivers/remoteproc/zynqmp_r5_remoteproc.c b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> new file mode 100644
> index 000000000000..9e228c9ecd43
> --- /dev/null
> +++ b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> @@ -0,0 +1,898 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Zynq R5 Remote Processor driver
> + *
> + * Based on origin OMAP and Zynq Remote Processor driver
> + *
> + */
> +
> +#include <linux/firmware/xlnx-zynqmp.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/mailbox_client.h>
> +#include <linux/mailbox/zynqmp-ipi-message.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/remoteproc.h>
> +#include <linux/skbuff.h>
> +#include <linux/sysfs.h>
> +
> +#include "remoteproc_internal.h"
> +
> +#define MAX_RPROCS	2 /* Support up to 2 RPU */
> +#define MAX_MEM_PNODES	4 /* Max power nodes for one RPU memory instance */
> +
> +#define BANK_LIST_PROP	"sram"
> +#define DDR_LIST_PROP	"memory-region"
> +
> +/* IPI buffer MAX length */
> +#define IPI_BUF_LEN_MAX	32U
> +/* RX mailbox client buffer max length */
> +#define RX_MBOX_CLIENT_BUF_MAX	(IPI_BUF_LEN_MAX + \
> +				 sizeof(struct zynqmp_ipi_message))
> +
> +/*
> + * Map each Xilinx on-chip SRAM  Bank address to their own respective
> + * pm_node_id.
> + */
> +struct sram_addr_data {
> +	phys_addr_t addr;
> +	enum pm_node_id id;
> +};
> +
> +#define NUM_SRAMS 4U
> +static const struct sram_addr_data zynqmp_banks[NUM_SRAMS] = {
> +	{0xffe00000UL, NODE_TCM_0_A},
> +	{0xffe20000UL, NODE_TCM_0_B},
> +	{0xffe90000UL, NODE_TCM_1_A},
> +	{0xffeb0000UL, NODE_TCM_1_B},
> +};
> +
> +/**
> + * struct zynqmp_r5_rproc - ZynqMP R5 core structure
> + *
> + * @rx_mc_buf: rx mailbox client buffer to save the rx message
> + * @tx_mc: tx mailbox client
> + * @rx_mc: rx mailbox client
> + * @mbox_work: mbox_work for the RPU remoteproc
> + * @tx_mc_skbs: socket buffers for tx mailbox client
> + * @dev: device of RPU instance
> + * @rproc: rproc handle
> + * @tx_chan: tx mailbox channel
> + * @rx_chan: rx mailbox channel
> + * @pnode_id: RPU CPU power domain id
> + * @elem: linked list item
> + */
> +struct zynqmp_r5_rproc {
> +	unsigned char rx_mc_buf[RX_MBOX_CLIENT_BUF_MAX];
> +	struct mbox_client tx_mc;
> +	struct mbox_client rx_mc;
> +	struct work_struct mbox_work;
> +	struct sk_buff_head tx_mc_skbs;
> +	struct device *dev;
> +	struct rproc *rproc;
> +	struct mbox_chan *tx_chan;
> +	struct mbox_chan *rx_chan;
> +	u32 pnode_id;
> +	struct list_head elem;
> +};
> +
> +/*
> + * r5_set_mode - set RPU operation mode
> + * @z_rproc: Remote processor private data
> + * @rpu_mode: mode specified by device tree to configure the RPU to
> + *
> + * set RPU operation mode
> + *
> + * Return: 0 for success, negative value for failure
> + */
> +static int r5_set_mode(struct zynqmp_r5_rproc *z_rproc,
> +		       enum rpu_oper_mode rpu_mode)
> +{
> +	enum rpu_tcm_comb tcm_mode;
> +	enum rpu_oper_mode cur_rpu_mode;
> +	int ret;
> +
> +	ret = zynqmp_pm_get_rpu_mode(z_rproc->pnode_id, &cur_rpu_mode);
> +	if (ret < 0)
> +		return ret;
> +
> +	if (rpu_mode != cur_rpu_mode) {
> +		ret = zynqmp_pm_set_rpu_mode(z_rproc->pnode_id, rpu_mode);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	tcm_mode = (rpu_mode == PM_RPU_MODE_LOCKSTEP) ?
> +		    PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> +	return zynqmp_pm_set_tcm_config(z_rproc->pnode_id, tcm_mode);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_release
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap TCM banks when powering down R5 core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_release(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> +	u32 pnode_id = (u64)mem->priv;
> +
> +	iounmap(mem->va);
> +	return zynqmp_pm_release_node(pnode_id);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_start
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Start R5 Core from designated boot address.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	enum rpu_boot_mem bootmem;
> +
> +	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
> +		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> +
> +	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
> +		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> +
> +	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
> +				      bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_stop
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Power down  R5 Core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +
> +	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
> +				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_alloc
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
> +				     struct rproc_mem_entry *mem)
> +{
> +	void *va;
> +
> +	va = ioremap_wc(mem->dma, mem->len);
> +	if (IS_ERR_OR_NULL(va))
> +		return -ENOMEM;
> +
> +	mem->va = va;
> +
> +	return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_release
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap memory-region carveout
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
> +				       struct rproc_mem_entry *mem)
> +{
> +	iounmap(mem->va);
> +	return 0;
> +}
> +
> +/*
> + * parse_mem_regions
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Construct rproc mem carveouts from carveout provided in
> + * memory-region property
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int parse_mem_regions(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	struct device_node *np = z_rproc->dev->of_node;
> +	struct device *dev = &rproc->dev;
> +	struct of_phandle_iterator it;
> +	struct rproc_mem_entry *mem;
> +	struct reserved_mem *rmem;
> +	int index = 0;
> +
> +	/* Register associated reserved memory regions */
> +	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
> +	while (of_phandle_iterator_next(&it) == 0) {
> +		rmem = of_reserved_mem_lookup(it.node);
> +		if (!rmem) {
> +			dev_err(dev, "unable to acquire %s\n", DDR_LIST_PROP);
> +			return -EINVAL;
> +		}
> +
> +		if (strstr(it.node->name, "vdev0buffer")) {
> +			mem = rproc_of_resm_mem_entry_init(dev, index,
> +							   rmem->size,
> +							   rmem->base,
> +							   "vdev0buffer");
> +		} else {
> +			/*
> +			 * The ensuing carveout is either for vring, which has
> +			 * device tree node name of length 15 characters, or
> +			 * firmware mem, where name is not used by remoteproc
> +			 *  core later on. So default to vring length of 15.
> +			 *
> +			 * Extra char for null-terminated string.
> +			 */
> +			char name[16];
> +
> +			if (strstr(it.node->name, "vdev0vring")) {
> +				int vring_id;
> +
> +				/*
> +				 * Expect "rpuXvdev0vringX" as documented
> +				 * in xilinx remoteproc device tree binding
> +				 */
> +				if (strlen(it.node->name) < 15) {
> +					dev_err(dev, "%pOF is less than 15 chars",
> +						it.node);
> +					return -EINVAL;
> +				}
> +
> +				/*
> +				 * can be 1 of multiple vring IDs per IPC channel
> +				 * e.g. 'vdev0vring0' and 'vdev0vring1'
> +				 *
> +				 * Here get vring ID from 15th char
> +				 */
> +				vring_id = it.node->name[14] - '0';
> +				snprintf(name, sizeof(name), "vdev0vring%d", vring_id);
> +			} else {
> +				strncpy(name, it.node->name, 16);
> +			}
> +
> +			mem = rproc_mem_entry_init(dev, NULL,
> +						   (dma_addr_t)rmem->base,
> +						   rmem->size, rmem->base,
> +						   zynqmp_r5_rproc_mem_alloc,
> +						   zynqmp_r5_rproc_mem_release,
> +						   name);
> +		}
> +
> +		if (!mem)
> +			return -ENOMEM;
> +
> +		rproc_add_carveout(rproc, mem);
> +		index++;
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * zynqmp_r5_pm_request_tcm
> + * @addr: base address of mem provided in R5 core's sram property.
> + *
> + * Given sram base address, determine its corresponding Xilinx
> + * Platform Management ID and then request access to this node
> + * so that it can be power up.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
> +{
> +	int i;
> +
> +	for (i = 0; i < NUM_SRAMS; i++) {
> +		if (zynqmp_banks[i].addr != addr)
> +			continue;
> +
> +		return zynqmp_pm_request_node(zynqmp_banks[i].id,
> +					      ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> +					      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> +	}
> +
> +	return -EINVAL;
> +}
> +
> +/*
> + * tcm_mem_alloc
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to initialize the va and da fields of
> + *
> + * Given TCM bank entry, this callback will set device address for R5
> + * running on TCM and also setup virtual address for TCM bank
> + * remoteproc carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_alloc(struct rproc *rproc,
> +			 struct rproc_mem_entry *mem)
> +{
> +	void *va;
> +	struct device *dev = rproc->dev.parent;
> +
> +	va = ioremap_wc(mem->dma, mem->len);
> +	if (IS_ERR_OR_NULL(va))
> +		return -ENOMEM;
> +
> +	/* Update memory entry va */
> +	mem->va = va;
> +
> +	/*
> +	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> +	 * while on the Linux side they are at 0xffexxxxx.
> +	 *
> +	 * Zero out the high 12 bits of the address. This will give
> +	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> +	 */
> +	mem->da &= 0x000fffff;
> +
> +	/*
> +	 * TCM Banks 1A and 1B still have to be translated.
> +	 *
> +	 * Below handle these two banks' absolute addresses (0xffe90000 and
> +	 * 0xffeb0000) and convert to the expected relative addresses
> +	 * (0x0 and 0x20000).
> +	 */
> +	if (mem->da == 0x90000 || mem->da == 0xB0000)
> +		mem->da -= 0x90000;
> +
> +	/* if translated TCM bank address is not valid report error */
> +	if (mem->da != 0x0 && mem->da != 0x20000) {
> +		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * parse_tcm_banks()
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Given R5 node in remoteproc instance
> + * allocate remoteproc carveout for TCM memory
> + * needed for firmware to be loaded
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int parse_tcm_banks(struct rproc *rproc)
> +{
> +	int i, num_banks;
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	struct device *dev = &rproc->dev;
> +	struct device_node *r5_node = z_rproc->dev->of_node;
> +
> +	/* go through TCM banks for r5 node */
> +	num_banks = of_count_phandle_with_args(r5_node, BANK_LIST_PROP, NULL);
> +	if (num_banks <= 0) {
> +		dev_err(dev, "need to specify TCM banks\n");
> +		return -EINVAL;
> +	}
> +	for (i = 0; i < num_banks; i++) {
> +		struct resource rsc;
> +		resource_size_t size;
> +		struct device_node *dt_node;
> +		struct rproc_mem_entry *mem;
> +		int ret;
> +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> +
> +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> +		if (!dt_node)
> +			return -EINVAL;
> +
> +		ret = of_address_to_resource(dt_node, 0, &rsc);
> +		if (ret < 0) {
> +			of_node_put(dt_node);
> +			return ret;
> +		}
> +		of_node_put(dt_node);
> +
> +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> +		if (ret < 0)
> +			return ret;
> +
> +		/* add carveout */
> +		size = resource_size(&rsc);
> +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> +					   (size_t)size, rsc.start,
> +					   tcm_mem_alloc,
> +					   tcm_mem_release,
> +					   rsc.name);
> +		if (!mem) {
> +			ret = zynqmp_pm_release_node(pnode_id);
> +			if (ret)
> +				dev_warn(dev,
> +					 "fail to release node: %x ret: %x\n",
> +					 pnode_id, ret);
> +			return -ENOMEM;
> +		}
> +
> +		mem->priv = (void *)(u64)pnode_id;
> +		rproc_add_carveout(rproc, mem);
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * zynqmp_r5_parse_fw()
> + * @rproc: single R5 core's corresponding rproc instance
> + * @fw: ptr to firmware to be loaded onto r5 core
> + *
> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> +	int ret;
> +
> +	ret = parse_tcm_banks(rproc);
> +	if (ret)
> +		return ret;
> +
> +	ret = parse_mem_regions(rproc);
> +	if (ret)
> +		return ret;
> +
> +	ret = rproc_elf_load_rsc_table(rproc, fw);
> +	if (ret == -EINVAL) {
> +		/*
> +		 * resource table only required for IPC.
> +		 * if not present, this is not necessarily an error;
> +		 * for example, loading r5 hello world application
> +		 * so simply inform user and keep going.
> +		 */
> +		dev_info(&rproc->dev, "no resource table found.\n");
> +		ret = 0;
> +	}
> +	return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> + * @rproc: r5 core's corresponding rproc structure
> + * @vqid: virtqueue ID
> + */
> +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	struct device *dev = rproc->dev.parent;
> +	struct zynqmp_ipi_message *mb_msg = NULL;
> +	unsigned int skb_len;
> +	struct sk_buff *skb;
> +	int ret;
> +
> +	/* If vqid is negative, do not pass the vqid to
> +	 * mailbox as vqid is supposed to be 0 or positive.
> +	 * It also gives a way to just kick instead but
> +	 * not use the IPI buffer.
> +	 *
> +	 * For now use negative vqid to assume no message will be
> +	 * passed with IPI buffer, but just raise interrupt.
> +	 * This will be faster as it doesn't need to copy the
> +	 * message to the ZynqMP/Versal IPI message buffer.
> +	 */
> +	if (vqid >= 0) {
> +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> +		skb = alloc_skb(skb_len, GFP_KERNEL);
> +		if (!skb)
> +			return;
> +
> +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb, skb_len);
> +		mb_msg->len = sizeof(vqid);
> +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> +
> +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> +	}
> +
> +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> +	if (ret < 0) {
> +		dev_warn(dev, "Failed to kick remote.\n");
> +		if (mb_msg) {
> +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> +			kfree_skb(skb);
> +		}
> +	}
> +}
> +
> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> +	.start		= zynqmp_r5_rproc_start,
> +	.stop		= zynqmp_r5_rproc_stop,
> +	.load		= rproc_elf_load_segments,
> +	.parse_fw	= zynqmp_r5_parse_fw,
> +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> +	.sanity_check	= rproc_elf_sanity_check,
> +	.get_boot_addr	= rproc_elf_get_boot_addr,
> +	.kick		= zynqmp_r5_rproc_kick,
> +};
> +

I have reviewed from the end of the file to here (see comments below) and aside
from minor comments, things look good.  I will review from here to the top
tomorrow.

Thanks,
Mathieu

> +/**
> + * event_notified_idr_cb() - event notified idr callback
> + * @id: idr id
> + * @ptr: pointer to idr private data
> + * @data: data passed to idr_for_each callback
> + *
> + * Pass notification to remoteproc virtio
> + *
> + * Return: 0. having return is to satisfy the idr_for_each() function
> + *          pointer input argument requirement.
> + **/
> +static int event_notified_idr_cb(int id, void *ptr, void *data)
> +{
> +	struct rproc *rproc = data;
> +
> +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> +		dev_dbg(&rproc->dev, "no message was found in idr with id: %d\n", id);
> +	return 0;
> +}
> +
> +/**
> + * handle_event_notified() - remoteproc notification work function
> + * @work: pointer to the work structure
> + *
> + * It checks each registered remoteproc notify IDs.
> + */
> +static void handle_event_notified(struct work_struct *work)
> +{
> +	struct zynqmp_r5_rproc *z_rproc;
> +	struct rproc *rproc;
> +
> +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> +	rproc = z_rproc->rproc;
> +
> +	/*
> +	 * This is to simply raise interrupt as ACK for remote.
> +	 *
> +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> +	 * mapping to remote processor.
> +	 */
> +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> +
> +	/*
> +	 * We only use IPI for interrupt. The firmware side may or may
> +	 * not write the notifyid when it trigger IPI.
> +	 * And thus, we scan through all the registered notifyids.
> +	 */
> +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> +}
> +
> +/**
> + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> + * @cl: mailbox client
> + * @msg: message pointer
> + *
> + * It will schedule the R5 notification work.
> + */
> +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> +{
> +	struct zynqmp_r5_rproc *z_rproc;
> +
> +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> +	/*
> +	 * Notification can be received with message payload.
> +	 * To handle this, check for message contents.
> +	 * If there are contents, copy to ipi message buffer payload location.
> +	 *
> +	 * If message is empty, then that means that was simple raising
> +	 * of interrupt. no payload to process.
> +	 *
> +	 * Note: enqueue work regardless of msg as kick may imply that remote
> +	 * is waiting for ack.
> +	 */
> +	if (msg) {
> +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> +		size_t len;
> +
> +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> +		buf_msg->len = len;
> +		memcpy(buf_msg->data, ipi_msg->data, len);
> +	}
> +	schedule_work(&z_rproc->mbox_work);
> +}
> +
> +/**
> + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> + * @cl: mailbox client
> + * @msg: pointer to the message which has been sent
> + * @r: status of last TX - OK or error
> + *
> + * It will be called by the mailbox framework when the last TX has done.
> + */
> +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg, int r)
> +{
> +	struct zynqmp_r5_rproc *z_rproc;
> +	struct sk_buff *skb;
> +
> +	if (!msg)
> +		return;
> +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> +	kfree_skb(skb);
> +}
> +
> +/**
> + * zynqmp_r5_setup_mbox() - Setup mailboxes
> + *			    this is used for each individual R5 core
> + *
> + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> + * @node: pointer of the device node
> + *
> + * Function to setup mailboxes to talk to RPU.
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> +				struct device_node *node)
> +{
> +	struct mbox_client *mclient;
> +
> +	/* Setup TX mailbox channel client */
> +	mclient			= &z_rproc->tx_mc;
> +	mclient->dev		= z_rproc->dev;
> +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> +	mclient->tx_block	= false;
> +	mclient->knows_txdone	= false;
> +	mclient->rx_callback	= NULL;
> +
> +	/* Setup RX mailbox channel client */
> +	mclient			= &z_rproc->rx_mc;
> +	mclient->dev		= z_rproc->dev;
> +	mclient->tx_done	= NULL;
> +	mclient->tx_block	= false;
> +	mclient->knows_txdone	= false;
> +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> +
> +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> +
> +	/* Request TX and RX channels */
> +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc->tx_mc, "tx");
> +	if (IS_ERR(z_rproc->tx_chan)) {
> +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> +		return PTR_ERR(z_rproc->tx_chan);
> +	}
> +
> +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc->rx_mc, "rx");
> +	if (IS_ERR(z_rproc->rx_chan)) {
> +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> +		return PTR_ERR(z_rproc->rx_chan);
> +	}
> +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> +
> +	return 0;
> +}
> +
> +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc *z_rproc)
> +{
> +	mbox_free_channel(z_rproc->tx_chan);
> +	mbox_free_channel(z_rproc->rx_chan);
> +}
> +
> +/**
> + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> + *		       this is called for each individual R5 core to
> + *		       set up mailbox, Xilinx platform manager unique ID,
> + *		       add to rproc core
> + *
> + * @pdev: domain platform device for current R5 core
> + * @node: pointer of the device node for current R5 core
> + * @rpu_mode: mode to configure RPU, split or lockstep
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device *pdev,
> +					       struct device_node *node,
> +					       enum rpu_oper_mode rpu_mode)
> +{
> +	int ret;
> +	struct device *dev = &pdev->dev;
> +	struct rproc *rproc_ptr;
> +	struct zynqmp_r5_rproc *z_rproc;
> +
> +	/* Allocate remoteproc instance */
> +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> +				     NULL, sizeof(struct zynqmp_r5_rproc));
> +	if (!rproc_ptr) {
> +		ret = -ENOMEM;
> +		goto error;
> +	}
> +
> +	rproc_ptr->auto_boot = false;
> +	z_rproc = rproc_ptr->priv;
> +	z_rproc->rproc = rproc_ptr;
> +	z_rproc->dev = dev;

Here z_rproc->dev is the same as z_rproc->rproc->dev->parent and as far as I can
tell z_rproc->rproc is available where z_rproc->dev is used.  As such
zynqmpq_r5_rproc::dev can likely be removed.

> +
> +	/* Set up DMA mask */
> +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> +	if (ret)
> +		goto error;
> +
> +	/* Get R5 power domain node */
> +	ret = of_property_read_u32(node, "power-domain", &z_rproc->pnode_id);
> +	if (ret)
> +		goto error;
> +
> +	ret = r5_set_mode(z_rproc, rpu_mode);
> +	if (ret)
> +		goto error;
> +
> +	if (of_property_read_bool(node, "mboxes")) {
> +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> +		if (ret)
> +			goto error;
> +	}
> +
> +	/* Add R5 remoteproc */
> +	ret = devm_rproc_add(dev, rproc_ptr);
> +	if (ret) {
> +		zynqmp_r5_cleanup_mbox(z_rproc);
> +		goto error;
> +	}
> +
> +	return z_rproc;
> +error:
> +	return ERR_PTR(ret);
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_probe()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * called when driver is probed, for each R5 core specified in DT,
> + * setup as needed to do remoteproc-related operations
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> +{
> +	int ret, core_count;
> +	struct device *dev = &pdev->dev;
> +	struct device_node *nc;
> +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> +	struct list_head *cluster; /* list to track each core's rproc */
> +	struct zynqmp_r5_rproc *z_rproc;
> +	struct platform_device *child_pdev;
> +	struct list_head *pos;
> +
> +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode", &rpu_mode);
> +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> +			ret, rpu_mode);
> +		return ret;
> +	}
> +
> +	dev_dbg(dev, "RPU configuration: %s\n",
> +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" : "split");
> +
> +	/*
> +	 * if 2 RPUs provided but one is lockstep, then we have an
> +	 * invalid configuration.
> +	 */
> +
> +	core_count = of_get_available_child_count(dev->of_node);
> +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> +	    core_count > MAX_RPROCS)
> +		return -EINVAL;
> +
> +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> +	if (!cluster)
> +		return -ENOMEM;
> +	INIT_LIST_HEAD(cluster);
> +
> +	ret = devm_of_platform_populate(dev);
> +	if (ret) {
> +		dev_err(dev, "devm_of_platform_populate failed, ret = %d\n",
> +			ret);

You can move the above "ret);" to the above line to avoid breaking it.  The 80
character per line rule was relaxed for cases like these.

> +		return ret;
> +	}
> +
> +	/* probe each individual r5 core's remoteproc-related info */
> +	for_each_available_child_of_node(dev->of_node, nc) {
> +		child_pdev = of_find_device_by_node(nc);
> +		if (!child_pdev) {
> +			dev_err(dev, "could not get R5 core platform device\n");
> +			ret = -ENODEV;
> +			goto out;
> +		}
> +
> +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> +			ret ? "Failed" : "Able",
> +			nc);

Staking - "nc);" could easily be on the previous line.

> +		if (IS_ERR(z_rproc)) {
> +			ret = PTR_ERR(z_rproc);
> +			goto out;
> +		}
> +		list_add_tail(&z_rproc->elem, cluster);
> +	}
> +	/* wire in so each core can be cleaned up at driver remove */
> +	platform_set_drvdata(pdev, cluster);
> +	return 0;
> +out:
> +	/*
> +	 * undo core0 upon any failures on core1 in split-mode
> +	 *
> +	 * in zynqmp_r5_probe z_rproc is set to null
> +	 * and ret to non-zero value if error

Not sure this comment applies anymore.

> +	 */
> +	list_for_each(pos, cluster) {
> +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> +		zynqmp_r5_cleanup_mbox(z_rproc);
> +	}
> +	return ret;
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_remove()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * When the driver is unloaded, clean up the mailboxes for each
> + * remoteproc that was initially probed.
> + */
> +static int zynqmp_r5_remoteproc_remove(struct platform_device *pdev)
> +{
> +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> +						 platform_get_drvdata(pdev);
> +	struct zynqmp_r5_rproc *z_rproc = NULL;
> +
> +	list_for_each_safe(pos, temp, cluster) {
> +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> +		zynqmp_r5_cleanup_mbox(z_rproc);
> +		list_del(pos);

In zynqmp_r5_remoteproc_probe() the element is not removed.  It doesn't make a
difference since the list isn't used for anything after this.  Regardless, pick
a heuristic and stick with it.

> +	}
> +	return 0;
> +}
> +
> +/* Match table for OF platform binding */
> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> +	{ /* end of list */ },
> +};
> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> +
> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> +	.probe = zynqmp_r5_remoteproc_probe,
> +	.remove = zynqmp_r5_remoteproc_remove,
> +	.driver = {
> +		.name = "zynqmp_r5_remoteproc",
> +		.of_match_table = zynqmp_r5_remoteproc_match,
> +	},
> +};
> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> +
> +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> +MODULE_LICENSE("GPL v2");
> -- 
> 2.17.1
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
  2021-01-11  2:02   ` Ben Levinsky
@ 2021-02-16 18:48     ` Mathieu Poirier
  -1 siblings, 0 replies; 28+ messages in thread
From: Mathieu Poirier @ 2021-02-16 18:48 UTC (permalink / raw)
  To: Ben Levinsky
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

On Sun, Jan 10, 2021 at 06:02:50PM -0800, Ben Levinsky wrote:
> R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
> remotproc driver, we can boot the R5 sub-system in two different
> configurations -
> 	* Split
> 	* Lockstep
> 
> The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
> 
> Signed-off-by: Wendy Liang <wendy.liang@xilinx.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> Signed-off-by: Ed Mooring <ed.mooring@xilinx.com>
> Signed-off-by: Jason Wu <j.wu@xilinx.com>
> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> ---
> v25:
> - reword error message for rpu configuration
> - change char name[15] to 16 chars to have null terminated string
>   in parse_mem_regions()
> - reword comments in tcm_mem_alloc
> - call xilinx platform cleanup in parse_tcm_banks() case of failure
> - check value of mbox_send_message in all instances within this driver 
> - indentation in zynqmp_r5_remoteproc driver
> - update style of constructing carveouts to match convention in ST
>   remoteproc drivers. No longer iterate through memory-region property
>   using of_count_phandle_with_args. Instead use of_phandle_iterator
> - use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per feedback
> - rework loop in zynqmp_r5_pm_request_sram as per feedback
> - fix comment and linebreak in tcm_mem_alloc description
> - add comments and remove extraneous devm_ioremap_wc call in tcm_mem_alloc
> - in parse_tcm_banks remove check for bank not being available
> - rework order of locals and remove else in zynqmp_r5_rproc_kick
> - document and update logic for zynqmp_r5_parse_fw
> - add dev_dbg output in event_notified_idr_cb if
>   rproc_vq_interrupt(rproc, id) == IRQ_NONE
> - add comment for handle_event_notified mbox_send_message function call
> - add comment for zynqmp_r5_mb_rx_cb
> - update zynqmp_r5_setup_mbox to match convention of mbox setup in ST
>   remoteproc drivers
> - change return in zynqmp_r5_setup_mbox to use PTR_ERR
> - add zynqmp_r5_cleanup_mbox
> - in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as arg.
> - in zynqmp_r5_probe use return of PTR_ERR and use zynqmp_r5_cleanup_mbox
> - in zynqmp_r5_remoteproc_probe update use return value of zynqmp_r5_probe as
>   either zynqmp_r5_rproc* or PTR_ERR
> - update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_probe
> - update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_remove
> ---
>  drivers/remoteproc/Kconfig                |   8 +
>  drivers/remoteproc/Makefile               |   1 +
>  drivers/remoteproc/zynqmp_r5_remoteproc.c | 898 ++++++++++++++++++++++
>  3 files changed, 907 insertions(+)
>  create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c
> 
> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> index c6659dfea7c7..c2fe54b1d94f 100644
> --- a/drivers/remoteproc/Kconfig
> +++ b/drivers/remoteproc/Kconfig
> @@ -275,6 +275,14 @@ config TI_K3_DSP_REMOTEPROC
>  	  It's safe to say N here if you're not interested in utilizing
>  	  the DSP slave processors.
>  
> +config ZYNQMP_R5_REMOTEPROC
> +	tristate "ZynqMP R5 remoteproc support"
> +	depends on PM && ARCH_ZYNQMP
> +	select RPMSG_VIRTIO
> +	select ZYNQMP_IPI_MBOX
> +	help
> +	  Say y or m here to support ZynqMP R5 remote processors via the remote
> +	  processor framework.
>  endif # REMOTEPROC
>  
>  endmenu
> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> index 3dfa28e6c701..ef1abff654c2 100644
> --- a/drivers/remoteproc/Makefile
> +++ b/drivers/remoteproc/Makefile
> @@ -33,3 +33,4 @@ obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
>  obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
>  obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
>  obj-$(CONFIG_TI_K3_DSP_REMOTEPROC)	+= ti_k3_dsp_remoteproc.o
> +obj-$(CONFIG_ZYNQMP_R5_REMOTEPROC)	+= zynqmp_r5_remoteproc.o
> diff --git a/drivers/remoteproc/zynqmp_r5_remoteproc.c b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> new file mode 100644
> index 000000000000..9e228c9ecd43
> --- /dev/null
> +++ b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> @@ -0,0 +1,898 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Zynq R5 Remote Processor driver
> + *
> + * Based on origin OMAP and Zynq Remote Processor driver
> + *
> + */
> +
> +#include <linux/firmware/xlnx-zynqmp.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/mailbox_client.h>
> +#include <linux/mailbox/zynqmp-ipi-message.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/remoteproc.h>
> +#include <linux/skbuff.h>
> +#include <linux/sysfs.h>
> +
> +#include "remoteproc_internal.h"
> +
> +#define MAX_RPROCS	2 /* Support up to 2 RPU */
> +#define MAX_MEM_PNODES	4 /* Max power nodes for one RPU memory instance */
> +
> +#define BANK_LIST_PROP	"sram"
> +#define DDR_LIST_PROP	"memory-region"
> +
> +/* IPI buffer MAX length */
> +#define IPI_BUF_LEN_MAX	32U
> +/* RX mailbox client buffer max length */
> +#define RX_MBOX_CLIENT_BUF_MAX	(IPI_BUF_LEN_MAX + \
> +				 sizeof(struct zynqmp_ipi_message))
> +
> +/*
> + * Map each Xilinx on-chip SRAM  Bank address to their own respective
> + * pm_node_id.
> + */
> +struct sram_addr_data {
> +	phys_addr_t addr;
> +	enum pm_node_id id;
> +};
> +
> +#define NUM_SRAMS 4U
> +static const struct sram_addr_data zynqmp_banks[NUM_SRAMS] = {
> +	{0xffe00000UL, NODE_TCM_0_A},
> +	{0xffe20000UL, NODE_TCM_0_B},
> +	{0xffe90000UL, NODE_TCM_1_A},
> +	{0xffeb0000UL, NODE_TCM_1_B},
> +};
> +
> +/**
> + * struct zynqmp_r5_rproc - ZynqMP R5 core structure
> + *
> + * @rx_mc_buf: rx mailbox client buffer to save the rx message
> + * @tx_mc: tx mailbox client
> + * @rx_mc: rx mailbox client
> + * @mbox_work: mbox_work for the RPU remoteproc
> + * @tx_mc_skbs: socket buffers for tx mailbox client
> + * @dev: device of RPU instance
> + * @rproc: rproc handle
> + * @tx_chan: tx mailbox channel
> + * @rx_chan: rx mailbox channel
> + * @pnode_id: RPU CPU power domain id
> + * @elem: linked list item
> + */
> +struct zynqmp_r5_rproc {
> +	unsigned char rx_mc_buf[RX_MBOX_CLIENT_BUF_MAX];
> +	struct mbox_client tx_mc;
> +	struct mbox_client rx_mc;
> +	struct work_struct mbox_work;
> +	struct sk_buff_head tx_mc_skbs;
> +	struct device *dev;
> +	struct rproc *rproc;
> +	struct mbox_chan *tx_chan;
> +	struct mbox_chan *rx_chan;
> +	u32 pnode_id;
> +	struct list_head elem;
> +};
> +
> +/*
> + * r5_set_mode - set RPU operation mode
> + * @z_rproc: Remote processor private data
> + * @rpu_mode: mode specified by device tree to configure the RPU to
> + *
> + * set RPU operation mode
> + *
> + * Return: 0 for success, negative value for failure
> + */
> +static int r5_set_mode(struct zynqmp_r5_rproc *z_rproc,
> +		       enum rpu_oper_mode rpu_mode)
> +{
> +	enum rpu_tcm_comb tcm_mode;
> +	enum rpu_oper_mode cur_rpu_mode;
> +	int ret;
> +
> +	ret = zynqmp_pm_get_rpu_mode(z_rproc->pnode_id, &cur_rpu_mode);
> +	if (ret < 0)
> +		return ret;
> +
> +	if (rpu_mode != cur_rpu_mode) {
> +		ret = zynqmp_pm_set_rpu_mode(z_rproc->pnode_id, rpu_mode);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	tcm_mode = (rpu_mode == PM_RPU_MODE_LOCKSTEP) ?
> +		    PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> +	return zynqmp_pm_set_tcm_config(z_rproc->pnode_id, tcm_mode);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_release
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap TCM banks when powering down R5 core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_release(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> +	u32 pnode_id = (u64)mem->priv;
> +
> +	iounmap(mem->va);
> +	return zynqmp_pm_release_node(pnode_id);

Is the opposite of that call zynqmp_pm_request_node()?  If so doing the power up
and down of the TCM banks at driver probe() and remove() is highly inefficient.
Consider using rproc_ops::prepare() and rproc_ops::unprepare().

> +}

Please move this just after tcm_mem_alloc()

> +
> +/*
> + * zynqmp_r5_rproc_start
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Start R5 Core from designated boot address.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	enum rpu_boot_mem bootmem;
> +
> +	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
> +		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> +
> +	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
> +		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> +
> +	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
> +				      bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_stop
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Power down  R5 Core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +
> +	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
> +				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_alloc
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
> +				     struct rproc_mem_entry *mem)
> +{
> +	void *va;
> +
> +	va = ioremap_wc(mem->dma, mem->len);
> +	if (IS_ERR_OR_NULL(va))
> +		return -ENOMEM;
> +
> +	mem->va = va;
> +
> +	return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_release
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap memory-region carveout
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
> +				       struct rproc_mem_entry *mem)
> +{
> +	iounmap(mem->va);
> +	return 0;
> +}
> +
> +/*
> + * parse_mem_regions
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Construct rproc mem carveouts from carveout provided in
> + * memory-region property
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int parse_mem_regions(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	struct device_node *np = z_rproc->dev->of_node;
> +	struct device *dev = &rproc->dev;
> +	struct of_phandle_iterator it;
> +	struct rproc_mem_entry *mem;
> +	struct reserved_mem *rmem;
> +	int index = 0;
> +
> +	/* Register associated reserved memory regions */
> +	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
> +	while (of_phandle_iterator_next(&it) == 0) {
> +		rmem = of_reserved_mem_lookup(it.node);
> +		if (!rmem) {
> +			dev_err(dev, "unable to acquire %s\n", DDR_LIST_PROP);
> +			return -EINVAL;
> +		}
> +
> +		if (strstr(it.node->name, "vdev0buffer")) {
> +			mem = rproc_of_resm_mem_entry_init(dev, index,
> +							   rmem->size,
> +							   rmem->base,
> +							   "vdev0buffer");
> +		} else {
> +			/*
> +			 * The ensuing carveout is either for vring, which has
> +			 * device tree node name of length 15 characters, or
> +			 * firmware mem, where name is not used by remoteproc
> +			 *  core later on. So default to vring length of 15.

s/"*  core"/"* core"

> +			 *
> +			 * Extra char for null-terminated string.
> +			 */
> +			char name[16];
> +
> +			if (strstr(it.node->name, "vdev0vring")) {
> +				int vring_id;
> +
> +				/*
> +				 * Expect "rpuXvdev0vringX" as documented
> +				 * in xilinx remoteproc device tree binding
> +				 */
> +				if (strlen(it.node->name) < 15) {
> +					dev_err(dev, "%pOF is less than 15 chars",
> +						it.node);
> +					return -EINVAL;
> +				}
> +
> +				/*
> +				 * can be 1 of multiple vring IDs per IPC channel
> +				 * e.g. 'vdev0vring0' and 'vdev0vring1'
> +				 *
> +				 * Here get vring ID from 15th char
> +				 */
> +				vring_id = it.node->name[14] - '0';
> +				snprintf(name, sizeof(name), "vdev0vring%d", vring_id);

Function strstr() returns a pointer to the beginning of the string being
searched for.  As such you should be able to do something like:

                        name = strstr(it.node->name, "vdev0vring");
                        if (name) {
                                ...

and simply use @name when calling rproc_mem_entry_init().

> +			} else {
> +				strncpy(name, it.node->name, 16);
> +			}
> +
> +			mem = rproc_mem_entry_init(dev, NULL,
> +						   (dma_addr_t)rmem->base,
> +						   rmem->size, rmem->base,
> +						   zynqmp_r5_rproc_mem_alloc,
> +						   zynqmp_r5_rproc_mem_release,
> +						   name);
> +		}
> +
> +		if (!mem)
> +			return -ENOMEM;
> +
> +		rproc_add_carveout(rproc, mem);
> +		index++;
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * zynqmp_r5_pm_request_tcm
> + * @addr: base address of mem provided in R5 core's sram property.
> + *
> + * Given sram base address, determine its corresponding Xilinx
> + * Platform Management ID and then request access to this node
> + * so that it can be power up.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
> +{
> +	int i;
> +
> +	for (i = 0; i < NUM_SRAMS; i++) {
> +		if (zynqmp_banks[i].addr != addr)
> +			continue;
> +
> +		return zynqmp_pm_request_node(zynqmp_banks[i].id,
> +					      ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> +					      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> +	}
> +
> +	return -EINVAL;
> +}
> +
> +/*
> + * tcm_mem_alloc
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to initialize the va and da fields of
> + *
> + * Given TCM bank entry, this callback will set device address for R5
> + * running on TCM and also setup virtual address for TCM bank
> + * remoteproc carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_alloc(struct rproc *rproc,
> +			 struct rproc_mem_entry *mem)
> +{
> +	void *va;
> +	struct device *dev = rproc->dev.parent;
> +
> +	va = ioremap_wc(mem->dma, mem->len);
> +	if (IS_ERR_OR_NULL(va))
> +		return -ENOMEM;
> +
> +	/* Update memory entry va */
> +	mem->va = va;
> +
> +	/*
> +	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> +	 * while on the Linux side they are at 0xffexxxxx.
> +	 *
> +	 * Zero out the high 12 bits of the address. This will give
> +	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> +	 */
> +	mem->da &= 0x000fffff;
> +
> +	/*
> +	 * TCM Banks 1A and 1B still have to be translated.
> +	 *
> +	 * Below handle these two banks' absolute addresses (0xffe90000 and
> +	 * 0xffeb0000) and convert to the expected relative addresses
> +	 * (0x0 and 0x20000).
> +	 */
> +	if (mem->da == 0x90000 || mem->da == 0xB0000)
> +		mem->da -= 0x90000;
> +
> +	/* if translated TCM bank address is not valid report error */
> +	if (mem->da != 0x0 && mem->da != 0x20000) {
> +		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * parse_tcm_banks()
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Given R5 node in remoteproc instance
> + * allocate remoteproc carveout for TCM memory
> + * needed for firmware to be loaded
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int parse_tcm_banks(struct rproc *rproc)
> +{
> +	int i, num_banks;
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	struct device *dev = &rproc->dev;
> +	struct device_node *r5_node = z_rproc->dev->of_node;
> +
> +	/* go through TCM banks for r5 node */
> +	num_banks = of_count_phandle_with_args(r5_node, BANK_LIST_PROP, NULL);
> +	if (num_banks <= 0) {
> +		dev_err(dev, "need to specify TCM banks\n");
> +		return -EINVAL;
> +	}
> +	for (i = 0; i < num_banks; i++) {
> +		struct resource rsc;
> +		resource_size_t size;
> +		struct device_node *dt_node;
> +		struct rproc_mem_entry *mem;
> +		int ret;
> +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> +
> +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> +		if (!dt_node)
> +			return -EINVAL;
> +
> +		ret = of_address_to_resource(dt_node, 0, &rsc);
> +		if (ret < 0) {
> +			of_node_put(dt_node);
> +			return ret;
> +		}
> +		of_node_put(dt_node);
> +
> +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> +		if (ret < 0)
> +			return ret;
> +
> +		/* add carveout */
> +		size = resource_size(&rsc);
> +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> +					   (size_t)size, rsc.start,
> +					   tcm_mem_alloc,
> +					   tcm_mem_release,
> +					   rsc.name);
> +		if (!mem) {
> +			ret = zynqmp_pm_release_node(pnode_id);

Where is @pnode_id initialised?

> +			if (ret)
> +				dev_warn(dev,
> +					 "fail to release node: %x ret: %x\n",
> +					 pnode_id, ret);
> +			return -ENOMEM;
> +		}
> +
> +		mem->priv = (void *)(u64)pnode_id;


And here too - how does this work when you test things out on your side? 

Please note that I will need a reply to these questions before I review another
set.

> +		rproc_add_carveout(rproc, mem);
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * zynqmp_r5_parse_fw()
> + * @rproc: single R5 core's corresponding rproc instance
> + * @fw: ptr to firmware to be loaded onto r5 core
> + *
> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> +	int ret;
> +
> +	ret = parse_tcm_banks(rproc);
> +	if (ret)
> +		return ret;
> +
> +	ret = parse_mem_regions(rproc);
> +	if (ret)
> +		return ret;
> +
> +	ret = rproc_elf_load_rsc_table(rproc, fw);
> +	if (ret == -EINVAL) {
> +		/*
> +		 * resource table only required for IPC.
> +		 * if not present, this is not necessarily an error;
> +		 * for example, loading r5 hello world application
> +		 * so simply inform user and keep going.
> +		 */
> +		dev_info(&rproc->dev, "no resource table found.\n");
> +		ret = 0;
> +	}
> +	return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> + * @rproc: r5 core's corresponding rproc structure
> + * @vqid: virtqueue ID
> + */
> +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	struct device *dev = rproc->dev.parent;
> +	struct zynqmp_ipi_message *mb_msg = NULL;
> +	unsigned int skb_len;
> +	struct sk_buff *skb;
> +	int ret;
> +
> +	/* If vqid is negative, do not pass the vqid to
> +	 * mailbox as vqid is supposed to be 0 or positive.
> +	 * It also gives a way to just kick instead but
> +	 * not use the IPI buffer.
> +	 *
> +	 * For now use negative vqid to assume no message will be
> +	 * passed with IPI buffer, but just raise interrupt.
> +	 * This will be faster as it doesn't need to copy the
> +	 * message to the ZynqMP/Versal IPI message buffer.
> +	 */
> +	if (vqid >= 0) {
> +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> +		skb = alloc_skb(skb_len, GFP_KERNEL);
> +		if (!skb)
> +			return;
> +
> +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb, skb_len);
> +		mb_msg->len = sizeof(vqid);
> +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> +
> +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> +	}
> +
> +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> +	if (ret < 0) {
> +		dev_warn(dev, "Failed to kick remote.\n");
> +		if (mb_msg) {
> +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> +			kfree_skb(skb);
> +		}
> +	}
> +}
> +

I am done reviewing this set.

Thanks,
Mathieu

> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> +	.start		= zynqmp_r5_rproc_start,
> +	.stop		= zynqmp_r5_rproc_stop,
> +	.load		= rproc_elf_load_segments,
> +	.parse_fw	= zynqmp_r5_parse_fw,
> +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> +	.sanity_check	= rproc_elf_sanity_check,
> +	.get_boot_addr	= rproc_elf_get_boot_addr,
> +	.kick		= zynqmp_r5_rproc_kick,
> +};
> +
> +/**
> + * event_notified_idr_cb() - event notified idr callback
> + * @id: idr id
> + * @ptr: pointer to idr private data
> + * @data: data passed to idr_for_each callback
> + *
> + * Pass notification to remoteproc virtio
> + *
> + * Return: 0. having return is to satisfy the idr_for_each() function
> + *          pointer input argument requirement.
> + **/
> +static int event_notified_idr_cb(int id, void *ptr, void *data)
> +{
> +	struct rproc *rproc = data;
> +
> +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> +		dev_dbg(&rproc->dev, "no message was found in idr with id: %d\n", id);
> +	return 0;
> +}
> +
> +/**
> + * handle_event_notified() - remoteproc notification work function
> + * @work: pointer to the work structure
> + *
> + * It checks each registered remoteproc notify IDs.
> + */
> +static void handle_event_notified(struct work_struct *work)
> +{
> +	struct zynqmp_r5_rproc *z_rproc;
> +	struct rproc *rproc;
> +
> +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> +	rproc = z_rproc->rproc;
> +
> +	/*
> +	 * This is to simply raise interrupt as ACK for remote.
> +	 *
> +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> +	 * mapping to remote processor.
> +	 */
> +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> +
> +	/*
> +	 * We only use IPI for interrupt. The firmware side may or may
> +	 * not write the notifyid when it trigger IPI.
> +	 * And thus, we scan through all the registered notifyids.
> +	 */
> +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> +}
> +
> +/**
> + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> + * @cl: mailbox client
> + * @msg: message pointer
> + *
> + * It will schedule the R5 notification work.
> + */
> +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> +{
> +	struct zynqmp_r5_rproc *z_rproc;
> +
> +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> +	/*
> +	 * Notification can be received with message payload.
> +	 * To handle this, check for message contents.
> +	 * If there are contents, copy to ipi message buffer payload location.
> +	 *
> +	 * If message is empty, then that means that was simple raising
> +	 * of interrupt. no payload to process.
> +	 *
> +	 * Note: enqueue work regardless of msg as kick may imply that remote
> +	 * is waiting for ack.
> +	 */
> +	if (msg) {
> +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> +		size_t len;
> +
> +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> +		buf_msg->len = len;
> +		memcpy(buf_msg->data, ipi_msg->data, len);
> +	}
> +	schedule_work(&z_rproc->mbox_work);
> +}
> +
> +/**
> + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> + * @cl: mailbox client
> + * @msg: pointer to the message which has been sent
> + * @r: status of last TX - OK or error
> + *
> + * It will be called by the mailbox framework when the last TX has done.
> + */
> +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg, int r)
> +{
> +	struct zynqmp_r5_rproc *z_rproc;
> +	struct sk_buff *skb;
> +
> +	if (!msg)
> +		return;
> +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> +	kfree_skb(skb);
> +}
> +
> +/**
> + * zynqmp_r5_setup_mbox() - Setup mailboxes
> + *			    this is used for each individual R5 core
> + *
> + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> + * @node: pointer of the device node
> + *
> + * Function to setup mailboxes to talk to RPU.
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> +				struct device_node *node)
> +{
> +	struct mbox_client *mclient;
> +
> +	/* Setup TX mailbox channel client */
> +	mclient			= &z_rproc->tx_mc;
> +	mclient->dev		= z_rproc->dev;
> +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> +	mclient->tx_block	= false;
> +	mclient->knows_txdone	= false;
> +	mclient->rx_callback	= NULL;
> +
> +	/* Setup RX mailbox channel client */
> +	mclient			= &z_rproc->rx_mc;
> +	mclient->dev		= z_rproc->dev;
> +	mclient->tx_done	= NULL;
> +	mclient->tx_block	= false;
> +	mclient->knows_txdone	= false;
> +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> +
> +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> +
> +	/* Request TX and RX channels */
> +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc->tx_mc, "tx");
> +	if (IS_ERR(z_rproc->tx_chan)) {
> +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> +		return PTR_ERR(z_rproc->tx_chan);
> +	}
> +
> +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc->rx_mc, "rx");
> +	if (IS_ERR(z_rproc->rx_chan)) {
> +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> +		return PTR_ERR(z_rproc->rx_chan);
> +	}
> +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> +
> +	return 0;
> +}
> +
> +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc *z_rproc)
> +{
> +	mbox_free_channel(z_rproc->tx_chan);
> +	mbox_free_channel(z_rproc->rx_chan);
> +}
> +
> +/**
> + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> + *		       this is called for each individual R5 core to
> + *		       set up mailbox, Xilinx platform manager unique ID,
> + *		       add to rproc core
> + *
> + * @pdev: domain platform device for current R5 core
> + * @node: pointer of the device node for current R5 core
> + * @rpu_mode: mode to configure RPU, split or lockstep
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device *pdev,
> +					       struct device_node *node,
> +					       enum rpu_oper_mode rpu_mode)
> +{
> +	int ret;
> +	struct device *dev = &pdev->dev;
> +	struct rproc *rproc_ptr;
> +	struct zynqmp_r5_rproc *z_rproc;
> +
> +	/* Allocate remoteproc instance */
> +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> +				     NULL, sizeof(struct zynqmp_r5_rproc));
> +	if (!rproc_ptr) {
> +		ret = -ENOMEM;
> +		goto error;
> +	}
> +
> +	rproc_ptr->auto_boot = false;
> +	z_rproc = rproc_ptr->priv;
> +	z_rproc->rproc = rproc_ptr;
> +	z_rproc->dev = dev;
> +
> +	/* Set up DMA mask */
> +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> +	if (ret)
> +		goto error;
> +
> +	/* Get R5 power domain node */
> +	ret = of_property_read_u32(node, "power-domain", &z_rproc->pnode_id);
> +	if (ret)
> +		goto error;
> +
> +	ret = r5_set_mode(z_rproc, rpu_mode);
> +	if (ret)
> +		goto error;
> +
> +	if (of_property_read_bool(node, "mboxes")) {
> +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> +		if (ret)
> +			goto error;
> +	}
> +
> +	/* Add R5 remoteproc */
> +	ret = devm_rproc_add(dev, rproc_ptr);
> +	if (ret) {
> +		zynqmp_r5_cleanup_mbox(z_rproc);
> +		goto error;
> +	}
> +
> +	return z_rproc;
> +error:
> +	return ERR_PTR(ret);
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_probe()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * called when driver is probed, for each R5 core specified in DT,
> + * setup as needed to do remoteproc-related operations
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> +{
> +	int ret, core_count;
> +	struct device *dev = &pdev->dev;
> +	struct device_node *nc;
> +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> +	struct list_head *cluster; /* list to track each core's rproc */
> +	struct zynqmp_r5_rproc *z_rproc;
> +	struct platform_device *child_pdev;
> +	struct list_head *pos;
> +
> +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode", &rpu_mode);
> +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> +			ret, rpu_mode);
> +		return ret;
> +	}
> +
> +	dev_dbg(dev, "RPU configuration: %s\n",
> +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" : "split");
> +
> +	/*
> +	 * if 2 RPUs provided but one is lockstep, then we have an
> +	 * invalid configuration.
> +	 */
> +
> +	core_count = of_get_available_child_count(dev->of_node);
> +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> +	    core_count > MAX_RPROCS)
> +		return -EINVAL;
> +
> +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> +	if (!cluster)
> +		return -ENOMEM;
> +	INIT_LIST_HEAD(cluster);
> +
> +	ret = devm_of_platform_populate(dev);
> +	if (ret) {
> +		dev_err(dev, "devm_of_platform_populate failed, ret = %d\n",
> +			ret);
> +		return ret;
> +	}
> +
> +	/* probe each individual r5 core's remoteproc-related info */
> +	for_each_available_child_of_node(dev->of_node, nc) {
> +		child_pdev = of_find_device_by_node(nc);
> +		if (!child_pdev) {
> +			dev_err(dev, "could not get R5 core platform device\n");
> +			ret = -ENODEV;
> +			goto out;
> +		}
> +
> +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> +			ret ? "Failed" : "Able",
> +			nc);
> +		if (IS_ERR(z_rproc)) {
> +			ret = PTR_ERR(z_rproc);
> +			goto out;
> +		}
> +		list_add_tail(&z_rproc->elem, cluster);
> +	}
> +	/* wire in so each core can be cleaned up at driver remove */
> +	platform_set_drvdata(pdev, cluster);
> +	return 0;
> +out:
> +	/*
> +	 * undo core0 upon any failures on core1 in split-mode
> +	 *
> +	 * in zynqmp_r5_probe z_rproc is set to null
> +	 * and ret to non-zero value if error
> +	 */
> +	list_for_each(pos, cluster) {
> +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> +		zynqmp_r5_cleanup_mbox(z_rproc);
> +	}
> +	return ret;
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_remove()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * When the driver is unloaded, clean up the mailboxes for each
> + * remoteproc that was initially probed.
> + */
> +static int zynqmp_r5_remoteproc_remove(struct platform_device *pdev)
> +{
> +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> +						 platform_get_drvdata(pdev);
> +	struct zynqmp_r5_rproc *z_rproc = NULL;
> +
> +	list_for_each_safe(pos, temp, cluster) {
> +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> +		zynqmp_r5_cleanup_mbox(z_rproc);
> +		list_del(pos);
> +	}
> +	return 0;
> +}
> +
> +/* Match table for OF platform binding */
> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> +	{ /* end of list */ },
> +};
> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> +
> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> +	.probe = zynqmp_r5_remoteproc_probe,
> +	.remove = zynqmp_r5_remoteproc_remove,
> +	.driver = {
> +		.name = "zynqmp_r5_remoteproc",
> +		.of_match_table = zynqmp_r5_remoteproc_match,
> +	},
> +};
> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> +
> +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> +MODULE_LICENSE("GPL v2");
> -- 
> 2.17.1
> 

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
@ 2021-02-16 18:48     ` Mathieu Poirier
  0 siblings, 0 replies; 28+ messages in thread
From: Mathieu Poirier @ 2021-02-16 18:48 UTC (permalink / raw)
  To: Ben Levinsky
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	michal.simek

On Sun, Jan 10, 2021 at 06:02:50PM -0800, Ben Levinsky wrote:
> R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
> remotproc driver, we can boot the R5 sub-system in two different
> configurations -
> 	* Split
> 	* Lockstep
> 
> The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
> Platform Management Unit that handles the R5 configuration, memory access
> and R5 lifecycle management. The interface to this manager is done in this
> driver via zynqmp_pm_* function calls.
> 
> Signed-off-by: Wendy Liang <wendy.liang@xilinx.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> Signed-off-by: Ed Mooring <ed.mooring@xilinx.com>
> Signed-off-by: Jason Wu <j.wu@xilinx.com>
> Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> ---
> v25:
> - reword error message for rpu configuration
> - change char name[15] to 16 chars to have null terminated string
>   in parse_mem_regions()
> - reword comments in tcm_mem_alloc
> - call xilinx platform cleanup in parse_tcm_banks() case of failure
> - check value of mbox_send_message in all instances within this driver 
> - indentation in zynqmp_r5_remoteproc driver
> - update style of constructing carveouts to match convention in ST
>   remoteproc drivers. No longer iterate through memory-region property
>   using of_count_phandle_with_args. Instead use of_phandle_iterator
> - use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per feedback
> - rework loop in zynqmp_r5_pm_request_sram as per feedback
> - fix comment and linebreak in tcm_mem_alloc description
> - add comments and remove extraneous devm_ioremap_wc call in tcm_mem_alloc
> - in parse_tcm_banks remove check for bank not being available
> - rework order of locals and remove else in zynqmp_r5_rproc_kick
> - document and update logic for zynqmp_r5_parse_fw
> - add dev_dbg output in event_notified_idr_cb if
>   rproc_vq_interrupt(rproc, id) == IRQ_NONE
> - add comment for handle_event_notified mbox_send_message function call
> - add comment for zynqmp_r5_mb_rx_cb
> - update zynqmp_r5_setup_mbox to match convention of mbox setup in ST
>   remoteproc drivers
> - change return in zynqmp_r5_setup_mbox to use PTR_ERR
> - add zynqmp_r5_cleanup_mbox
> - in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as arg.
> - in zynqmp_r5_probe use return of PTR_ERR and use zynqmp_r5_cleanup_mbox
> - in zynqmp_r5_remoteproc_probe update use return value of zynqmp_r5_probe as
>   either zynqmp_r5_rproc* or PTR_ERR
> - update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_probe
> - update loop that cleans up cluster and mboxes in zynqmp_r5_remoteproc_remove
> ---
>  drivers/remoteproc/Kconfig                |   8 +
>  drivers/remoteproc/Makefile               |   1 +
>  drivers/remoteproc/zynqmp_r5_remoteproc.c | 898 ++++++++++++++++++++++
>  3 files changed, 907 insertions(+)
>  create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c
> 
> diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> index c6659dfea7c7..c2fe54b1d94f 100644
> --- a/drivers/remoteproc/Kconfig
> +++ b/drivers/remoteproc/Kconfig
> @@ -275,6 +275,14 @@ config TI_K3_DSP_REMOTEPROC
>  	  It's safe to say N here if you're not interested in utilizing
>  	  the DSP slave processors.
>  
> +config ZYNQMP_R5_REMOTEPROC
> +	tristate "ZynqMP R5 remoteproc support"
> +	depends on PM && ARCH_ZYNQMP
> +	select RPMSG_VIRTIO
> +	select ZYNQMP_IPI_MBOX
> +	help
> +	  Say y or m here to support ZynqMP R5 remote processors via the remote
> +	  processor framework.
>  endif # REMOTEPROC
>  
>  endmenu
> diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> index 3dfa28e6c701..ef1abff654c2 100644
> --- a/drivers/remoteproc/Makefile
> +++ b/drivers/remoteproc/Makefile
> @@ -33,3 +33,4 @@ obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
>  obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
>  obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
>  obj-$(CONFIG_TI_K3_DSP_REMOTEPROC)	+= ti_k3_dsp_remoteproc.o
> +obj-$(CONFIG_ZYNQMP_R5_REMOTEPROC)	+= zynqmp_r5_remoteproc.o
> diff --git a/drivers/remoteproc/zynqmp_r5_remoteproc.c b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> new file mode 100644
> index 000000000000..9e228c9ecd43
> --- /dev/null
> +++ b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> @@ -0,0 +1,898 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Zynq R5 Remote Processor driver
> + *
> + * Based on origin OMAP and Zynq Remote Processor driver
> + *
> + */
> +
> +#include <linux/firmware/xlnx-zynqmp.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/mailbox_client.h>
> +#include <linux/mailbox/zynqmp-ipi-message.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/platform_device.h>
> +#include <linux/remoteproc.h>
> +#include <linux/skbuff.h>
> +#include <linux/sysfs.h>
> +
> +#include "remoteproc_internal.h"
> +
> +#define MAX_RPROCS	2 /* Support up to 2 RPU */
> +#define MAX_MEM_PNODES	4 /* Max power nodes for one RPU memory instance */
> +
> +#define BANK_LIST_PROP	"sram"
> +#define DDR_LIST_PROP	"memory-region"
> +
> +/* IPI buffer MAX length */
> +#define IPI_BUF_LEN_MAX	32U
> +/* RX mailbox client buffer max length */
> +#define RX_MBOX_CLIENT_BUF_MAX	(IPI_BUF_LEN_MAX + \
> +				 sizeof(struct zynqmp_ipi_message))
> +
> +/*
> + * Map each Xilinx on-chip SRAM  Bank address to their own respective
> + * pm_node_id.
> + */
> +struct sram_addr_data {
> +	phys_addr_t addr;
> +	enum pm_node_id id;
> +};
> +
> +#define NUM_SRAMS 4U
> +static const struct sram_addr_data zynqmp_banks[NUM_SRAMS] = {
> +	{0xffe00000UL, NODE_TCM_0_A},
> +	{0xffe20000UL, NODE_TCM_0_B},
> +	{0xffe90000UL, NODE_TCM_1_A},
> +	{0xffeb0000UL, NODE_TCM_1_B},
> +};
> +
> +/**
> + * struct zynqmp_r5_rproc - ZynqMP R5 core structure
> + *
> + * @rx_mc_buf: rx mailbox client buffer to save the rx message
> + * @tx_mc: tx mailbox client
> + * @rx_mc: rx mailbox client
> + * @mbox_work: mbox_work for the RPU remoteproc
> + * @tx_mc_skbs: socket buffers for tx mailbox client
> + * @dev: device of RPU instance
> + * @rproc: rproc handle
> + * @tx_chan: tx mailbox channel
> + * @rx_chan: rx mailbox channel
> + * @pnode_id: RPU CPU power domain id
> + * @elem: linked list item
> + */
> +struct zynqmp_r5_rproc {
> +	unsigned char rx_mc_buf[RX_MBOX_CLIENT_BUF_MAX];
> +	struct mbox_client tx_mc;
> +	struct mbox_client rx_mc;
> +	struct work_struct mbox_work;
> +	struct sk_buff_head tx_mc_skbs;
> +	struct device *dev;
> +	struct rproc *rproc;
> +	struct mbox_chan *tx_chan;
> +	struct mbox_chan *rx_chan;
> +	u32 pnode_id;
> +	struct list_head elem;
> +};
> +
> +/*
> + * r5_set_mode - set RPU operation mode
> + * @z_rproc: Remote processor private data
> + * @rpu_mode: mode specified by device tree to configure the RPU to
> + *
> + * set RPU operation mode
> + *
> + * Return: 0 for success, negative value for failure
> + */
> +static int r5_set_mode(struct zynqmp_r5_rproc *z_rproc,
> +		       enum rpu_oper_mode rpu_mode)
> +{
> +	enum rpu_tcm_comb tcm_mode;
> +	enum rpu_oper_mode cur_rpu_mode;
> +	int ret;
> +
> +	ret = zynqmp_pm_get_rpu_mode(z_rproc->pnode_id, &cur_rpu_mode);
> +	if (ret < 0)
> +		return ret;
> +
> +	if (rpu_mode != cur_rpu_mode) {
> +		ret = zynqmp_pm_set_rpu_mode(z_rproc->pnode_id, rpu_mode);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	tcm_mode = (rpu_mode == PM_RPU_MODE_LOCKSTEP) ?
> +		    PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> +	return zynqmp_pm_set_tcm_config(z_rproc->pnode_id, tcm_mode);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_release
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap TCM banks when powering down R5 core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_release(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> +	u32 pnode_id = (u64)mem->priv;
> +
> +	iounmap(mem->va);
> +	return zynqmp_pm_release_node(pnode_id);

Is the opposite of that call zynqmp_pm_request_node()?  If so doing the power up
and down of the TCM banks at driver probe() and remove() is highly inefficient.
Consider using rproc_ops::prepare() and rproc_ops::unprepare().

> +}

Please move this just after tcm_mem_alloc()

> +
> +/*
> + * zynqmp_r5_rproc_start
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Start R5 Core from designated boot address.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	enum rpu_boot_mem bootmem;
> +
> +	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
> +		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> +
> +	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
> +		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> +
> +	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
> +				      bootmem, ZYNQMP_PM_REQUEST_ACK_NO);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_stop
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Power down  R5 Core.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +
> +	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
> +				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_alloc
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to map
> + *
> + * Callback to map va for memory-region's carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
> +				     struct rproc_mem_entry *mem)
> +{
> +	void *va;
> +
> +	va = ioremap_wc(mem->dma, mem->len);
> +	if (IS_ERR_OR_NULL(va))
> +		return -ENOMEM;
> +
> +	mem->va = va;
> +
> +	return 0;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_mem_release
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to unmap
> + *
> + * Unmap memory-region carveout
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
> +				       struct rproc_mem_entry *mem)
> +{
> +	iounmap(mem->va);
> +	return 0;
> +}
> +
> +/*
> + * parse_mem_regions
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Construct rproc mem carveouts from carveout provided in
> + * memory-region property
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int parse_mem_regions(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	struct device_node *np = z_rproc->dev->of_node;
> +	struct device *dev = &rproc->dev;
> +	struct of_phandle_iterator it;
> +	struct rproc_mem_entry *mem;
> +	struct reserved_mem *rmem;
> +	int index = 0;
> +
> +	/* Register associated reserved memory regions */
> +	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
> +	while (of_phandle_iterator_next(&it) == 0) {
> +		rmem = of_reserved_mem_lookup(it.node);
> +		if (!rmem) {
> +			dev_err(dev, "unable to acquire %s\n", DDR_LIST_PROP);
> +			return -EINVAL;
> +		}
> +
> +		if (strstr(it.node->name, "vdev0buffer")) {
> +			mem = rproc_of_resm_mem_entry_init(dev, index,
> +							   rmem->size,
> +							   rmem->base,
> +							   "vdev0buffer");
> +		} else {
> +			/*
> +			 * The ensuing carveout is either for vring, which has
> +			 * device tree node name of length 15 characters, or
> +			 * firmware mem, where name is not used by remoteproc
> +			 *  core later on. So default to vring length of 15.

s/"*  core"/"* core"

> +			 *
> +			 * Extra char for null-terminated string.
> +			 */
> +			char name[16];
> +
> +			if (strstr(it.node->name, "vdev0vring")) {
> +				int vring_id;
> +
> +				/*
> +				 * Expect "rpuXvdev0vringX" as documented
> +				 * in xilinx remoteproc device tree binding
> +				 */
> +				if (strlen(it.node->name) < 15) {
> +					dev_err(dev, "%pOF is less than 15 chars",
> +						it.node);
> +					return -EINVAL;
> +				}
> +
> +				/*
> +				 * can be 1 of multiple vring IDs per IPC channel
> +				 * e.g. 'vdev0vring0' and 'vdev0vring1'
> +				 *
> +				 * Here get vring ID from 15th char
> +				 */
> +				vring_id = it.node->name[14] - '0';
> +				snprintf(name, sizeof(name), "vdev0vring%d", vring_id);

Function strstr() returns a pointer to the beginning of the string being
searched for.  As such you should be able to do something like:

                        name = strstr(it.node->name, "vdev0vring");
                        if (name) {
                                ...

and simply use @name when calling rproc_mem_entry_init().

> +			} else {
> +				strncpy(name, it.node->name, 16);
> +			}
> +
> +			mem = rproc_mem_entry_init(dev, NULL,
> +						   (dma_addr_t)rmem->base,
> +						   rmem->size, rmem->base,
> +						   zynqmp_r5_rproc_mem_alloc,
> +						   zynqmp_r5_rproc_mem_release,
> +						   name);
> +		}
> +
> +		if (!mem)
> +			return -ENOMEM;
> +
> +		rproc_add_carveout(rproc, mem);
> +		index++;
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * zynqmp_r5_pm_request_tcm
> + * @addr: base address of mem provided in R5 core's sram property.
> + *
> + * Given sram base address, determine its corresponding Xilinx
> + * Platform Management ID and then request access to this node
> + * so that it can be power up.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
> +{
> +	int i;
> +
> +	for (i = 0; i < NUM_SRAMS; i++) {
> +		if (zynqmp_banks[i].addr != addr)
> +			continue;
> +
> +		return zynqmp_pm_request_node(zynqmp_banks[i].id,
> +					      ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> +					      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> +	}
> +
> +	return -EINVAL;
> +}
> +
> +/*
> + * tcm_mem_alloc
> + * @rproc: single R5 core's corresponding rproc instance
> + * @mem: mem entry to initialize the va and da fields of
> + *
> + * Given TCM bank entry, this callback will set device address for R5
> + * running on TCM and also setup virtual address for TCM bank
> + * remoteproc carveout.
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int tcm_mem_alloc(struct rproc *rproc,
> +			 struct rproc_mem_entry *mem)
> +{
> +	void *va;
> +	struct device *dev = rproc->dev.parent;
> +
> +	va = ioremap_wc(mem->dma, mem->len);
> +	if (IS_ERR_OR_NULL(va))
> +		return -ENOMEM;
> +
> +	/* Update memory entry va */
> +	mem->va = va;
> +
> +	/*
> +	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> +	 * while on the Linux side they are at 0xffexxxxx.
> +	 *
> +	 * Zero out the high 12 bits of the address. This will give
> +	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> +	 */
> +	mem->da &= 0x000fffff;
> +
> +	/*
> +	 * TCM Banks 1A and 1B still have to be translated.
> +	 *
> +	 * Below handle these two banks' absolute addresses (0xffe90000 and
> +	 * 0xffeb0000) and convert to the expected relative addresses
> +	 * (0x0 and 0x20000).
> +	 */
> +	if (mem->da == 0x90000 || mem->da == 0xB0000)
> +		mem->da -= 0x90000;
> +
> +	/* if translated TCM bank address is not valid report error */
> +	if (mem->da != 0x0 && mem->da != 0x20000) {
> +		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * parse_tcm_banks()
> + * @rproc: single R5 core's corresponding rproc instance
> + *
> + * Given R5 node in remoteproc instance
> + * allocate remoteproc carveout for TCM memory
> + * needed for firmware to be loaded
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int parse_tcm_banks(struct rproc *rproc)
> +{
> +	int i, num_banks;
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	struct device *dev = &rproc->dev;
> +	struct device_node *r5_node = z_rproc->dev->of_node;
> +
> +	/* go through TCM banks for r5 node */
> +	num_banks = of_count_phandle_with_args(r5_node, BANK_LIST_PROP, NULL);
> +	if (num_banks <= 0) {
> +		dev_err(dev, "need to specify TCM banks\n");
> +		return -EINVAL;
> +	}
> +	for (i = 0; i < num_banks; i++) {
> +		struct resource rsc;
> +		resource_size_t size;
> +		struct device_node *dt_node;
> +		struct rproc_mem_entry *mem;
> +		int ret;
> +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> +
> +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> +		if (!dt_node)
> +			return -EINVAL;
> +
> +		ret = of_address_to_resource(dt_node, 0, &rsc);
> +		if (ret < 0) {
> +			of_node_put(dt_node);
> +			return ret;
> +		}
> +		of_node_put(dt_node);
> +
> +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> +		if (ret < 0)
> +			return ret;
> +
> +		/* add carveout */
> +		size = resource_size(&rsc);
> +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> +					   (size_t)size, rsc.start,
> +					   tcm_mem_alloc,
> +					   tcm_mem_release,
> +					   rsc.name);
> +		if (!mem) {
> +			ret = zynqmp_pm_release_node(pnode_id);

Where is @pnode_id initialised?

> +			if (ret)
> +				dev_warn(dev,
> +					 "fail to release node: %x ret: %x\n",
> +					 pnode_id, ret);
> +			return -ENOMEM;
> +		}
> +
> +		mem->priv = (void *)(u64)pnode_id;


And here too - how does this work when you test things out on your side? 

Please note that I will need a reply to these questions before I review another
set.

> +		rproc_add_carveout(rproc, mem);
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * zynqmp_r5_parse_fw()
> + * @rproc: single R5 core's corresponding rproc instance
> + * @fw: ptr to firmware to be loaded onto r5 core
> + *
> + * When loading firmware, ensure the necessary carveouts are in remoteproc
> + *
> + * return 0 on success, otherwise non-zero value on failure
> + */
> +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> +	int ret;
> +
> +	ret = parse_tcm_banks(rproc);
> +	if (ret)
> +		return ret;
> +
> +	ret = parse_mem_regions(rproc);
> +	if (ret)
> +		return ret;
> +
> +	ret = rproc_elf_load_rsc_table(rproc, fw);
> +	if (ret == -EINVAL) {
> +		/*
> +		 * resource table only required for IPC.
> +		 * if not present, this is not necessarily an error;
> +		 * for example, loading r5 hello world application
> +		 * so simply inform user and keep going.
> +		 */
> +		dev_info(&rproc->dev, "no resource table found.\n");
> +		ret = 0;
> +	}
> +	return ret;
> +}
> +
> +/*
> + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> + * @rproc: r5 core's corresponding rproc structure
> + * @vqid: virtqueue ID
> + */
> +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> +{
> +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> +	struct device *dev = rproc->dev.parent;
> +	struct zynqmp_ipi_message *mb_msg = NULL;
> +	unsigned int skb_len;
> +	struct sk_buff *skb;
> +	int ret;
> +
> +	/* If vqid is negative, do not pass the vqid to
> +	 * mailbox as vqid is supposed to be 0 or positive.
> +	 * It also gives a way to just kick instead but
> +	 * not use the IPI buffer.
> +	 *
> +	 * For now use negative vqid to assume no message will be
> +	 * passed with IPI buffer, but just raise interrupt.
> +	 * This will be faster as it doesn't need to copy the
> +	 * message to the ZynqMP/Versal IPI message buffer.
> +	 */
> +	if (vqid >= 0) {
> +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> +		skb = alloc_skb(skb_len, GFP_KERNEL);
> +		if (!skb)
> +			return;
> +
> +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb, skb_len);
> +		mb_msg->len = sizeof(vqid);
> +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> +
> +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> +	}
> +
> +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> +	if (ret < 0) {
> +		dev_warn(dev, "Failed to kick remote.\n");
> +		if (mb_msg) {
> +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> +			kfree_skb(skb);
> +		}
> +	}
> +}
> +

I am done reviewing this set.

Thanks,
Mathieu

> +static struct rproc_ops zynqmp_r5_rproc_ops = {
> +	.start		= zynqmp_r5_rproc_start,
> +	.stop		= zynqmp_r5_rproc_stop,
> +	.load		= rproc_elf_load_segments,
> +	.parse_fw	= zynqmp_r5_parse_fw,
> +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> +	.sanity_check	= rproc_elf_sanity_check,
> +	.get_boot_addr	= rproc_elf_get_boot_addr,
> +	.kick		= zynqmp_r5_rproc_kick,
> +};
> +
> +/**
> + * event_notified_idr_cb() - event notified idr callback
> + * @id: idr id
> + * @ptr: pointer to idr private data
> + * @data: data passed to idr_for_each callback
> + *
> + * Pass notification to remoteproc virtio
> + *
> + * Return: 0. having return is to satisfy the idr_for_each() function
> + *          pointer input argument requirement.
> + **/
> +static int event_notified_idr_cb(int id, void *ptr, void *data)
> +{
> +	struct rproc *rproc = data;
> +
> +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> +		dev_dbg(&rproc->dev, "no message was found in idr with id: %d\n", id);
> +	return 0;
> +}
> +
> +/**
> + * handle_event_notified() - remoteproc notification work function
> + * @work: pointer to the work structure
> + *
> + * It checks each registered remoteproc notify IDs.
> + */
> +static void handle_event_notified(struct work_struct *work)
> +{
> +	struct zynqmp_r5_rproc *z_rproc;
> +	struct rproc *rproc;
> +
> +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> +	rproc = z_rproc->rproc;
> +
> +	/*
> +	 * This is to simply raise interrupt as ACK for remote.
> +	 *
> +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> +	 * mapping to remote processor.
> +	 */
> +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> +
> +	/*
> +	 * We only use IPI for interrupt. The firmware side may or may
> +	 * not write the notifyid when it trigger IPI.
> +	 * And thus, we scan through all the registered notifyids.
> +	 */
> +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> +}
> +
> +/**
> + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> + * @cl: mailbox client
> + * @msg: message pointer
> + *
> + * It will schedule the R5 notification work.
> + */
> +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> +{
> +	struct zynqmp_r5_rproc *z_rproc;
> +
> +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> +	/*
> +	 * Notification can be received with message payload.
> +	 * To handle this, check for message contents.
> +	 * If there are contents, copy to ipi message buffer payload location.
> +	 *
> +	 * If message is empty, then that means that was simple raising
> +	 * of interrupt. no payload to process.
> +	 *
> +	 * Note: enqueue work regardless of msg as kick may imply that remote
> +	 * is waiting for ack.
> +	 */
> +	if (msg) {
> +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> +		size_t len;
> +
> +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> +		buf_msg->len = len;
> +		memcpy(buf_msg->data, ipi_msg->data, len);
> +	}
> +	schedule_work(&z_rproc->mbox_work);
> +}
> +
> +/**
> + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> + * @cl: mailbox client
> + * @msg: pointer to the message which has been sent
> + * @r: status of last TX - OK or error
> + *
> + * It will be called by the mailbox framework when the last TX has done.
> + */
> +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg, int r)
> +{
> +	struct zynqmp_r5_rproc *z_rproc;
> +	struct sk_buff *skb;
> +
> +	if (!msg)
> +		return;
> +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> +	kfree_skb(skb);
> +}
> +
> +/**
> + * zynqmp_r5_setup_mbox() - Setup mailboxes
> + *			    this is used for each individual R5 core
> + *
> + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> + * @node: pointer of the device node
> + *
> + * Function to setup mailboxes to talk to RPU.
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> +				struct device_node *node)
> +{
> +	struct mbox_client *mclient;
> +
> +	/* Setup TX mailbox channel client */
> +	mclient			= &z_rproc->tx_mc;
> +	mclient->dev		= z_rproc->dev;
> +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> +	mclient->tx_block	= false;
> +	mclient->knows_txdone	= false;
> +	mclient->rx_callback	= NULL;
> +
> +	/* Setup RX mailbox channel client */
> +	mclient			= &z_rproc->rx_mc;
> +	mclient->dev		= z_rproc->dev;
> +	mclient->tx_done	= NULL;
> +	mclient->tx_block	= false;
> +	mclient->knows_txdone	= false;
> +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> +
> +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> +
> +	/* Request TX and RX channels */
> +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc->tx_mc, "tx");
> +	if (IS_ERR(z_rproc->tx_chan)) {
> +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> +		return PTR_ERR(z_rproc->tx_chan);
> +	}
> +
> +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc->rx_mc, "rx");
> +	if (IS_ERR(z_rproc->rx_chan)) {
> +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> +		return PTR_ERR(z_rproc->rx_chan);
> +	}
> +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> +
> +	return 0;
> +}
> +
> +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc *z_rproc)
> +{
> +	mbox_free_channel(z_rproc->tx_chan);
> +	mbox_free_channel(z_rproc->rx_chan);
> +}
> +
> +/**
> + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> + *		       this is called for each individual R5 core to
> + *		       set up mailbox, Xilinx platform manager unique ID,
> + *		       add to rproc core
> + *
> + * @pdev: domain platform device for current R5 core
> + * @node: pointer of the device node for current R5 core
> + * @rpu_mode: mode to configure RPU, split or lockstep
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device *pdev,
> +					       struct device_node *node,
> +					       enum rpu_oper_mode rpu_mode)
> +{
> +	int ret;
> +	struct device *dev = &pdev->dev;
> +	struct rproc *rproc_ptr;
> +	struct zynqmp_r5_rproc *z_rproc;
> +
> +	/* Allocate remoteproc instance */
> +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
> +				     NULL, sizeof(struct zynqmp_r5_rproc));
> +	if (!rproc_ptr) {
> +		ret = -ENOMEM;
> +		goto error;
> +	}
> +
> +	rproc_ptr->auto_boot = false;
> +	z_rproc = rproc_ptr->priv;
> +	z_rproc->rproc = rproc_ptr;
> +	z_rproc->dev = dev;
> +
> +	/* Set up DMA mask */
> +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> +	if (ret)
> +		goto error;
> +
> +	/* Get R5 power domain node */
> +	ret = of_property_read_u32(node, "power-domain", &z_rproc->pnode_id);
> +	if (ret)
> +		goto error;
> +
> +	ret = r5_set_mode(z_rproc, rpu_mode);
> +	if (ret)
> +		goto error;
> +
> +	if (of_property_read_bool(node, "mboxes")) {
> +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> +		if (ret)
> +			goto error;
> +	}
> +
> +	/* Add R5 remoteproc */
> +	ret = devm_rproc_add(dev, rproc_ptr);
> +	if (ret) {
> +		zynqmp_r5_cleanup_mbox(z_rproc);
> +		goto error;
> +	}
> +
> +	return z_rproc;
> +error:
> +	return ERR_PTR(ret);
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_probe()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * called when driver is probed, for each R5 core specified in DT,
> + * setup as needed to do remoteproc-related operations
> + *
> + * Return: 0 for success, negative value for failure.
> + */
> +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> +{
> +	int ret, core_count;
> +	struct device *dev = &pdev->dev;
> +	struct device_node *nc;
> +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> +	struct list_head *cluster; /* list to track each core's rproc */
> +	struct zynqmp_r5_rproc *z_rproc;
> +	struct platform_device *child_pdev;
> +	struct list_head *pos;
> +
> +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode", &rpu_mode);
> +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> +			ret, rpu_mode);
> +		return ret;
> +	}
> +
> +	dev_dbg(dev, "RPU configuration: %s\n",
> +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" : "split");
> +
> +	/*
> +	 * if 2 RPUs provided but one is lockstep, then we have an
> +	 * invalid configuration.
> +	 */
> +
> +	core_count = of_get_available_child_count(dev->of_node);
> +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> +	    core_count > MAX_RPROCS)
> +		return -EINVAL;
> +
> +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> +	if (!cluster)
> +		return -ENOMEM;
> +	INIT_LIST_HEAD(cluster);
> +
> +	ret = devm_of_platform_populate(dev);
> +	if (ret) {
> +		dev_err(dev, "devm_of_platform_populate failed, ret = %d\n",
> +			ret);
> +		return ret;
> +	}
> +
> +	/* probe each individual r5 core's remoteproc-related info */
> +	for_each_available_child_of_node(dev->of_node, nc) {
> +		child_pdev = of_find_device_by_node(nc);
> +		if (!child_pdev) {
> +			dev_err(dev, "could not get R5 core platform device\n");
> +			ret = -ENODEV;
> +			goto out;
> +		}
> +
> +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> +			ret ? "Failed" : "Able",
> +			nc);
> +		if (IS_ERR(z_rproc)) {
> +			ret = PTR_ERR(z_rproc);
> +			goto out;
> +		}
> +		list_add_tail(&z_rproc->elem, cluster);
> +	}
> +	/* wire in so each core can be cleaned up at driver remove */
> +	platform_set_drvdata(pdev, cluster);
> +	return 0;
> +out:
> +	/*
> +	 * undo core0 upon any failures on core1 in split-mode
> +	 *
> +	 * in zynqmp_r5_probe z_rproc is set to null
> +	 * and ret to non-zero value if error
> +	 */
> +	list_for_each(pos, cluster) {
> +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> +		zynqmp_r5_cleanup_mbox(z_rproc);
> +	}
> +	return ret;
> +}
> +
> +/*
> + * zynqmp_r5_remoteproc_remove()
> + *
> + * @pdev: domain platform device for R5 cluster
> + *
> + * When the driver is unloaded, clean up the mailboxes for each
> + * remoteproc that was initially probed.
> + */
> +static int zynqmp_r5_remoteproc_remove(struct platform_device *pdev)
> +{
> +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> +						 platform_get_drvdata(pdev);
> +	struct zynqmp_r5_rproc *z_rproc = NULL;
> +
> +	list_for_each_safe(pos, temp, cluster) {
> +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> +		zynqmp_r5_cleanup_mbox(z_rproc);
> +		list_del(pos);
> +	}
> +	return 0;
> +}
> +
> +/* Match table for OF platform binding */
> +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> +	{ /* end of list */ },
> +};
> +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> +
> +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> +	.probe = zynqmp_r5_remoteproc_probe,
> +	.remove = zynqmp_r5_remoteproc_remove,
> +	.driver = {
> +		.name = "zynqmp_r5_remoteproc",
> +		.of_match_table = zynqmp_r5_remoteproc_match,
> +	},
> +};
> +module_platform_driver(zynqmp_r5_remoteproc_driver);
> +
> +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> +MODULE_LICENSE("GPL v2");
> -- 
> 2.17.1
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 28+ messages in thread

* RE: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
  2021-02-16 18:48     ` Mathieu Poirier
@ 2021-02-17 16:47       ` Ben Levinsky
  -1 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-02-17 16:47 UTC (permalink / raw)
  To: Mathieu Poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	Michal Simek

Hi Mathieu,

Please see my replies to your questions inline

Again thanks for the continued review of this set

Cheers
Ben

> -----Original Message-----
> From: Mathieu Poirier <mathieu.poirier@linaro.org>
> Sent: Tuesday, February 16, 2021 10:48 AM
> To: Ben Levinsky <BLEVINSK@xilinx.com>
> Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal Simek
> <michals@xilinx.com>
> Subject: Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc
> driver
> 
> On Sun, Jan 10, 2021 at 06:02:50PM -0800, Ben Levinsky wrote:
> > R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
> > remotproc driver, we can boot the R5 sub-system in two different
> > configurations -
> > 	* Split
> > 	* Lockstep
> >
> > The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
> > Platform Management Unit that handles the R5 configuration, memory
> access
> > and R5 lifecycle management. The interface to this manager is done in this
> > driver via zynqmp_pm_* function calls.
> >
> > Signed-off-by: Wendy Liang <wendy.liang@xilinx.com>
> > Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> > Signed-off-by: Ed Mooring <ed.mooring@xilinx.com>
> > Signed-off-by: Jason Wu <j.wu@xilinx.com>
> > Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> > ---
> > v25:
> > - reword error message for rpu configuration
> > - change char name[15] to 16 chars to have null terminated string
> >   in parse_mem_regions()
> > - reword comments in tcm_mem_alloc
> > - call xilinx platform cleanup in parse_tcm_banks() case of failure
> > - check value of mbox_send_message in all instances within this driver
> > - indentation in zynqmp_r5_remoteproc driver
> > - update style of constructing carveouts to match convention in ST
> >   remoteproc drivers. No longer iterate through memory-region property
> >   using of_count_phandle_with_args. Instead use of_phandle_iterator
> > - use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per
> feedback
> > - rework loop in zynqmp_r5_pm_request_sram as per feedback
> > - fix comment and linebreak in tcm_mem_alloc description
> > - add comments and remove extraneous devm_ioremap_wc call in
> tcm_mem_alloc
> > - in parse_tcm_banks remove check for bank not being available
> > - rework order of locals and remove else in zynqmp_r5_rproc_kick
> > - document and update logic for zynqmp_r5_parse_fw
> > - add dev_dbg output in event_notified_idr_cb if
> >   rproc_vq_interrupt(rproc, id) == IRQ_NONE
> > - add comment for handle_event_notified mbox_send_message function
> call
> > - add comment for zynqmp_r5_mb_rx_cb
> > - update zynqmp_r5_setup_mbox to match convention of mbox setup in ST
> >   remoteproc drivers
> > - change return in zynqmp_r5_setup_mbox to use PTR_ERR
> > - add zynqmp_r5_cleanup_mbox
> > - in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as arg.
> > - in zynqmp_r5_probe use return of PTR_ERR and use
> zynqmp_r5_cleanup_mbox
> > - in zynqmp_r5_remoteproc_probe update use return value of
> zynqmp_r5_probe as
> >   either zynqmp_r5_rproc* or PTR_ERR
> > - update loop that cleans up cluster and mboxes in
> zynqmp_r5_remoteproc_probe
> > - update loop that cleans up cluster and mboxes in
> zynqmp_r5_remoteproc_remove
> > ---
> >  drivers/remoteproc/Kconfig                |   8 +
> >  drivers/remoteproc/Makefile               |   1 +
> >  drivers/remoteproc/zynqmp_r5_remoteproc.c | 898
> ++++++++++++++++++++++
> >  3 files changed, 907 insertions(+)
> >  create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c
> >
> > diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> > index c6659dfea7c7..c2fe54b1d94f 100644
> > --- a/drivers/remoteproc/Kconfig
> > +++ b/drivers/remoteproc/Kconfig
> > @@ -275,6 +275,14 @@ config TI_K3_DSP_REMOTEPROC
> >  	  It's safe to say N here if you're not interested in utilizing
> >  	  the DSP slave processors.
> >
> > +config ZYNQMP_R5_REMOTEPROC
> > +	tristate "ZynqMP R5 remoteproc support"
> > +	depends on PM && ARCH_ZYNQMP
> > +	select RPMSG_VIRTIO
> > +	select ZYNQMP_IPI_MBOX
> > +	help
> > +	  Say y or m here to support ZynqMP R5 remote processors via the
> remote
> > +	  processor framework.
> >  endif # REMOTEPROC
> >
> >  endmenu
> > diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> > index 3dfa28e6c701..ef1abff654c2 100644
> > --- a/drivers/remoteproc/Makefile
> > +++ b/drivers/remoteproc/Makefile
> > @@ -33,3 +33,4 @@ obj-$(CONFIG_ST_REMOTEPROC)		+=
> st_remoteproc.o
> >  obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
> >  obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
> >  obj-$(CONFIG_TI_K3_DSP_REMOTEPROC)	+= ti_k3_dsp_remoteproc.o
> > +obj-$(CONFIG_ZYNQMP_R5_REMOTEPROC)	+= zynqmp_r5_remoteproc.o
> > diff --git a/drivers/remoteproc/zynqmp_r5_remoteproc.c
> b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > new file mode 100644
> > index 000000000000..9e228c9ecd43
> > --- /dev/null
> > +++ b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > @@ -0,0 +1,898 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Zynq R5 Remote Processor driver
> > + *
> > + * Based on origin OMAP and Zynq Remote Processor driver
> > + *
> > + */
> > +
> > +#include <linux/firmware/xlnx-zynqmp.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/kernel.h>
> > +#include <linux/list.h>
> > +#include <linux/mailbox_client.h>
> > +#include <linux/mailbox/zynqmp-ipi-message.h>
> > +#include <linux/module.h>
> > +#include <linux/of_address.h>
> > +#include <linux/of_platform.h>
> > +#include <linux/of_reserved_mem.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/remoteproc.h>
> > +#include <linux/skbuff.h>
> > +#include <linux/sysfs.h>
> > +
> > +#include "remoteproc_internal.h"
> > +
> > +#define MAX_RPROCS	2 /* Support up to 2 RPU */
> > +#define MAX_MEM_PNODES	4 /* Max power nodes for one RPU memory
> instance */
> > +
> > +#define BANK_LIST_PROP	"sram"
> > +#define DDR_LIST_PROP	"memory-region"
> > +
> > +/* IPI buffer MAX length */
> > +#define IPI_BUF_LEN_MAX	32U
> > +/* RX mailbox client buffer max length */
> > +#define RX_MBOX_CLIENT_BUF_MAX	(IPI_BUF_LEN_MAX + \
> > +				 sizeof(struct zynqmp_ipi_message))
> > +
> > +/*
> > + * Map each Xilinx on-chip SRAM  Bank address to their own respective
> > + * pm_node_id.
> > + */
> > +struct sram_addr_data {
> > +	phys_addr_t addr;
> > +	enum pm_node_id id;
> > +};
> > +
> > +#define NUM_SRAMS 4U
> > +static const struct sram_addr_data zynqmp_banks[NUM_SRAMS] = {
> > +	{0xffe00000UL, NODE_TCM_0_A},
> > +	{0xffe20000UL, NODE_TCM_0_B},
> > +	{0xffe90000UL, NODE_TCM_1_A},
> > +	{0xffeb0000UL, NODE_TCM_1_B},
> > +};
> > +
> > +/**
> > + * struct zynqmp_r5_rproc - ZynqMP R5 core structure
> > + *
> > + * @rx_mc_buf: rx mailbox client buffer to save the rx message
> > + * @tx_mc: tx mailbox client
> > + * @rx_mc: rx mailbox client
> > + * @mbox_work: mbox_work for the RPU remoteproc
> > + * @tx_mc_skbs: socket buffers for tx mailbox client
> > + * @dev: device of RPU instance
> > + * @rproc: rproc handle
> > + * @tx_chan: tx mailbox channel
> > + * @rx_chan: rx mailbox channel
> > + * @pnode_id: RPU CPU power domain id
> > + * @elem: linked list item
> > + */
> > +struct zynqmp_r5_rproc {
> > +	unsigned char rx_mc_buf[RX_MBOX_CLIENT_BUF_MAX];
> > +	struct mbox_client tx_mc;
> > +	struct mbox_client rx_mc;
> > +	struct work_struct mbox_work;
> > +	struct sk_buff_head tx_mc_skbs;
> > +	struct device *dev;
> > +	struct rproc *rproc;
> > +	struct mbox_chan *tx_chan;
> > +	struct mbox_chan *rx_chan;
> > +	u32 pnode_id;
> > +	struct list_head elem;
> > +};
> > +
> > +/*
> > + * r5_set_mode - set RPU operation mode
> > + * @z_rproc: Remote processor private data
> > + * @rpu_mode: mode specified by device tree to configure the RPU to
> > + *
> > + * set RPU operation mode
> > + *
> > + * Return: 0 for success, negative value for failure
> > + */
> > +static int r5_set_mode(struct zynqmp_r5_rproc *z_rproc,
> > +		       enum rpu_oper_mode rpu_mode)
> > +{
> > +	enum rpu_tcm_comb tcm_mode;
> > +	enum rpu_oper_mode cur_rpu_mode;
> > +	int ret;
> > +
> > +	ret = zynqmp_pm_get_rpu_mode(z_rproc->pnode_id,
> &cur_rpu_mode);
> > +	if (ret < 0)
> > +		return ret;
> > +
> > +	if (rpu_mode != cur_rpu_mode) {
> > +		ret = zynqmp_pm_set_rpu_mode(z_rproc->pnode_id,
> rpu_mode);
> > +		if (ret < 0)
> > +			return ret;
> > +	}
> > +
> > +	tcm_mode = (rpu_mode == PM_RPU_MODE_LOCKSTEP) ?
> > +		    PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> > +	return zynqmp_pm_set_tcm_config(z_rproc->pnode_id, tcm_mode);
> > +}
> > +
> > +/*
> > + * zynqmp_r5_rproc_mem_release
> > + * @rproc: single R5 core's corresponding rproc instance
> > + * @mem: mem entry to unmap
> > + *
> > + * Unmap TCM banks when powering down R5 core.
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int tcm_mem_release(struct rproc *rproc, struct rproc_mem_entry
> *mem)
> > +{
> > +	u32 pnode_id = (u64)mem->priv;
> > +
> > +	iounmap(mem->va);
> > +	return zynqmp_pm_release_node(pnode_id);
> 
> Is the opposite of that call zynqmp_pm_request_node()?  If so doing the
> power up
> and down of the TCM banks at driver probe() and remove() is highly
> inefficient.
> Consider using rproc_ops::prepare() and rproc_ops::unprepare().
> 
> > +}
> 
> Please move this just after tcm_mem_alloc()
> 
> > +
> > +/*
> > + * zynqmp_r5_rproc_start
> > + * @rproc: single R5 core's corresponding rproc instance
> > + *
> > + * Start R5 Core from designated boot address.
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > +	enum rpu_boot_mem bootmem;
> > +
> > +	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
> > +		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> > +
> > +	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
> > +		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> > +
> > +	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
> > +				      bootmem,
> ZYNQMP_PM_REQUEST_ACK_NO);
> > +}
> > +
> > +/*
> > + * zynqmp_r5_rproc_stop
> > + * @rproc: single R5 core's corresponding rproc instance
> > + *
> > + * Power down  R5 Core.
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > +
> > +	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
> > +				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > +}
> > +
> > +/*
> > + * zynqmp_r5_rproc_mem_alloc
> > + * @rproc: single R5 core's corresponding rproc instance
> > + * @mem: mem entry to map
> > + *
> > + * Callback to map va for memory-region's carveout.
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
> > +				     struct rproc_mem_entry *mem)
> > +{
> > +	void *va;
> > +
> > +	va = ioremap_wc(mem->dma, mem->len);
> > +	if (IS_ERR_OR_NULL(va))
> > +		return -ENOMEM;
> > +
> > +	mem->va = va;
> > +
> > +	return 0;
> > +}
> > +
> > +/*
> > + * zynqmp_r5_rproc_mem_release
> > + * @rproc: single R5 core's corresponding rproc instance
> > + * @mem: mem entry to unmap
> > + *
> > + * Unmap memory-region carveout
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
> > +				       struct rproc_mem_entry *mem)
> > +{
> > +	iounmap(mem->va);
> > +	return 0;
> > +}
> > +
> > +/*
> > + * parse_mem_regions
> > + * @rproc: single R5 core's corresponding rproc instance
> > + *
> > + * Construct rproc mem carveouts from carveout provided in
> > + * memory-region property
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int parse_mem_regions(struct rproc *rproc)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > +	struct device_node *np = z_rproc->dev->of_node;
> > +	struct device *dev = &rproc->dev;
> > +	struct of_phandle_iterator it;
> > +	struct rproc_mem_entry *mem;
> > +	struct reserved_mem *rmem;
> > +	int index = 0;
> > +
> > +	/* Register associated reserved memory regions */
> > +	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
> > +	while (of_phandle_iterator_next(&it) == 0) {
> > +		rmem = of_reserved_mem_lookup(it.node);
> > +		if (!rmem) {
> > +			dev_err(dev, "unable to acquire %s\n",
> DDR_LIST_PROP);
> > +			return -EINVAL;
> > +		}
> > +
> > +		if (strstr(it.node->name, "vdev0buffer")) {
> > +			mem = rproc_of_resm_mem_entry_init(dev, index,
> > +							   rmem->size,
> > +							   rmem->base,
> > +							   "vdev0buffer");
> > +		} else {
> > +			/*
> > +			 * The ensuing carveout is either for vring, which has
> > +			 * device tree node name of length 15 characters, or
> > +			 * firmware mem, where name is not used by
> remoteproc
> > +			 *  core later on. So default to vring length of 15.
> 
> s/"*  core"/"* core"
> 
> > +			 *
> > +			 * Extra char for null-terminated string.
> > +			 */
> > +			char name[16];
> > +
> > +			if (strstr(it.node->name, "vdev0vring")) {
> > +				int vring_id;
> > +
> > +				/*
> > +				 * Expect "rpuXvdev0vringX" as documented
> > +				 * in xilinx remoteproc device tree binding
> > +				 */
> > +				if (strlen(it.node->name) < 15) {
> > +					dev_err(dev, "%pOF is less than 15
> chars",
> > +						it.node);
> > +					return -EINVAL;
> > +				}
> > +
> > +				/*
> > +				 * can be 1 of multiple vring IDs per IPC
> channel
> > +				 * e.g. 'vdev0vring0' and 'vdev0vring1'
> > +				 *
> > +				 * Here get vring ID from 15th char
> > +				 */
> > +				vring_id = it.node->name[14] - '0';
> > +				snprintf(name, sizeof(name), "vdev0vring%d",
> vring_id);
> 
> Function strstr() returns a pointer to the beginning of the string being
> searched for.  As such you should be able to do something like:
> 
>                         name = strstr(it.node->name, "vdev0vring");
>                         if (name) {
>                                 ...
> 
> and simply use @name when calling rproc_mem_entry_init().
> 
> > +			} else {
> > +				strncpy(name, it.node->name, 16);
> > +			}
> > +
> > +			mem = rproc_mem_entry_init(dev, NULL,
> > +						   (dma_addr_t)rmem->base,
> > +						   rmem->size, rmem->base,
> > +
> zynqmp_r5_rproc_mem_alloc,
> > +
> zynqmp_r5_rproc_mem_release,
> > +						   name);
> > +		}
> > +
> > +		if (!mem)
> > +			return -ENOMEM;
> > +
> > +		rproc_add_carveout(rproc, mem);
> > +		index++;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +/*
> > + * zynqmp_r5_pm_request_tcm
> > + * @addr: base address of mem provided in R5 core's sram property.
> > + *
> > + * Given sram base address, determine its corresponding Xilinx
> > + * Platform Management ID and then request access to this node
> > + * so that it can be power up.
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
> > +{
> > +	int i;
> > +
> > +	for (i = 0; i < NUM_SRAMS; i++) {
> > +		if (zynqmp_banks[i].addr != addr)
> > +			continue;
> > +
> > +		return zynqmp_pm_request_node(zynqmp_banks[i].id,
> > +					      ZYNQMP_PM_CAPABILITY_ACCESS,
> 0,
> > +
> ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > +	}
> > +
> > +	return -EINVAL;
> > +}
> > +
> > +/*
> > + * tcm_mem_alloc
> > + * @rproc: single R5 core's corresponding rproc instance
> > + * @mem: mem entry to initialize the va and da fields of
> > + *
> > + * Given TCM bank entry, this callback will set device address for R5
> > + * running on TCM and also setup virtual address for TCM bank
> > + * remoteproc carveout.
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int tcm_mem_alloc(struct rproc *rproc,
> > +			 struct rproc_mem_entry *mem)
> > +{
> > +	void *va;
> > +	struct device *dev = rproc->dev.parent;
> > +
> > +	va = ioremap_wc(mem->dma, mem->len);
> > +	if (IS_ERR_OR_NULL(va))
> > +		return -ENOMEM;
> > +
> > +	/* Update memory entry va */
> > +	mem->va = va;
> > +
> > +	/*
> > +	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> > +	 * while on the Linux side they are at 0xffexxxxx.
> > +	 *
> > +	 * Zero out the high 12 bits of the address. This will give
> > +	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> > +	 */
> > +	mem->da &= 0x000fffff;
> > +
> > +	/*
> > +	 * TCM Banks 1A and 1B still have to be translated.
> > +	 *
> > +	 * Below handle these two banks' absolute addresses (0xffe90000 and
> > +	 * 0xffeb0000) and convert to the expected relative addresses
> > +	 * (0x0 and 0x20000).
> > +	 */
> > +	if (mem->da == 0x90000 || mem->da == 0xB0000)
> > +		mem->da -= 0x90000;
> > +
> > +	/* if translated TCM bank address is not valid report error */
> > +	if (mem->da != 0x0 && mem->da != 0x20000) {
> > +		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
> > +		return -EINVAL;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +/*
> > + * parse_tcm_banks()
> > + * @rproc: single R5 core's corresponding rproc instance
> > + *
> > + * Given R5 node in remoteproc instance
> > + * allocate remoteproc carveout for TCM memory
> > + * needed for firmware to be loaded
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int parse_tcm_banks(struct rproc *rproc)
> > +{
> > +	int i, num_banks;
> > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > +	struct device *dev = &rproc->dev;
> > +	struct device_node *r5_node = z_rproc->dev->of_node;
> > +
> > +	/* go through TCM banks for r5 node */
> > +	num_banks = of_count_phandle_with_args(r5_node,
> BANK_LIST_PROP, NULL);
> > +	if (num_banks <= 0) {
> > +		dev_err(dev, "need to specify TCM banks\n");
> > +		return -EINVAL;
> > +	}
> > +	for (i = 0; i < num_banks; i++) {
> > +		struct resource rsc;
> > +		resource_size_t size;
> > +		struct device_node *dt_node;
> > +		struct rproc_mem_entry *mem;
> > +		int ret;
> > +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> > +
> > +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> > +		if (!dt_node)
> > +			return -EINVAL;
> > +
> > +		ret = of_address_to_resource(dt_node, 0, &rsc);
> > +		if (ret < 0) {
> > +			of_node_put(dt_node);
> > +			return ret;
> > +		}
> > +		of_node_put(dt_node);
> > +
> > +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> > +		if (ret < 0)
> > +			return ret;
> > +
> > +		/* add carveout */
> > +		size = resource_size(&rsc);
> > +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> > +					   (size_t)size, rsc.start,
> > +					   tcm_mem_alloc,
> > +					   tcm_mem_release,
> > +					   rsc.name);
> > +		if (!mem) {
> > +			ret = zynqmp_pm_release_node(pnode_id);
> 
> Where is @pnode_id initialised?

Good catch. Previously this was value set in an earlier iteration. I will fix this as follows:
- release the R5 cluster by instead passing z_rproc->pnode_id
- release the TCM bank via call to release_sram

This will also remove the variable pnode_id.

> 
> > +			if (ret)
> > +				dev_warn(dev,
> > +					 "fail to release node: %x ret: %x\n",
> > +					 pnode_id, ret);
> > +			return -ENOMEM;
> > +		}
> > +
> > +		mem->priv = (void *)(u64)pnode_id;
> 
> 
> And here too - how does this work when you test things out on your side?
> 
> Please note that I will need a reply to these questions before I review another
> set.
> 
Here I will update as follows:
- set mem->priv to value from zynqmp_banks lookup table that contains power node IDs


> > +		rproc_add_carveout(rproc, mem);
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +/*
> > + * zynqmp_r5_parse_fw()
> > + * @rproc: single R5 core's corresponding rproc instance
> > + * @fw: ptr to firmware to be loaded onto r5 core
> > + *
> > + * When loading firmware, ensure the necessary carveouts are in
> remoteproc
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware
> *fw)
> > +{
> > +	int ret;
> > +
> > +	ret = parse_tcm_banks(rproc);
> > +	if (ret)
> > +		return ret;
> > +
> > +	ret = parse_mem_regions(rproc);
> > +	if (ret)
> > +		return ret;
> > +
> > +	ret = rproc_elf_load_rsc_table(rproc, fw);
> > +	if (ret == -EINVAL) {
> > +		/*
> > +		 * resource table only required for IPC.
> > +		 * if not present, this is not necessarily an error;
> > +		 * for example, loading r5 hello world application
> > +		 * so simply inform user and keep going.
> > +		 */
> > +		dev_info(&rproc->dev, "no resource table found.\n");
> > +		ret = 0;
> > +	}
> > +	return ret;
> > +}
> > +
> > +/*
> > + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> > + * @rproc: r5 core's corresponding rproc structure
> > + * @vqid: virtqueue ID
> > + */
> > +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > +	struct device *dev = rproc->dev.parent;
> > +	struct zynqmp_ipi_message *mb_msg = NULL;
> > +	unsigned int skb_len;
> > +	struct sk_buff *skb;
> > +	int ret;
> > +
> > +	/* If vqid is negative, do not pass the vqid to
> > +	 * mailbox as vqid is supposed to be 0 or positive.
> > +	 * It also gives a way to just kick instead but
> > +	 * not use the IPI buffer.
> > +	 *
> > +	 * For now use negative vqid to assume no message will be
> > +	 * passed with IPI buffer, but just raise interrupt.
> > +	 * This will be faster as it doesn't need to copy the
> > +	 * message to the ZynqMP/Versal IPI message buffer.
> > +	 */
> > +	if (vqid >= 0) {
> > +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> > +		skb = alloc_skb(skb_len, GFP_KERNEL);
> > +		if (!skb)
> > +			return;
> > +
> > +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb,
> skb_len);
> > +		mb_msg->len = sizeof(vqid);
> > +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> > +
> > +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> > +	}
> > +
> > +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> > +	if (ret < 0) {
> > +		dev_warn(dev, "Failed to kick remote.\n");
> > +		if (mb_msg) {
> > +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> > +			kfree_skb(skb);
> > +		}
> > +	}
> > +}
> > +
> 
> I am done reviewing this set.
> 
> Thanks,
> Mathieu
> 
> > +static struct rproc_ops zynqmp_r5_rproc_ops = {
> > +	.start		= zynqmp_r5_rproc_start,
> > +	.stop		= zynqmp_r5_rproc_stop,
> > +	.load		= rproc_elf_load_segments,
> > +	.parse_fw	= zynqmp_r5_parse_fw,
> > +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > +	.sanity_check	= rproc_elf_sanity_check,
> > +	.get_boot_addr	= rproc_elf_get_boot_addr,
> > +	.kick		= zynqmp_r5_rproc_kick,
> > +};
> > +
> > +/**
> > + * event_notified_idr_cb() - event notified idr callback
> > + * @id: idr id
> > + * @ptr: pointer to idr private data
> > + * @data: data passed to idr_for_each callback
> > + *
> > + * Pass notification to remoteproc virtio
> > + *
> > + * Return: 0. having return is to satisfy the idr_for_each() function
> > + *          pointer input argument requirement.
> > + **/
> > +static int event_notified_idr_cb(int id, void *ptr, void *data)
> > +{
> > +	struct rproc *rproc = data;
> > +
> > +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> > +		dev_dbg(&rproc->dev, "no message was found in idr with id:
> %d\n", id);
> > +	return 0;
> > +}
> > +
> > +/**
> > + * handle_event_notified() - remoteproc notification work function
> > + * @work: pointer to the work structure
> > + *
> > + * It checks each registered remoteproc notify IDs.
> > + */
> > +static void handle_event_notified(struct work_struct *work)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc;
> > +	struct rproc *rproc;
> > +
> > +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> > +	rproc = z_rproc->rproc;
> > +
> > +	/*
> > +	 * This is to simply raise interrupt as ACK for remote.
> > +	 *
> > +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> > +	 * mapping to remote processor.
> > +	 */
> > +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> > +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> > +
> > +	/*
> > +	 * We only use IPI for interrupt. The firmware side may or may
> > +	 * not write the notifyid when it trigger IPI.
> > +	 * And thus, we scan through all the registered notifyids.
> > +	 */
> > +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> > +}
> > +
> > +/**
> > + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> > + * @cl: mailbox client
> > + * @msg: message pointer
> > + *
> > + * It will schedule the R5 notification work.
> > + */
> > +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc;
> > +
> > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> > +	/*
> > +	 * Notification can be received with message payload.
> > +	 * To handle this, check for message contents.
> > +	 * If there are contents, copy to ipi message buffer payload location.
> > +	 *
> > +	 * If message is empty, then that means that was simple raising
> > +	 * of interrupt. no payload to process.
> > +	 *
> > +	 * Note: enqueue work regardless of msg as kick may imply that
> remote
> > +	 * is waiting for ack.
> > +	 */
> > +	if (msg) {
> > +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> > +		size_t len;
> > +
> > +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> > +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> > +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> > +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> > +		buf_msg->len = len;
> > +		memcpy(buf_msg->data, ipi_msg->data, len);
> > +	}
> > +	schedule_work(&z_rproc->mbox_work);
> > +}
> > +
> > +/**
> > + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> > + * @cl: mailbox client
> > + * @msg: pointer to the message which has been sent
> > + * @r: status of last TX - OK or error
> > + *
> > + * It will be called by the mailbox framework when the last TX has done.
> > + */
> > +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg, int
> r)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc;
> > +	struct sk_buff *skb;
> > +
> > +	if (!msg)
> > +		return;
> > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> > +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> > +	kfree_skb(skb);
> > +}
> > +
> > +/**
> > + * zynqmp_r5_setup_mbox() - Setup mailboxes
> > + *			    this is used for each individual R5 core
> > + *
> > + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> > + * @node: pointer of the device node
> > + *
> > + * Function to setup mailboxes to talk to RPU.
> > + *
> > + * Return: 0 for success, negative value for failure.
> > + */
> > +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> > +				struct device_node *node)
> > +{
> > +	struct mbox_client *mclient;
> > +
> > +	/* Setup TX mailbox channel client */
> > +	mclient			= &z_rproc->tx_mc;
> > +	mclient->dev		= z_rproc->dev;
> > +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> > +	mclient->tx_block	= false;
> > +	mclient->knows_txdone	= false;
> > +	mclient->rx_callback	= NULL;
> > +
> > +	/* Setup RX mailbox channel client */
> > +	mclient			= &z_rproc->rx_mc;
> > +	mclient->dev		= z_rproc->dev;
> > +	mclient->tx_done	= NULL;
> > +	mclient->tx_block	= false;
> > +	mclient->knows_txdone	= false;
> > +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> > +
> > +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> > +
> > +	/* Request TX and RX channels */
> > +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc-
> >tx_mc, "tx");
> > +	if (IS_ERR(z_rproc->tx_chan)) {
> > +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> > +		return PTR_ERR(z_rproc->tx_chan);
> > +	}
> > +
> > +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc-
> >rx_mc, "rx");
> > +	if (IS_ERR(z_rproc->rx_chan)) {
> > +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> > +		return PTR_ERR(z_rproc->rx_chan);
> > +	}
> > +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> > +
> > +	return 0;
> > +}
> > +
> > +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc *z_rproc)
> > +{
> > +	mbox_free_channel(z_rproc->tx_chan);
> > +	mbox_free_channel(z_rproc->rx_chan);
> > +}
> > +
> > +/**
> > + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> > + *		       this is called for each individual R5 core to
> > + *		       set up mailbox, Xilinx platform manager unique ID,
> > + *		       add to rproc core
> > + *
> > + * @pdev: domain platform device for current R5 core
> > + * @node: pointer of the device node for current R5 core
> > + * @rpu_mode: mode to configure RPU, split or lockstep
> > + *
> > + * Return: 0 for success, negative value for failure.
> > + */
> > +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device
> *pdev,
> > +					       struct device_node *node,
> > +					       enum rpu_oper_mode
> rpu_mode)
> > +{
> > +	int ret;
> > +	struct device *dev = &pdev->dev;
> > +	struct rproc *rproc_ptr;
> > +	struct zynqmp_r5_rproc *z_rproc;
> > +
> > +	/* Allocate remoteproc instance */
> > +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev),
> &zynqmp_r5_rproc_ops,
> > +				     NULL, sizeof(struct zynqmp_r5_rproc));
> > +	if (!rproc_ptr) {
> > +		ret = -ENOMEM;
> > +		goto error;
> > +	}
> > +
> > +	rproc_ptr->auto_boot = false;
> > +	z_rproc = rproc_ptr->priv;
> > +	z_rproc->rproc = rproc_ptr;
> > +	z_rproc->dev = dev;
> > +
> > +	/* Set up DMA mask */
> > +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> > +	if (ret)
> > +		goto error;
> > +
> > +	/* Get R5 power domain node */
> > +	ret = of_property_read_u32(node, "power-domain", &z_rproc-
> >pnode_id);
> > +	if (ret)
> > +		goto error;
> > +
> > +	ret = r5_set_mode(z_rproc, rpu_mode);
> > +	if (ret)
> > +		goto error;
> > +
> > +	if (of_property_read_bool(node, "mboxes")) {
> > +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> > +		if (ret)
> > +			goto error;
> > +	}
> > +
> > +	/* Add R5 remoteproc */
> > +	ret = devm_rproc_add(dev, rproc_ptr);
> > +	if (ret) {
> > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > +		goto error;
> > +	}
> > +
> > +	return z_rproc;
> > +error:
> > +	return ERR_PTR(ret);
> > +}
> > +
> > +/*
> > + * zynqmp_r5_remoteproc_probe()
> > + *
> > + * @pdev: domain platform device for R5 cluster
> > + *
> > + * called when driver is probed, for each R5 core specified in DT,
> > + * setup as needed to do remoteproc-related operations
> > + *
> > + * Return: 0 for success, negative value for failure.
> > + */
> > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > +{
> > +	int ret, core_count;
> > +	struct device *dev = &pdev->dev;
> > +	struct device_node *nc;
> > +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> > +	struct list_head *cluster; /* list to track each core's rproc */
> > +	struct zynqmp_r5_rproc *z_rproc;
> > +	struct platform_device *child_pdev;
> > +	struct list_head *pos;
> > +
> > +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode",
> &rpu_mode);
> > +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> > +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> > +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> > +			ret, rpu_mode);
> > +		return ret;
> > +	}
> > +
> > +	dev_dbg(dev, "RPU configuration: %s\n",
> > +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" :
> "split");
> > +
> > +	/*
> > +	 * if 2 RPUs provided but one is lockstep, then we have an
> > +	 * invalid configuration.
> > +	 */
> > +
> > +	core_count = of_get_available_child_count(dev->of_node);
> > +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> > +	    core_count > MAX_RPROCS)
> > +		return -EINVAL;
> > +
> > +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > +	if (!cluster)
> > +		return -ENOMEM;
> > +	INIT_LIST_HEAD(cluster);
> > +
> > +	ret = devm_of_platform_populate(dev);
> > +	if (ret) {
> > +		dev_err(dev, "devm_of_platform_populate failed, ret =
> %d\n",
> > +			ret);
> > +		return ret;
> > +	}
> > +
> > +	/* probe each individual r5 core's remoteproc-related info */
> > +	for_each_available_child_of_node(dev->of_node, nc) {
> > +		child_pdev = of_find_device_by_node(nc);
> > +		if (!child_pdev) {
> > +			dev_err(dev, "could not get R5 core platform
> device\n");
> > +			ret = -ENODEV;
> > +			goto out;
> > +		}
> > +
> > +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> > +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> > +			ret ? "Failed" : "Able",
> > +			nc);
> > +		if (IS_ERR(z_rproc)) {
> > +			ret = PTR_ERR(z_rproc);
> > +			goto out;
> > +		}
> > +		list_add_tail(&z_rproc->elem, cluster);
> > +	}
> > +	/* wire in so each core can be cleaned up at driver remove */
> > +	platform_set_drvdata(pdev, cluster);
> > +	return 0;
> > +out:
> > +	/*
> > +	 * undo core0 upon any failures on core1 in split-mode
> > +	 *
> > +	 * in zynqmp_r5_probe z_rproc is set to null
> > +	 * and ret to non-zero value if error
> > +	 */
> > +	list_for_each(pos, cluster) {
> > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > +	}
> > +	return ret;
> > +}
> > +
> > +/*
> > + * zynqmp_r5_remoteproc_remove()
> > + *
> > + * @pdev: domain platform device for R5 cluster
> > + *
> > + * When the driver is unloaded, clean up the mailboxes for each
> > + * remoteproc that was initially probed.
> > + */
> > +static int zynqmp_r5_remoteproc_remove(struct platform_device *pdev)
> > +{
> > +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> > +						 platform_get_drvdata(pdev);
> > +	struct zynqmp_r5_rproc *z_rproc = NULL;
> > +
> > +	list_for_each_safe(pos, temp, cluster) {
> > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > +		list_del(pos);
> > +	}
> > +	return 0;
> > +}
> > +
> > +/* Match table for OF platform binding */
> > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> > +	{ /* end of list */ },
> > +};
> > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > +
> > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > +	.probe = zynqmp_r5_remoteproc_probe,
> > +	.remove = zynqmp_r5_remoteproc_remove,
> > +	.driver = {
> > +		.name = "zynqmp_r5_remoteproc",
> > +		.of_match_table = zynqmp_r5_remoteproc_match,
> > +	},
> > +};
> > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > +
> > +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> > +MODULE_LICENSE("GPL v2");
> > --
> > 2.17.1
> >

^ permalink raw reply	[flat|nested] 28+ messages in thread

* RE: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
@ 2021-02-17 16:47       ` Ben Levinsky
  0 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-02-17 16:47 UTC (permalink / raw)
  To: Mathieu Poirier
  Cc: devicetree, Michal Simek, linux-remoteproc, linux-kernel,
	linux-arm-kernel

Hi Mathieu,

Please see my replies to your questions inline

Again thanks for the continued review of this set

Cheers
Ben

> -----Original Message-----
> From: Mathieu Poirier <mathieu.poirier@linaro.org>
> Sent: Tuesday, February 16, 2021 10:48 AM
> To: Ben Levinsky <BLEVINSK@xilinx.com>
> Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal Simek
> <michals@xilinx.com>
> Subject: Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc
> driver
> 
> On Sun, Jan 10, 2021 at 06:02:50PM -0800, Ben Levinsky wrote:
> > R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
> > remotproc driver, we can boot the R5 sub-system in two different
> > configurations -
> > 	* Split
> > 	* Lockstep
> >
> > The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
> > Platform Management Unit that handles the R5 configuration, memory
> access
> > and R5 lifecycle management. The interface to this manager is done in this
> > driver via zynqmp_pm_* function calls.
> >
> > Signed-off-by: Wendy Liang <wendy.liang@xilinx.com>
> > Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> > Signed-off-by: Ed Mooring <ed.mooring@xilinx.com>
> > Signed-off-by: Jason Wu <j.wu@xilinx.com>
> > Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> > ---
> > v25:
> > - reword error message for rpu configuration
> > - change char name[15] to 16 chars to have null terminated string
> >   in parse_mem_regions()
> > - reword comments in tcm_mem_alloc
> > - call xilinx platform cleanup in parse_tcm_banks() case of failure
> > - check value of mbox_send_message in all instances within this driver
> > - indentation in zynqmp_r5_remoteproc driver
> > - update style of constructing carveouts to match convention in ST
> >   remoteproc drivers. No longer iterate through memory-region property
> >   using of_count_phandle_with_args. Instead use of_phandle_iterator
> > - use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per
> feedback
> > - rework loop in zynqmp_r5_pm_request_sram as per feedback
> > - fix comment and linebreak in tcm_mem_alloc description
> > - add comments and remove extraneous devm_ioremap_wc call in
> tcm_mem_alloc
> > - in parse_tcm_banks remove check for bank not being available
> > - rework order of locals and remove else in zynqmp_r5_rproc_kick
> > - document and update logic for zynqmp_r5_parse_fw
> > - add dev_dbg output in event_notified_idr_cb if
> >   rproc_vq_interrupt(rproc, id) == IRQ_NONE
> > - add comment for handle_event_notified mbox_send_message function
> call
> > - add comment for zynqmp_r5_mb_rx_cb
> > - update zynqmp_r5_setup_mbox to match convention of mbox setup in ST
> >   remoteproc drivers
> > - change return in zynqmp_r5_setup_mbox to use PTR_ERR
> > - add zynqmp_r5_cleanup_mbox
> > - in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as arg.
> > - in zynqmp_r5_probe use return of PTR_ERR and use
> zynqmp_r5_cleanup_mbox
> > - in zynqmp_r5_remoteproc_probe update use return value of
> zynqmp_r5_probe as
> >   either zynqmp_r5_rproc* or PTR_ERR
> > - update loop that cleans up cluster and mboxes in
> zynqmp_r5_remoteproc_probe
> > - update loop that cleans up cluster and mboxes in
> zynqmp_r5_remoteproc_remove
> > ---
> >  drivers/remoteproc/Kconfig                |   8 +
> >  drivers/remoteproc/Makefile               |   1 +
> >  drivers/remoteproc/zynqmp_r5_remoteproc.c | 898
> ++++++++++++++++++++++
> >  3 files changed, 907 insertions(+)
> >  create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c
> >
> > diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> > index c6659dfea7c7..c2fe54b1d94f 100644
> > --- a/drivers/remoteproc/Kconfig
> > +++ b/drivers/remoteproc/Kconfig
> > @@ -275,6 +275,14 @@ config TI_K3_DSP_REMOTEPROC
> >  	  It's safe to say N here if you're not interested in utilizing
> >  	  the DSP slave processors.
> >
> > +config ZYNQMP_R5_REMOTEPROC
> > +	tristate "ZynqMP R5 remoteproc support"
> > +	depends on PM && ARCH_ZYNQMP
> > +	select RPMSG_VIRTIO
> > +	select ZYNQMP_IPI_MBOX
> > +	help
> > +	  Say y or m here to support ZynqMP R5 remote processors via the
> remote
> > +	  processor framework.
> >  endif # REMOTEPROC
> >
> >  endmenu
> > diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> > index 3dfa28e6c701..ef1abff654c2 100644
> > --- a/drivers/remoteproc/Makefile
> > +++ b/drivers/remoteproc/Makefile
> > @@ -33,3 +33,4 @@ obj-$(CONFIG_ST_REMOTEPROC)		+=
> st_remoteproc.o
> >  obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
> >  obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
> >  obj-$(CONFIG_TI_K3_DSP_REMOTEPROC)	+= ti_k3_dsp_remoteproc.o
> > +obj-$(CONFIG_ZYNQMP_R5_REMOTEPROC)	+= zynqmp_r5_remoteproc.o
> > diff --git a/drivers/remoteproc/zynqmp_r5_remoteproc.c
> b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > new file mode 100644
> > index 000000000000..9e228c9ecd43
> > --- /dev/null
> > +++ b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > @@ -0,0 +1,898 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Zynq R5 Remote Processor driver
> > + *
> > + * Based on origin OMAP and Zynq Remote Processor driver
> > + *
> > + */
> > +
> > +#include <linux/firmware/xlnx-zynqmp.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/kernel.h>
> > +#include <linux/list.h>
> > +#include <linux/mailbox_client.h>
> > +#include <linux/mailbox/zynqmp-ipi-message.h>
> > +#include <linux/module.h>
> > +#include <linux/of_address.h>
> > +#include <linux/of_platform.h>
> > +#include <linux/of_reserved_mem.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/remoteproc.h>
> > +#include <linux/skbuff.h>
> > +#include <linux/sysfs.h>
> > +
> > +#include "remoteproc_internal.h"
> > +
> > +#define MAX_RPROCS	2 /* Support up to 2 RPU */
> > +#define MAX_MEM_PNODES	4 /* Max power nodes for one RPU memory
> instance */
> > +
> > +#define BANK_LIST_PROP	"sram"
> > +#define DDR_LIST_PROP	"memory-region"
> > +
> > +/* IPI buffer MAX length */
> > +#define IPI_BUF_LEN_MAX	32U
> > +/* RX mailbox client buffer max length */
> > +#define RX_MBOX_CLIENT_BUF_MAX	(IPI_BUF_LEN_MAX + \
> > +				 sizeof(struct zynqmp_ipi_message))
> > +
> > +/*
> > + * Map each Xilinx on-chip SRAM  Bank address to their own respective
> > + * pm_node_id.
> > + */
> > +struct sram_addr_data {
> > +	phys_addr_t addr;
> > +	enum pm_node_id id;
> > +};
> > +
> > +#define NUM_SRAMS 4U
> > +static const struct sram_addr_data zynqmp_banks[NUM_SRAMS] = {
> > +	{0xffe00000UL, NODE_TCM_0_A},
> > +	{0xffe20000UL, NODE_TCM_0_B},
> > +	{0xffe90000UL, NODE_TCM_1_A},
> > +	{0xffeb0000UL, NODE_TCM_1_B},
> > +};
> > +
> > +/**
> > + * struct zynqmp_r5_rproc - ZynqMP R5 core structure
> > + *
> > + * @rx_mc_buf: rx mailbox client buffer to save the rx message
> > + * @tx_mc: tx mailbox client
> > + * @rx_mc: rx mailbox client
> > + * @mbox_work: mbox_work for the RPU remoteproc
> > + * @tx_mc_skbs: socket buffers for tx mailbox client
> > + * @dev: device of RPU instance
> > + * @rproc: rproc handle
> > + * @tx_chan: tx mailbox channel
> > + * @rx_chan: rx mailbox channel
> > + * @pnode_id: RPU CPU power domain id
> > + * @elem: linked list item
> > + */
> > +struct zynqmp_r5_rproc {
> > +	unsigned char rx_mc_buf[RX_MBOX_CLIENT_BUF_MAX];
> > +	struct mbox_client tx_mc;
> > +	struct mbox_client rx_mc;
> > +	struct work_struct mbox_work;
> > +	struct sk_buff_head tx_mc_skbs;
> > +	struct device *dev;
> > +	struct rproc *rproc;
> > +	struct mbox_chan *tx_chan;
> > +	struct mbox_chan *rx_chan;
> > +	u32 pnode_id;
> > +	struct list_head elem;
> > +};
> > +
> > +/*
> > + * r5_set_mode - set RPU operation mode
> > + * @z_rproc: Remote processor private data
> > + * @rpu_mode: mode specified by device tree to configure the RPU to
> > + *
> > + * set RPU operation mode
> > + *
> > + * Return: 0 for success, negative value for failure
> > + */
> > +static int r5_set_mode(struct zynqmp_r5_rproc *z_rproc,
> > +		       enum rpu_oper_mode rpu_mode)
> > +{
> > +	enum rpu_tcm_comb tcm_mode;
> > +	enum rpu_oper_mode cur_rpu_mode;
> > +	int ret;
> > +
> > +	ret = zynqmp_pm_get_rpu_mode(z_rproc->pnode_id,
> &cur_rpu_mode);
> > +	if (ret < 0)
> > +		return ret;
> > +
> > +	if (rpu_mode != cur_rpu_mode) {
> > +		ret = zynqmp_pm_set_rpu_mode(z_rproc->pnode_id,
> rpu_mode);
> > +		if (ret < 0)
> > +			return ret;
> > +	}
> > +
> > +	tcm_mode = (rpu_mode == PM_RPU_MODE_LOCKSTEP) ?
> > +		    PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> > +	return zynqmp_pm_set_tcm_config(z_rproc->pnode_id, tcm_mode);
> > +}
> > +
> > +/*
> > + * zynqmp_r5_rproc_mem_release
> > + * @rproc: single R5 core's corresponding rproc instance
> > + * @mem: mem entry to unmap
> > + *
> > + * Unmap TCM banks when powering down R5 core.
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int tcm_mem_release(struct rproc *rproc, struct rproc_mem_entry
> *mem)
> > +{
> > +	u32 pnode_id = (u64)mem->priv;
> > +
> > +	iounmap(mem->va);
> > +	return zynqmp_pm_release_node(pnode_id);
> 
> Is the opposite of that call zynqmp_pm_request_node()?  If so doing the
> power up
> and down of the TCM banks at driver probe() and remove() is highly
> inefficient.
> Consider using rproc_ops::prepare() and rproc_ops::unprepare().
> 
> > +}
> 
> Please move this just after tcm_mem_alloc()
> 
> > +
> > +/*
> > + * zynqmp_r5_rproc_start
> > + * @rproc: single R5 core's corresponding rproc instance
> > + *
> > + * Start R5 Core from designated boot address.
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > +	enum rpu_boot_mem bootmem;
> > +
> > +	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
> > +		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> > +
> > +	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
> > +		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> > +
> > +	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
> > +				      bootmem,
> ZYNQMP_PM_REQUEST_ACK_NO);
> > +}
> > +
> > +/*
> > + * zynqmp_r5_rproc_stop
> > + * @rproc: single R5 core's corresponding rproc instance
> > + *
> > + * Power down  R5 Core.
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > +
> > +	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
> > +				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > +}
> > +
> > +/*
> > + * zynqmp_r5_rproc_mem_alloc
> > + * @rproc: single R5 core's corresponding rproc instance
> > + * @mem: mem entry to map
> > + *
> > + * Callback to map va for memory-region's carveout.
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
> > +				     struct rproc_mem_entry *mem)
> > +{
> > +	void *va;
> > +
> > +	va = ioremap_wc(mem->dma, mem->len);
> > +	if (IS_ERR_OR_NULL(va))
> > +		return -ENOMEM;
> > +
> > +	mem->va = va;
> > +
> > +	return 0;
> > +}
> > +
> > +/*
> > + * zynqmp_r5_rproc_mem_release
> > + * @rproc: single R5 core's corresponding rproc instance
> > + * @mem: mem entry to unmap
> > + *
> > + * Unmap memory-region carveout
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
> > +				       struct rproc_mem_entry *mem)
> > +{
> > +	iounmap(mem->va);
> > +	return 0;
> > +}
> > +
> > +/*
> > + * parse_mem_regions
> > + * @rproc: single R5 core's corresponding rproc instance
> > + *
> > + * Construct rproc mem carveouts from carveout provided in
> > + * memory-region property
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int parse_mem_regions(struct rproc *rproc)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > +	struct device_node *np = z_rproc->dev->of_node;
> > +	struct device *dev = &rproc->dev;
> > +	struct of_phandle_iterator it;
> > +	struct rproc_mem_entry *mem;
> > +	struct reserved_mem *rmem;
> > +	int index = 0;
> > +
> > +	/* Register associated reserved memory regions */
> > +	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
> > +	while (of_phandle_iterator_next(&it) == 0) {
> > +		rmem = of_reserved_mem_lookup(it.node);
> > +		if (!rmem) {
> > +			dev_err(dev, "unable to acquire %s\n",
> DDR_LIST_PROP);
> > +			return -EINVAL;
> > +		}
> > +
> > +		if (strstr(it.node->name, "vdev0buffer")) {
> > +			mem = rproc_of_resm_mem_entry_init(dev, index,
> > +							   rmem->size,
> > +							   rmem->base,
> > +							   "vdev0buffer");
> > +		} else {
> > +			/*
> > +			 * The ensuing carveout is either for vring, which has
> > +			 * device tree node name of length 15 characters, or
> > +			 * firmware mem, where name is not used by
> remoteproc
> > +			 *  core later on. So default to vring length of 15.
> 
> s/"*  core"/"* core"
> 
> > +			 *
> > +			 * Extra char for null-terminated string.
> > +			 */
> > +			char name[16];
> > +
> > +			if (strstr(it.node->name, "vdev0vring")) {
> > +				int vring_id;
> > +
> > +				/*
> > +				 * Expect "rpuXvdev0vringX" as documented
> > +				 * in xilinx remoteproc device tree binding
> > +				 */
> > +				if (strlen(it.node->name) < 15) {
> > +					dev_err(dev, "%pOF is less than 15
> chars",
> > +						it.node);
> > +					return -EINVAL;
> > +				}
> > +
> > +				/*
> > +				 * can be 1 of multiple vring IDs per IPC
> channel
> > +				 * e.g. 'vdev0vring0' and 'vdev0vring1'
> > +				 *
> > +				 * Here get vring ID from 15th char
> > +				 */
> > +				vring_id = it.node->name[14] - '0';
> > +				snprintf(name, sizeof(name), "vdev0vring%d",
> vring_id);
> 
> Function strstr() returns a pointer to the beginning of the string being
> searched for.  As such you should be able to do something like:
> 
>                         name = strstr(it.node->name, "vdev0vring");
>                         if (name) {
>                                 ...
> 
> and simply use @name when calling rproc_mem_entry_init().
> 
> > +			} else {
> > +				strncpy(name, it.node->name, 16);
> > +			}
> > +
> > +			mem = rproc_mem_entry_init(dev, NULL,
> > +						   (dma_addr_t)rmem->base,
> > +						   rmem->size, rmem->base,
> > +
> zynqmp_r5_rproc_mem_alloc,
> > +
> zynqmp_r5_rproc_mem_release,
> > +						   name);
> > +		}
> > +
> > +		if (!mem)
> > +			return -ENOMEM;
> > +
> > +		rproc_add_carveout(rproc, mem);
> > +		index++;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +/*
> > + * zynqmp_r5_pm_request_tcm
> > + * @addr: base address of mem provided in R5 core's sram property.
> > + *
> > + * Given sram base address, determine its corresponding Xilinx
> > + * Platform Management ID and then request access to this node
> > + * so that it can be power up.
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
> > +{
> > +	int i;
> > +
> > +	for (i = 0; i < NUM_SRAMS; i++) {
> > +		if (zynqmp_banks[i].addr != addr)
> > +			continue;
> > +
> > +		return zynqmp_pm_request_node(zynqmp_banks[i].id,
> > +					      ZYNQMP_PM_CAPABILITY_ACCESS,
> 0,
> > +
> ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > +	}
> > +
> > +	return -EINVAL;
> > +}
> > +
> > +/*
> > + * tcm_mem_alloc
> > + * @rproc: single R5 core's corresponding rproc instance
> > + * @mem: mem entry to initialize the va and da fields of
> > + *
> > + * Given TCM bank entry, this callback will set device address for R5
> > + * running on TCM and also setup virtual address for TCM bank
> > + * remoteproc carveout.
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int tcm_mem_alloc(struct rproc *rproc,
> > +			 struct rproc_mem_entry *mem)
> > +{
> > +	void *va;
> > +	struct device *dev = rproc->dev.parent;
> > +
> > +	va = ioremap_wc(mem->dma, mem->len);
> > +	if (IS_ERR_OR_NULL(va))
> > +		return -ENOMEM;
> > +
> > +	/* Update memory entry va */
> > +	mem->va = va;
> > +
> > +	/*
> > +	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> > +	 * while on the Linux side they are at 0xffexxxxx.
> > +	 *
> > +	 * Zero out the high 12 bits of the address. This will give
> > +	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> > +	 */
> > +	mem->da &= 0x000fffff;
> > +
> > +	/*
> > +	 * TCM Banks 1A and 1B still have to be translated.
> > +	 *
> > +	 * Below handle these two banks' absolute addresses (0xffe90000 and
> > +	 * 0xffeb0000) and convert to the expected relative addresses
> > +	 * (0x0 and 0x20000).
> > +	 */
> > +	if (mem->da == 0x90000 || mem->da == 0xB0000)
> > +		mem->da -= 0x90000;
> > +
> > +	/* if translated TCM bank address is not valid report error */
> > +	if (mem->da != 0x0 && mem->da != 0x20000) {
> > +		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
> > +		return -EINVAL;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +/*
> > + * parse_tcm_banks()
> > + * @rproc: single R5 core's corresponding rproc instance
> > + *
> > + * Given R5 node in remoteproc instance
> > + * allocate remoteproc carveout for TCM memory
> > + * needed for firmware to be loaded
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int parse_tcm_banks(struct rproc *rproc)
> > +{
> > +	int i, num_banks;
> > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > +	struct device *dev = &rproc->dev;
> > +	struct device_node *r5_node = z_rproc->dev->of_node;
> > +
> > +	/* go through TCM banks for r5 node */
> > +	num_banks = of_count_phandle_with_args(r5_node,
> BANK_LIST_PROP, NULL);
> > +	if (num_banks <= 0) {
> > +		dev_err(dev, "need to specify TCM banks\n");
> > +		return -EINVAL;
> > +	}
> > +	for (i = 0; i < num_banks; i++) {
> > +		struct resource rsc;
> > +		resource_size_t size;
> > +		struct device_node *dt_node;
> > +		struct rproc_mem_entry *mem;
> > +		int ret;
> > +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> > +
> > +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> > +		if (!dt_node)
> > +			return -EINVAL;
> > +
> > +		ret = of_address_to_resource(dt_node, 0, &rsc);
> > +		if (ret < 0) {
> > +			of_node_put(dt_node);
> > +			return ret;
> > +		}
> > +		of_node_put(dt_node);
> > +
> > +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> > +		if (ret < 0)
> > +			return ret;
> > +
> > +		/* add carveout */
> > +		size = resource_size(&rsc);
> > +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> > +					   (size_t)size, rsc.start,
> > +					   tcm_mem_alloc,
> > +					   tcm_mem_release,
> > +					   rsc.name);
> > +		if (!mem) {
> > +			ret = zynqmp_pm_release_node(pnode_id);
> 
> Where is @pnode_id initialised?

Good catch. Previously this was value set in an earlier iteration. I will fix this as follows:
- release the R5 cluster by instead passing z_rproc->pnode_id
- release the TCM bank via call to release_sram

This will also remove the variable pnode_id.

> 
> > +			if (ret)
> > +				dev_warn(dev,
> > +					 "fail to release node: %x ret: %x\n",
> > +					 pnode_id, ret);
> > +			return -ENOMEM;
> > +		}
> > +
> > +		mem->priv = (void *)(u64)pnode_id;
> 
> 
> And here too - how does this work when you test things out on your side?
> 
> Please note that I will need a reply to these questions before I review another
> set.
> 
Here I will update as follows:
- set mem->priv to value from zynqmp_banks lookup table that contains power node IDs


> > +		rproc_add_carveout(rproc, mem);
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +/*
> > + * zynqmp_r5_parse_fw()
> > + * @rproc: single R5 core's corresponding rproc instance
> > + * @fw: ptr to firmware to be loaded onto r5 core
> > + *
> > + * When loading firmware, ensure the necessary carveouts are in
> remoteproc
> > + *
> > + * return 0 on success, otherwise non-zero value on failure
> > + */
> > +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware
> *fw)
> > +{
> > +	int ret;
> > +
> > +	ret = parse_tcm_banks(rproc);
> > +	if (ret)
> > +		return ret;
> > +
> > +	ret = parse_mem_regions(rproc);
> > +	if (ret)
> > +		return ret;
> > +
> > +	ret = rproc_elf_load_rsc_table(rproc, fw);
> > +	if (ret == -EINVAL) {
> > +		/*
> > +		 * resource table only required for IPC.
> > +		 * if not present, this is not necessarily an error;
> > +		 * for example, loading r5 hello world application
> > +		 * so simply inform user and keep going.
> > +		 */
> > +		dev_info(&rproc->dev, "no resource table found.\n");
> > +		ret = 0;
> > +	}
> > +	return ret;
> > +}
> > +
> > +/*
> > + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> > + * @rproc: r5 core's corresponding rproc structure
> > + * @vqid: virtqueue ID
> > + */
> > +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > +	struct device *dev = rproc->dev.parent;
> > +	struct zynqmp_ipi_message *mb_msg = NULL;
> > +	unsigned int skb_len;
> > +	struct sk_buff *skb;
> > +	int ret;
> > +
> > +	/* If vqid is negative, do not pass the vqid to
> > +	 * mailbox as vqid is supposed to be 0 or positive.
> > +	 * It also gives a way to just kick instead but
> > +	 * not use the IPI buffer.
> > +	 *
> > +	 * For now use negative vqid to assume no message will be
> > +	 * passed with IPI buffer, but just raise interrupt.
> > +	 * This will be faster as it doesn't need to copy the
> > +	 * message to the ZynqMP/Versal IPI message buffer.
> > +	 */
> > +	if (vqid >= 0) {
> > +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> > +		skb = alloc_skb(skb_len, GFP_KERNEL);
> > +		if (!skb)
> > +			return;
> > +
> > +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb,
> skb_len);
> > +		mb_msg->len = sizeof(vqid);
> > +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> > +
> > +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> > +	}
> > +
> > +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> > +	if (ret < 0) {
> > +		dev_warn(dev, "Failed to kick remote.\n");
> > +		if (mb_msg) {
> > +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> > +			kfree_skb(skb);
> > +		}
> > +	}
> > +}
> > +
> 
> I am done reviewing this set.
> 
> Thanks,
> Mathieu
> 
> > +static struct rproc_ops zynqmp_r5_rproc_ops = {
> > +	.start		= zynqmp_r5_rproc_start,
> > +	.stop		= zynqmp_r5_rproc_stop,
> > +	.load		= rproc_elf_load_segments,
> > +	.parse_fw	= zynqmp_r5_parse_fw,
> > +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > +	.sanity_check	= rproc_elf_sanity_check,
> > +	.get_boot_addr	= rproc_elf_get_boot_addr,
> > +	.kick		= zynqmp_r5_rproc_kick,
> > +};
> > +
> > +/**
> > + * event_notified_idr_cb() - event notified idr callback
> > + * @id: idr id
> > + * @ptr: pointer to idr private data
> > + * @data: data passed to idr_for_each callback
> > + *
> > + * Pass notification to remoteproc virtio
> > + *
> > + * Return: 0. having return is to satisfy the idr_for_each() function
> > + *          pointer input argument requirement.
> > + **/
> > +static int event_notified_idr_cb(int id, void *ptr, void *data)
> > +{
> > +	struct rproc *rproc = data;
> > +
> > +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> > +		dev_dbg(&rproc->dev, "no message was found in idr with id:
> %d\n", id);
> > +	return 0;
> > +}
> > +
> > +/**
> > + * handle_event_notified() - remoteproc notification work function
> > + * @work: pointer to the work structure
> > + *
> > + * It checks each registered remoteproc notify IDs.
> > + */
> > +static void handle_event_notified(struct work_struct *work)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc;
> > +	struct rproc *rproc;
> > +
> > +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> > +	rproc = z_rproc->rproc;
> > +
> > +	/*
> > +	 * This is to simply raise interrupt as ACK for remote.
> > +	 *
> > +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> > +	 * mapping to remote processor.
> > +	 */
> > +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> > +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> > +
> > +	/*
> > +	 * We only use IPI for interrupt. The firmware side may or may
> > +	 * not write the notifyid when it trigger IPI.
> > +	 * And thus, we scan through all the registered notifyids.
> > +	 */
> > +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> > +}
> > +
> > +/**
> > + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> > + * @cl: mailbox client
> > + * @msg: message pointer
> > + *
> > + * It will schedule the R5 notification work.
> > + */
> > +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc;
> > +
> > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> > +	/*
> > +	 * Notification can be received with message payload.
> > +	 * To handle this, check for message contents.
> > +	 * If there are contents, copy to ipi message buffer payload location.
> > +	 *
> > +	 * If message is empty, then that means that was simple raising
> > +	 * of interrupt. no payload to process.
> > +	 *
> > +	 * Note: enqueue work regardless of msg as kick may imply that
> remote
> > +	 * is waiting for ack.
> > +	 */
> > +	if (msg) {
> > +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> > +		size_t len;
> > +
> > +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> > +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> > +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> > +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> > +		buf_msg->len = len;
> > +		memcpy(buf_msg->data, ipi_msg->data, len);
> > +	}
> > +	schedule_work(&z_rproc->mbox_work);
> > +}
> > +
> > +/**
> > + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> > + * @cl: mailbox client
> > + * @msg: pointer to the message which has been sent
> > + * @r: status of last TX - OK or error
> > + *
> > + * It will be called by the mailbox framework when the last TX has done.
> > + */
> > +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg, int
> r)
> > +{
> > +	struct zynqmp_r5_rproc *z_rproc;
> > +	struct sk_buff *skb;
> > +
> > +	if (!msg)
> > +		return;
> > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> > +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> > +	kfree_skb(skb);
> > +}
> > +
> > +/**
> > + * zynqmp_r5_setup_mbox() - Setup mailboxes
> > + *			    this is used for each individual R5 core
> > + *
> > + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> > + * @node: pointer of the device node
> > + *
> > + * Function to setup mailboxes to talk to RPU.
> > + *
> > + * Return: 0 for success, negative value for failure.
> > + */
> > +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> > +				struct device_node *node)
> > +{
> > +	struct mbox_client *mclient;
> > +
> > +	/* Setup TX mailbox channel client */
> > +	mclient			= &z_rproc->tx_mc;
> > +	mclient->dev		= z_rproc->dev;
> > +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> > +	mclient->tx_block	= false;
> > +	mclient->knows_txdone	= false;
> > +	mclient->rx_callback	= NULL;
> > +
> > +	/* Setup RX mailbox channel client */
> > +	mclient			= &z_rproc->rx_mc;
> > +	mclient->dev		= z_rproc->dev;
> > +	mclient->tx_done	= NULL;
> > +	mclient->tx_block	= false;
> > +	mclient->knows_txdone	= false;
> > +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> > +
> > +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> > +
> > +	/* Request TX and RX channels */
> > +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc-
> >tx_mc, "tx");
> > +	if (IS_ERR(z_rproc->tx_chan)) {
> > +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> > +		return PTR_ERR(z_rproc->tx_chan);
> > +	}
> > +
> > +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc-
> >rx_mc, "rx");
> > +	if (IS_ERR(z_rproc->rx_chan)) {
> > +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> > +		return PTR_ERR(z_rproc->rx_chan);
> > +	}
> > +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> > +
> > +	return 0;
> > +}
> > +
> > +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc *z_rproc)
> > +{
> > +	mbox_free_channel(z_rproc->tx_chan);
> > +	mbox_free_channel(z_rproc->rx_chan);
> > +}
> > +
> > +/**
> > + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> > + *		       this is called for each individual R5 core to
> > + *		       set up mailbox, Xilinx platform manager unique ID,
> > + *		       add to rproc core
> > + *
> > + * @pdev: domain platform device for current R5 core
> > + * @node: pointer of the device node for current R5 core
> > + * @rpu_mode: mode to configure RPU, split or lockstep
> > + *
> > + * Return: 0 for success, negative value for failure.
> > + */
> > +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device
> *pdev,
> > +					       struct device_node *node,
> > +					       enum rpu_oper_mode
> rpu_mode)
> > +{
> > +	int ret;
> > +	struct device *dev = &pdev->dev;
> > +	struct rproc *rproc_ptr;
> > +	struct zynqmp_r5_rproc *z_rproc;
> > +
> > +	/* Allocate remoteproc instance */
> > +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev),
> &zynqmp_r5_rproc_ops,
> > +				     NULL, sizeof(struct zynqmp_r5_rproc));
> > +	if (!rproc_ptr) {
> > +		ret = -ENOMEM;
> > +		goto error;
> > +	}
> > +
> > +	rproc_ptr->auto_boot = false;
> > +	z_rproc = rproc_ptr->priv;
> > +	z_rproc->rproc = rproc_ptr;
> > +	z_rproc->dev = dev;
> > +
> > +	/* Set up DMA mask */
> > +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> > +	if (ret)
> > +		goto error;
> > +
> > +	/* Get R5 power domain node */
> > +	ret = of_property_read_u32(node, "power-domain", &z_rproc-
> >pnode_id);
> > +	if (ret)
> > +		goto error;
> > +
> > +	ret = r5_set_mode(z_rproc, rpu_mode);
> > +	if (ret)
> > +		goto error;
> > +
> > +	if (of_property_read_bool(node, "mboxes")) {
> > +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> > +		if (ret)
> > +			goto error;
> > +	}
> > +
> > +	/* Add R5 remoteproc */
> > +	ret = devm_rproc_add(dev, rproc_ptr);
> > +	if (ret) {
> > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > +		goto error;
> > +	}
> > +
> > +	return z_rproc;
> > +error:
> > +	return ERR_PTR(ret);
> > +}
> > +
> > +/*
> > + * zynqmp_r5_remoteproc_probe()
> > + *
> > + * @pdev: domain platform device for R5 cluster
> > + *
> > + * called when driver is probed, for each R5 core specified in DT,
> > + * setup as needed to do remoteproc-related operations
> > + *
> > + * Return: 0 for success, negative value for failure.
> > + */
> > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > +{
> > +	int ret, core_count;
> > +	struct device *dev = &pdev->dev;
> > +	struct device_node *nc;
> > +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> > +	struct list_head *cluster; /* list to track each core's rproc */
> > +	struct zynqmp_r5_rproc *z_rproc;
> > +	struct platform_device *child_pdev;
> > +	struct list_head *pos;
> > +
> > +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode",
> &rpu_mode);
> > +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> > +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> > +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> > +			ret, rpu_mode);
> > +		return ret;
> > +	}
> > +
> > +	dev_dbg(dev, "RPU configuration: %s\n",
> > +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" :
> "split");
> > +
> > +	/*
> > +	 * if 2 RPUs provided but one is lockstep, then we have an
> > +	 * invalid configuration.
> > +	 */
> > +
> > +	core_count = of_get_available_child_count(dev->of_node);
> > +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> > +	    core_count > MAX_RPROCS)
> > +		return -EINVAL;
> > +
> > +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > +	if (!cluster)
> > +		return -ENOMEM;
> > +	INIT_LIST_HEAD(cluster);
> > +
> > +	ret = devm_of_platform_populate(dev);
> > +	if (ret) {
> > +		dev_err(dev, "devm_of_platform_populate failed, ret =
> %d\n",
> > +			ret);
> > +		return ret;
> > +	}
> > +
> > +	/* probe each individual r5 core's remoteproc-related info */
> > +	for_each_available_child_of_node(dev->of_node, nc) {
> > +		child_pdev = of_find_device_by_node(nc);
> > +		if (!child_pdev) {
> > +			dev_err(dev, "could not get R5 core platform
> device\n");
> > +			ret = -ENODEV;
> > +			goto out;
> > +		}
> > +
> > +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> > +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> > +			ret ? "Failed" : "Able",
> > +			nc);
> > +		if (IS_ERR(z_rproc)) {
> > +			ret = PTR_ERR(z_rproc);
> > +			goto out;
> > +		}
> > +		list_add_tail(&z_rproc->elem, cluster);
> > +	}
> > +	/* wire in so each core can be cleaned up at driver remove */
> > +	platform_set_drvdata(pdev, cluster);
> > +	return 0;
> > +out:
> > +	/*
> > +	 * undo core0 upon any failures on core1 in split-mode
> > +	 *
> > +	 * in zynqmp_r5_probe z_rproc is set to null
> > +	 * and ret to non-zero value if error
> > +	 */
> > +	list_for_each(pos, cluster) {
> > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > +	}
> > +	return ret;
> > +}
> > +
> > +/*
> > + * zynqmp_r5_remoteproc_remove()
> > + *
> > + * @pdev: domain platform device for R5 cluster
> > + *
> > + * When the driver is unloaded, clean up the mailboxes for each
> > + * remoteproc that was initially probed.
> > + */
> > +static int zynqmp_r5_remoteproc_remove(struct platform_device *pdev)
> > +{
> > +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> > +						 platform_get_drvdata(pdev);
> > +	struct zynqmp_r5_rproc *z_rproc = NULL;
> > +
> > +	list_for_each_safe(pos, temp, cluster) {
> > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > +		list_del(pos);
> > +	}
> > +	return 0;
> > +}
> > +
> > +/* Match table for OF platform binding */
> > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> > +	{ /* end of list */ },
> > +};
> > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > +
> > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > +	.probe = zynqmp_r5_remoteproc_probe,
> > +	.remove = zynqmp_r5_remoteproc_remove,
> > +	.driver = {
> > +		.name = "zynqmp_r5_remoteproc",
> > +		.of_match_table = zynqmp_r5_remoteproc_match,
> > +	},
> > +};
> > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > +
> > +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> > +MODULE_LICENSE("GPL v2");
> > --
> > 2.17.1
> >

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 28+ messages in thread

* RE: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
  2021-02-17 16:47       ` Ben Levinsky
@ 2021-02-17 19:11         ` Ben Levinsky
  -1 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-02-17 19:11 UTC (permalink / raw)
  To: Mathieu Poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	Michal Simek

Hi Mathieu,

I worked on this further and had 1 more comment on the prepare/unprepare for TCM banks below

> -----Original Message-----
> From: Ben Levinsky
> Sent: Wednesday, February 17, 2021 8:47 AM
> To: Mathieu Poirier <mathieu.poirier@linaro.org>
> Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal Simek
> <michals@xilinx.com>
> Subject: RE: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc
> driver
> 
> Hi Mathieu,
> 
> Please see my replies to your questions inline
> 
> Again thanks for the continued review of this set
> 
> Cheers
> Ben
> 
> > -----Original Message-----
> > From: Mathieu Poirier <mathieu.poirier@linaro.org>
> > Sent: Tuesday, February 16, 2021 10:48 AM
> > To: Ben Levinsky <BLEVINSK@xilinx.com>
> > Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal Simek
> > <michals@xilinx.com>
> > Subject: Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc
> > driver
> >
> > On Sun, Jan 10, 2021 at 06:02:50PM -0800, Ben Levinsky wrote:
> > > R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
> > > remotproc driver, we can boot the R5 sub-system in two different
> > > configurations -
> > > 	* Split
> > > 	* Lockstep
> > >
> > > The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
> > > Platform Management Unit that handles the R5 configuration, memory
> > access
> > > and R5 lifecycle management. The interface to this manager is done in this
> > > driver via zynqmp_pm_* function calls.
> > >
> > > Signed-off-by: Wendy Liang <wendy.liang@xilinx.com>
> > > Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> > > Signed-off-by: Ed Mooring <ed.mooring@xilinx.com>
> > > Signed-off-by: Jason Wu <j.wu@xilinx.com>
> > > Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> > > ---
> > > v25:
> > > - reword error message for rpu configuration
> > > - change char name[15] to 16 chars to have null terminated string
> > >   in parse_mem_regions()
> > > - reword comments in tcm_mem_alloc
> > > - call xilinx platform cleanup in parse_tcm_banks() case of failure
> > > - check value of mbox_send_message in all instances within this driver
> > > - indentation in zynqmp_r5_remoteproc driver
> > > - update style of constructing carveouts to match convention in ST
> > >   remoteproc drivers. No longer iterate through memory-region property
> > >   using of_count_phandle_with_args. Instead use of_phandle_iterator
> > > - use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per
> > feedback
> > > - rework loop in zynqmp_r5_pm_request_sram as per feedback
> > > - fix comment and linebreak in tcm_mem_alloc description
> > > - add comments and remove extraneous devm_ioremap_wc call in
> > tcm_mem_alloc
> > > - in parse_tcm_banks remove check for bank not being available
> > > - rework order of locals and remove else in zynqmp_r5_rproc_kick
> > > - document and update logic for zynqmp_r5_parse_fw
> > > - add dev_dbg output in event_notified_idr_cb if
> > >   rproc_vq_interrupt(rproc, id) == IRQ_NONE
> > > - add comment for handle_event_notified mbox_send_message function
> > call
> > > - add comment for zynqmp_r5_mb_rx_cb
> > > - update zynqmp_r5_setup_mbox to match convention of mbox setup in
> ST
> > >   remoteproc drivers
> > > - change return in zynqmp_r5_setup_mbox to use PTR_ERR
> > > - add zynqmp_r5_cleanup_mbox
> > > - in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as
> arg.
> > > - in zynqmp_r5_probe use return of PTR_ERR and use
> > zynqmp_r5_cleanup_mbox
> > > - in zynqmp_r5_remoteproc_probe update use return value of
> > zynqmp_r5_probe as
> > >   either zynqmp_r5_rproc* or PTR_ERR
> > > - update loop that cleans up cluster and mboxes in
> > zynqmp_r5_remoteproc_probe
> > > - update loop that cleans up cluster and mboxes in
> > zynqmp_r5_remoteproc_remove
> > > ---
> > >  drivers/remoteproc/Kconfig                |   8 +
> > >  drivers/remoteproc/Makefile               |   1 +
> > >  drivers/remoteproc/zynqmp_r5_remoteproc.c | 898
> > ++++++++++++++++++++++
> > >  3 files changed, 907 insertions(+)
> > >  create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c
> > >
> > > diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> > > index c6659dfea7c7..c2fe54b1d94f 100644
> > > --- a/drivers/remoteproc/Kconfig
> > > +++ b/drivers/remoteproc/Kconfig
> > > @@ -275,6 +275,14 @@ config TI_K3_DSP_REMOTEPROC
> > >  	  It's safe to say N here if you're not interested in utilizing
> > >  	  the DSP slave processors.
> > >
> > > +config ZYNQMP_R5_REMOTEPROC
> > > +	tristate "ZynqMP R5 remoteproc support"
> > > +	depends on PM && ARCH_ZYNQMP
> > > +	select RPMSG_VIRTIO
> > > +	select ZYNQMP_IPI_MBOX
> > > +	help
> > > +	  Say y or m here to support ZynqMP R5 remote processors via the
> > remote
> > > +	  processor framework.
> > >  endif # REMOTEPROC
> > >
> > >  endmenu
> > > diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> > > index 3dfa28e6c701..ef1abff654c2 100644
> > > --- a/drivers/remoteproc/Makefile
> > > +++ b/drivers/remoteproc/Makefile
> > > @@ -33,3 +33,4 @@ obj-$(CONFIG_ST_REMOTEPROC)		+=
> > st_remoteproc.o
> > >  obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
> > >  obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
> > >  obj-$(CONFIG_TI_K3_DSP_REMOTEPROC)	+= ti_k3_dsp_remoteproc.o
> > > +obj-$(CONFIG_ZYNQMP_R5_REMOTEPROC)	+= zynqmp_r5_remoteproc.o
> > > diff --git a/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > > new file mode 100644
> > > index 000000000000..9e228c9ecd43
> > > --- /dev/null
> > > +++ b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > > @@ -0,0 +1,898 @@
> > > +// SPDX-License-Identifier: GPL-2.0
> > > +/*
> > > + * Zynq R5 Remote Processor driver
> > > + *
> > > + * Based on origin OMAP and Zynq Remote Processor driver
> > > + *
> > > + */
> > > +
> > > +#include <linux/firmware/xlnx-zynqmp.h>
> > > +#include <linux/interrupt.h>
> > > +#include <linux/kernel.h>
> > > +#include <linux/list.h>
> > > +#include <linux/mailbox_client.h>
> > > +#include <linux/mailbox/zynqmp-ipi-message.h>
> > > +#include <linux/module.h>
> > > +#include <linux/of_address.h>
> > > +#include <linux/of_platform.h>
> > > +#include <linux/of_reserved_mem.h>
> > > +#include <linux/platform_device.h>
> > > +#include <linux/remoteproc.h>
> > > +#include <linux/skbuff.h>
> > > +#include <linux/sysfs.h>
> > > +
> > > +#include "remoteproc_internal.h"
> > > +
> > > +#define MAX_RPROCS	2 /* Support up to 2 RPU */
> > > +#define MAX_MEM_PNODES	4 /* Max power nodes for one RPU
> memory
> > instance */
> > > +
> > > +#define BANK_LIST_PROP	"sram"
> > > +#define DDR_LIST_PROP	"memory-region"
> > > +
> > > +/* IPI buffer MAX length */
> > > +#define IPI_BUF_LEN_MAX	32U
> > > +/* RX mailbox client buffer max length */
> > > +#define RX_MBOX_CLIENT_BUF_MAX	(IPI_BUF_LEN_MAX + \
> > > +				 sizeof(struct zynqmp_ipi_message))
> > > +
> > > +/*
> > > + * Map each Xilinx on-chip SRAM  Bank address to their own respective
> > > + * pm_node_id.
> > > + */
> > > +struct sram_addr_data {
> > > +	phys_addr_t addr;
> > > +	enum pm_node_id id;
> > > +};
> > > +
> > > +#define NUM_SRAMS 4U
> > > +static const struct sram_addr_data zynqmp_banks[NUM_SRAMS] = {
> > > +	{0xffe00000UL, NODE_TCM_0_A},
> > > +	{0xffe20000UL, NODE_TCM_0_B},
> > > +	{0xffe90000UL, NODE_TCM_1_A},
> > > +	{0xffeb0000UL, NODE_TCM_1_B},
> > > +};
> > > +
> > > +/**
> > > + * struct zynqmp_r5_rproc - ZynqMP R5 core structure
> > > + *
> > > + * @rx_mc_buf: rx mailbox client buffer to save the rx message
> > > + * @tx_mc: tx mailbox client
> > > + * @rx_mc: rx mailbox client
> > > + * @mbox_work: mbox_work for the RPU remoteproc
> > > + * @tx_mc_skbs: socket buffers for tx mailbox client
> > > + * @dev: device of RPU instance
> > > + * @rproc: rproc handle
> > > + * @tx_chan: tx mailbox channel
> > > + * @rx_chan: rx mailbox channel
> > > + * @pnode_id: RPU CPU power domain id
> > > + * @elem: linked list item
> > > + */
> > > +struct zynqmp_r5_rproc {
> > > +	unsigned char rx_mc_buf[RX_MBOX_CLIENT_BUF_MAX];
> > > +	struct mbox_client tx_mc;
> > > +	struct mbox_client rx_mc;
> > > +	struct work_struct mbox_work;
> > > +	struct sk_buff_head tx_mc_skbs;
> > > +	struct device *dev;
> > > +	struct rproc *rproc;
> > > +	struct mbox_chan *tx_chan;
> > > +	struct mbox_chan *rx_chan;
> > > +	u32 pnode_id;
> > > +	struct list_head elem;
> > > +};
> > > +
> > > +/*
> > > + * r5_set_mode - set RPU operation mode
> > > + * @z_rproc: Remote processor private data
> > > + * @rpu_mode: mode specified by device tree to configure the RPU to
> > > + *
> > > + * set RPU operation mode
> > > + *
> > > + * Return: 0 for success, negative value for failure
> > > + */
> > > +static int r5_set_mode(struct zynqmp_r5_rproc *z_rproc,
> > > +		       enum rpu_oper_mode rpu_mode)
> > > +{
> > > +	enum rpu_tcm_comb tcm_mode;
> > > +	enum rpu_oper_mode cur_rpu_mode;
> > > +	int ret;
> > > +
> > > +	ret = zynqmp_pm_get_rpu_mode(z_rproc->pnode_id,
> > &cur_rpu_mode);
> > > +	if (ret < 0)
> > > +		return ret;
> > > +
> > > +	if (rpu_mode != cur_rpu_mode) {
> > > +		ret = zynqmp_pm_set_rpu_mode(z_rproc->pnode_id,
> > rpu_mode);
> > > +		if (ret < 0)
> > > +			return ret;
> > > +	}
> > > +
> > > +	tcm_mode = (rpu_mode == PM_RPU_MODE_LOCKSTEP) ?
> > > +		    PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> > > +	return zynqmp_pm_set_tcm_config(z_rproc->pnode_id, tcm_mode);
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_rproc_mem_release
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + * @mem: mem entry to unmap
> > > + *
> > > + * Unmap TCM banks when powering down R5 core.
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int tcm_mem_release(struct rproc *rproc, struct rproc_mem_entry
> > *mem)
> > > +{
> > > +	u32 pnode_id = (u64)mem->priv;
> > > +
> > > +	iounmap(mem->va);
> > > +	return zynqmp_pm_release_node(pnode_id);
> >
> > Is the opposite of that call zynqmp_pm_request_node()?  If so doing the
> > power up
> > and down of the TCM banks at driver probe() and remove() is highly
> > inefficient.
> > Consider using rproc_ops::prepare() and rproc_ops::unprepare().
> >
If I understand your comment correctly this is about handling the request_node() and release_node() calls. While its true that these two functions can be moved to prepare and unprepared I think these are still occurring not at probe nor remove.

For request_node() of the TCM bank this is occurring in parse_fw() in remoteproc_core as in the zynqmp_r5 driver shows below:

	.parse_fw	= zynqmp_r5_parse_fw,
	...
	Zynqmp_parse_fw calls parse_tcm_banks

	static int parse_tcm_banks(struct rproc *rproc){
		...
		ret = zynqmp_r5_pm_request_sram(rsc.start);
		...
 		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
					   (size_t)size, rsc.start,
					   tcm_mem_alloc,
					   tcm_mem_release,
					   rsc.name);

For release_node() this is occurring not at remove but instead is managed by remoteproc_core and tied to the lifecyle of the remoteproc carveout since release_node() is within the function tcm_mem_release() and therefore rproc_resource_cleanup() will end up triggering 
TCM bank cleanup.


Still it is understandable to prefer prepare/unprepared to contain the 2 lines.


If you still prefer prepare/unprepared is the following implementation ok?

prepare: 
	Just move request_node() line here	
unprepare: 
	similarly this is only moving the 1 line of release_node().

If this suggestion satisfies what you are saying I will do so, otherwise please correct my understanding


Cheers
Ben 



> > > +}
> >
> > Please move this just after tcm_mem_alloc()
> >
> > > +
> > > +/*
> > > + * zynqmp_r5_rproc_start
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + *
> > > + * Start R5 Core from designated boot address.
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > +	enum rpu_boot_mem bootmem;
> > > +
> > > +	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
> > > +		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> > > +
> > > +	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
> > > +		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> > > +
> > > +	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
> > > +				      bootmem,
> > ZYNQMP_PM_REQUEST_ACK_NO);
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_rproc_stop
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + *
> > > + * Power down  R5 Core.
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > +
> > > +	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
> > > +				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_rproc_mem_alloc
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + * @mem: mem entry to map
> > > + *
> > > + * Callback to map va for memory-region's carveout.
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
> > > +				     struct rproc_mem_entry *mem)
> > > +{
> > > +	void *va;
> > > +
> > > +	va = ioremap_wc(mem->dma, mem->len);
> > > +	if (IS_ERR_OR_NULL(va))
> > > +		return -ENOMEM;
> > > +
> > > +	mem->va = va;
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_rproc_mem_release
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + * @mem: mem entry to unmap
> > > + *
> > > + * Unmap memory-region carveout
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
> > > +				       struct rproc_mem_entry *mem)
> > > +{
> > > +	iounmap(mem->va);
> > > +	return 0;
> > > +}
> > > +
> > > +/*
> > > + * parse_mem_regions
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + *
> > > + * Construct rproc mem carveouts from carveout provided in
> > > + * memory-region property
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int parse_mem_regions(struct rproc *rproc)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > +	struct device_node *np = z_rproc->dev->of_node;
> > > +	struct device *dev = &rproc->dev;
> > > +	struct of_phandle_iterator it;
> > > +	struct rproc_mem_entry *mem;
> > > +	struct reserved_mem *rmem;
> > > +	int index = 0;
> > > +
> > > +	/* Register associated reserved memory regions */
> > > +	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
> > > +	while (of_phandle_iterator_next(&it) == 0) {
> > > +		rmem = of_reserved_mem_lookup(it.node);
> > > +		if (!rmem) {
> > > +			dev_err(dev, "unable to acquire %s\n",
> > DDR_LIST_PROP);
> > > +			return -EINVAL;
> > > +		}
> > > +
> > > +		if (strstr(it.node->name, "vdev0buffer")) {
> > > +			mem = rproc_of_resm_mem_entry_init(dev, index,
> > > +							   rmem->size,
> > > +							   rmem->base,
> > > +							   "vdev0buffer");
> > > +		} else {
> > > +			/*
> > > +			 * The ensuing carveout is either for vring, which has
> > > +			 * device tree node name of length 15 characters, or
> > > +			 * firmware mem, where name is not used by
> > remoteproc
> > > +			 *  core later on. So default to vring length of 15.
> >
> > s/"*  core"/"* core"
> >
> > > +			 *
> > > +			 * Extra char for null-terminated string.
> > > +			 */
> > > +			char name[16];
> > > +
> > > +			if (strstr(it.node->name, "vdev0vring")) {
> > > +				int vring_id;
> > > +
> > > +				/*
> > > +				 * Expect "rpuXvdev0vringX" as documented
> > > +				 * in xilinx remoteproc device tree binding
> > > +				 */
> > > +				if (strlen(it.node->name) < 15) {
> > > +					dev_err(dev, "%pOF is less than 15
> > chars",
> > > +						it.node);
> > > +					return -EINVAL;
> > > +				}
> > > +
> > > +				/*
> > > +				 * can be 1 of multiple vring IDs per IPC
> > channel
> > > +				 * e.g. 'vdev0vring0' and 'vdev0vring1'
> > > +				 *
> > > +				 * Here get vring ID from 15th char
> > > +				 */
> > > +				vring_id = it.node->name[14] - '0';
> > > +				snprintf(name, sizeof(name), "vdev0vring%d",
> > vring_id);
> >
> > Function strstr() returns a pointer to the beginning of the string being
> > searched for.  As such you should be able to do something like:
> >
> >                         name = strstr(it.node->name, "vdev0vring");
> >                         if (name) {
> >                                 ...
> >
> > and simply use @name when calling rproc_mem_entry_init().
> >
> > > +			} else {
> > > +				strncpy(name, it.node->name, 16);
> > > +			}
> > > +
> > > +			mem = rproc_mem_entry_init(dev, NULL,
> > > +						   (dma_addr_t)rmem->base,
> > > +						   rmem->size, rmem->base,
> > > +
> > zynqmp_r5_rproc_mem_alloc,
> > > +
> > zynqmp_r5_rproc_mem_release,
> > > +						   name);
> > > +		}
> > > +
> > > +		if (!mem)
> > > +			return -ENOMEM;
> > > +
> > > +		rproc_add_carveout(rproc, mem);
> > > +		index++;
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_pm_request_tcm
> > > + * @addr: base address of mem provided in R5 core's sram property.
> > > + *
> > > + * Given sram base address, determine its corresponding Xilinx
> > > + * Platform Management ID and then request access to this node
> > > + * so that it can be power up.
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
> > > +{
> > > +	int i;
> > > +
> > > +	for (i = 0; i < NUM_SRAMS; i++) {
> > > +		if (zynqmp_banks[i].addr != addr)
> > > +			continue;
> > > +
> > > +		return zynqmp_pm_request_node(zynqmp_banks[i].id,
> > > +					      ZYNQMP_PM_CAPABILITY_ACCESS,
> > 0,
> > > +
> > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > +	}
> > > +
> > > +	return -EINVAL;
> > > +}
> > > +
> > > +/*
> > > + * tcm_mem_alloc
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + * @mem: mem entry to initialize the va and da fields of
> > > + *
> > > + * Given TCM bank entry, this callback will set device address for R5
> > > + * running on TCM and also setup virtual address for TCM bank
> > > + * remoteproc carveout.
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int tcm_mem_alloc(struct rproc *rproc,
> > > +			 struct rproc_mem_entry *mem)
> > > +{
> > > +	void *va;
> > > +	struct device *dev = rproc->dev.parent;
> > > +
> > > +	va = ioremap_wc(mem->dma, mem->len);
> > > +	if (IS_ERR_OR_NULL(va))
> > > +		return -ENOMEM;
> > > +
> > > +	/* Update memory entry va */
> > > +	mem->va = va;
> > > +
> > > +	/*
> > > +	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> > > +	 * while on the Linux side they are at 0xffexxxxx.
> > > +	 *
> > > +	 * Zero out the high 12 bits of the address. This will give
> > > +	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> > > +	 */
> > > +	mem->da &= 0x000fffff;
> > > +
> > > +	/*
> > > +	 * TCM Banks 1A and 1B still have to be translated.
> > > +	 *
> > > +	 * Below handle these two banks' absolute addresses (0xffe90000 and
> > > +	 * 0xffeb0000) and convert to the expected relative addresses
> > > +	 * (0x0 and 0x20000).
> > > +	 */
> > > +	if (mem->da == 0x90000 || mem->da == 0xB0000)
> > > +		mem->da -= 0x90000;
> > > +
> > > +	/* if translated TCM bank address is not valid report error */
> > > +	if (mem->da != 0x0 && mem->da != 0x20000) {
> > > +		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
> > > +		return -EINVAL;
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +/*
> > > + * parse_tcm_banks()
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + *
> > > + * Given R5 node in remoteproc instance
> > > + * allocate remoteproc carveout for TCM memory
> > > + * needed for firmware to be loaded
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int parse_tcm_banks(struct rproc *rproc)
> > > +{
> > > +	int i, num_banks;
> > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > +	struct device *dev = &rproc->dev;
> > > +	struct device_node *r5_node = z_rproc->dev->of_node;
> > > +
> > > +	/* go through TCM banks for r5 node */
> > > +	num_banks = of_count_phandle_with_args(r5_node,
> > BANK_LIST_PROP, NULL);
> > > +	if (num_banks <= 0) {
> > > +		dev_err(dev, "need to specify TCM banks\n");
> > > +		return -EINVAL;
> > > +	}
> > > +	for (i = 0; i < num_banks; i++) {
> > > +		struct resource rsc;
> > > +		resource_size_t size;
> > > +		struct device_node *dt_node;
> > > +		struct rproc_mem_entry *mem;
> > > +		int ret;
> > > +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> > > +
> > > +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> > > +		if (!dt_node)
> > > +			return -EINVAL;
> > > +
> > > +		ret = of_address_to_resource(dt_node, 0, &rsc);
> > > +		if (ret < 0) {
> > > +			of_node_put(dt_node);
> > > +			return ret;
> > > +		}
> > > +		of_node_put(dt_node);
> > > +
> > > +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> > > +		if (ret < 0)
> > > +			return ret;
> > > +
> > > +		/* add carveout */
> > > +		size = resource_size(&rsc);
> > > +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> > > +					   (size_t)size, rsc.start,
> > > +					   tcm_mem_alloc,
> > > +					   tcm_mem_release,
> > > +					   rsc.name);
> > > +		if (!mem) {
> > > +			ret = zynqmp_pm_release_node(pnode_id);
> >
> > Where is @pnode_id initialised?
> 
> Good catch. Previously this was value set in an earlier iteration. I will fix this
> as follows:
> - release the R5 cluster by instead passing z_rproc->pnode_id
> - release the TCM bank via call to release_sram
> 
> This will also remove the variable pnode_id.
> 
> >
> > > +			if (ret)
> > > +				dev_warn(dev,
> > > +					 "fail to release node: %x ret: %x\n",
> > > +					 pnode_id, ret);
> > > +			return -ENOMEM;
> > > +		}
> > > +
> > > +		mem->priv = (void *)(u64)pnode_id;
> >
> >
> > And here too - how does this work when you test things out on your side?
> >
> > Please note that I will need a reply to these questions before I review
> another
> > set.
> >
> Here I will update as follows:
> - set mem->priv to value from zynqmp_banks lookup table that contains
> power node IDs
> 
> 
> > > +		rproc_add_carveout(rproc, mem);
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_parse_fw()
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + * @fw: ptr to firmware to be loaded onto r5 core
> > > + *
> > > + * When loading firmware, ensure the necessary carveouts are in
> > remoteproc
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware
> > *fw)
> > > +{
> > > +	int ret;
> > > +
> > > +	ret = parse_tcm_banks(rproc);
> > > +	if (ret)
> > > +		return ret;
> > > +
> > > +	ret = parse_mem_regions(rproc);
> > > +	if (ret)
> > > +		return ret;
> > > +
> > > +	ret = rproc_elf_load_rsc_table(rproc, fw);
> > > +	if (ret == -EINVAL) {
> > > +		/*
> > > +		 * resource table only required for IPC.
> > > +		 * if not present, this is not necessarily an error;
> > > +		 * for example, loading r5 hello world application
> > > +		 * so simply inform user and keep going.
> > > +		 */
> > > +		dev_info(&rproc->dev, "no resource table found.\n");
> > > +		ret = 0;
> > > +	}
> > > +	return ret;
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> > > + * @rproc: r5 core's corresponding rproc structure
> > > + * @vqid: virtqueue ID
> > > + */
> > > +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > +	struct device *dev = rproc->dev.parent;
> > > +	struct zynqmp_ipi_message *mb_msg = NULL;
> > > +	unsigned int skb_len;
> > > +	struct sk_buff *skb;
> > > +	int ret;
> > > +
> > > +	/* If vqid is negative, do not pass the vqid to
> > > +	 * mailbox as vqid is supposed to be 0 or positive.
> > > +	 * It also gives a way to just kick instead but
> > > +	 * not use the IPI buffer.
> > > +	 *
> > > +	 * For now use negative vqid to assume no message will be
> > > +	 * passed with IPI buffer, but just raise interrupt.
> > > +	 * This will be faster as it doesn't need to copy the
> > > +	 * message to the ZynqMP/Versal IPI message buffer.
> > > +	 */
> > > +	if (vqid >= 0) {
> > > +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> > > +		skb = alloc_skb(skb_len, GFP_KERNEL);
> > > +		if (!skb)
> > > +			return;
> > > +
> > > +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb,
> > skb_len);
> > > +		mb_msg->len = sizeof(vqid);
> > > +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> > > +
> > > +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> > > +	}
> > > +
> > > +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> > > +	if (ret < 0) {
> > > +		dev_warn(dev, "Failed to kick remote.\n");
> > > +		if (mb_msg) {
> > > +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> > > +			kfree_skb(skb);
> > > +		}
> > > +	}
> > > +}
> > > +
> >
> > I am done reviewing this set.
> >
> > Thanks,
> > Mathieu
> >
> > > +static struct rproc_ops zynqmp_r5_rproc_ops = {
> > > +	.start		= zynqmp_r5_rproc_start,
> > > +	.stop		= zynqmp_r5_rproc_stop,
> > > +	.load		= rproc_elf_load_segments,
> > > +	.parse_fw	= zynqmp_r5_parse_fw,
> > > +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > > +	.sanity_check	= rproc_elf_sanity_check,
> > > +	.get_boot_addr	= rproc_elf_get_boot_addr,
> > > +	.kick		= zynqmp_r5_rproc_kick,
> > > +};
> > > +
> > > +/**
> > > + * event_notified_idr_cb() - event notified idr callback
> > > + * @id: idr id
> > > + * @ptr: pointer to idr private data
> > > + * @data: data passed to idr_for_each callback
> > > + *
> > > + * Pass notification to remoteproc virtio
> > > + *
> > > + * Return: 0. having return is to satisfy the idr_for_each() function
> > > + *          pointer input argument requirement.
> > > + **/
> > > +static int event_notified_idr_cb(int id, void *ptr, void *data)
> > > +{
> > > +	struct rproc *rproc = data;
> > > +
> > > +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> > > +		dev_dbg(&rproc->dev, "no message was found in idr with id:
> > %d\n", id);
> > > +	return 0;
> > > +}
> > > +
> > > +/**
> > > + * handle_event_notified() - remoteproc notification work function
> > > + * @work: pointer to the work structure
> > > + *
> > > + * It checks each registered remoteproc notify IDs.
> > > + */
> > > +static void handle_event_notified(struct work_struct *work)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc;
> > > +	struct rproc *rproc;
> > > +
> > > +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> > > +	rproc = z_rproc->rproc;
> > > +
> > > +	/*
> > > +	 * This is to simply raise interrupt as ACK for remote.
> > > +	 *
> > > +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> > > +	 * mapping to remote processor.
> > > +	 */
> > > +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> > > +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> > > +
> > > +	/*
> > > +	 * We only use IPI for interrupt. The firmware side may or may
> > > +	 * not write the notifyid when it trigger IPI.
> > > +	 * And thus, we scan through all the registered notifyids.
> > > +	 */
> > > +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> > > +}
> > > +
> > > +/**
> > > + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> > > + * @cl: mailbox client
> > > + * @msg: message pointer
> > > + *
> > > + * It will schedule the R5 notification work.
> > > + */
> > > +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc;
> > > +
> > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> > > +	/*
> > > +	 * Notification can be received with message payload.
> > > +	 * To handle this, check for message contents.
> > > +	 * If there are contents, copy to ipi message buffer payload location.
> > > +	 *
> > > +	 * If message is empty, then that means that was simple raising
> > > +	 * of interrupt. no payload to process.
> > > +	 *
> > > +	 * Note: enqueue work regardless of msg as kick may imply that
> > remote
> > > +	 * is waiting for ack.
> > > +	 */
> > > +	if (msg) {
> > > +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> > > +		size_t len;
> > > +
> > > +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> > > +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> > > +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> > > +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> > > +		buf_msg->len = len;
> > > +		memcpy(buf_msg->data, ipi_msg->data, len);
> > > +	}
> > > +	schedule_work(&z_rproc->mbox_work);
> > > +}
> > > +
> > > +/**
> > > + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> > > + * @cl: mailbox client
> > > + * @msg: pointer to the message which has been sent
> > > + * @r: status of last TX - OK or error
> > > + *
> > > + * It will be called by the mailbox framework when the last TX has done.
> > > + */
> > > +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg,
> int
> > r)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc;
> > > +	struct sk_buff *skb;
> > > +
> > > +	if (!msg)
> > > +		return;
> > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> > > +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> > > +	kfree_skb(skb);
> > > +}
> > > +
> > > +/**
> > > + * zynqmp_r5_setup_mbox() - Setup mailboxes
> > > + *			    this is used for each individual R5 core
> > > + *
> > > + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> > > + * @node: pointer of the device node
> > > + *
> > > + * Function to setup mailboxes to talk to RPU.
> > > + *
> > > + * Return: 0 for success, negative value for failure.
> > > + */
> > > +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> > > +				struct device_node *node)
> > > +{
> > > +	struct mbox_client *mclient;
> > > +
> > > +	/* Setup TX mailbox channel client */
> > > +	mclient			= &z_rproc->tx_mc;
> > > +	mclient->dev		= z_rproc->dev;
> > > +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> > > +	mclient->tx_block	= false;
> > > +	mclient->knows_txdone	= false;
> > > +	mclient->rx_callback	= NULL;
> > > +
> > > +	/* Setup RX mailbox channel client */
> > > +	mclient			= &z_rproc->rx_mc;
> > > +	mclient->dev		= z_rproc->dev;
> > > +	mclient->tx_done	= NULL;
> > > +	mclient->tx_block	= false;
> > > +	mclient->knows_txdone	= false;
> > > +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> > > +
> > > +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> > > +
> > > +	/* Request TX and RX channels */
> > > +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc-
> > >tx_mc, "tx");
> > > +	if (IS_ERR(z_rproc->tx_chan)) {
> > > +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> > > +		return PTR_ERR(z_rproc->tx_chan);
> > > +	}
> > > +
> > > +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc-
> > >rx_mc, "rx");
> > > +	if (IS_ERR(z_rproc->rx_chan)) {
> > > +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> > > +		return PTR_ERR(z_rproc->rx_chan);
> > > +	}
> > > +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc *z_rproc)
> > > +{
> > > +	mbox_free_channel(z_rproc->tx_chan);
> > > +	mbox_free_channel(z_rproc->rx_chan);
> > > +}
> > > +
> > > +/**
> > > + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> > > + *		       this is called for each individual R5 core to
> > > + *		       set up mailbox, Xilinx platform manager unique ID,
> > > + *		       add to rproc core
> > > + *
> > > + * @pdev: domain platform device for current R5 core
> > > + * @node: pointer of the device node for current R5 core
> > > + * @rpu_mode: mode to configure RPU, split or lockstep
> > > + *
> > > + * Return: 0 for success, negative value for failure.
> > > + */
> > > +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device
> > *pdev,
> > > +					       struct device_node *node,
> > > +					       enum rpu_oper_mode
> > rpu_mode)
> > > +{
> > > +	int ret;
> > > +	struct device *dev = &pdev->dev;
> > > +	struct rproc *rproc_ptr;
> > > +	struct zynqmp_r5_rproc *z_rproc;
> > > +
> > > +	/* Allocate remoteproc instance */
> > > +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev),
> > &zynqmp_r5_rproc_ops,
> > > +				     NULL, sizeof(struct zynqmp_r5_rproc));
> > > +	if (!rproc_ptr) {
> > > +		ret = -ENOMEM;
> > > +		goto error;
> > > +	}
> > > +
> > > +	rproc_ptr->auto_boot = false;
> > > +	z_rproc = rproc_ptr->priv;
> > > +	z_rproc->rproc = rproc_ptr;
> > > +	z_rproc->dev = dev;
> > > +
> > > +	/* Set up DMA mask */
> > > +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> > > +	if (ret)
> > > +		goto error;
> > > +
> > > +	/* Get R5 power domain node */
> > > +	ret = of_property_read_u32(node, "power-domain", &z_rproc-
> > >pnode_id);
> > > +	if (ret)
> > > +		goto error;
> > > +
> > > +	ret = r5_set_mode(z_rproc, rpu_mode);
> > > +	if (ret)
> > > +		goto error;
> > > +
> > > +	if (of_property_read_bool(node, "mboxes")) {
> > > +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> > > +		if (ret)
> > > +			goto error;
> > > +	}
> > > +
> > > +	/* Add R5 remoteproc */
> > > +	ret = devm_rproc_add(dev, rproc_ptr);
> > > +	if (ret) {
> > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > +		goto error;
> > > +	}
> > > +
> > > +	return z_rproc;
> > > +error:
> > > +	return ERR_PTR(ret);
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_remoteproc_probe()
> > > + *
> > > + * @pdev: domain platform device for R5 cluster
> > > + *
> > > + * called when driver is probed, for each R5 core specified in DT,
> > > + * setup as needed to do remoteproc-related operations
> > > + *
> > > + * Return: 0 for success, negative value for failure.
> > > + */
> > > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > > +{
> > > +	int ret, core_count;
> > > +	struct device *dev = &pdev->dev;
> > > +	struct device_node *nc;
> > > +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> > > +	struct list_head *cluster; /* list to track each core's rproc */
> > > +	struct zynqmp_r5_rproc *z_rproc;
> > > +	struct platform_device *child_pdev;
> > > +	struct list_head *pos;
> > > +
> > > +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode",
> > &rpu_mode);
> > > +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> > > +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> > > +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> > > +			ret, rpu_mode);
> > > +		return ret;
> > > +	}
> > > +
> > > +	dev_dbg(dev, "RPU configuration: %s\n",
> > > +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" :
> > "split");
> > > +
> > > +	/*
> > > +	 * if 2 RPUs provided but one is lockstep, then we have an
> > > +	 * invalid configuration.
> > > +	 */
> > > +
> > > +	core_count = of_get_available_child_count(dev->of_node);
> > > +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> > > +	    core_count > MAX_RPROCS)
> > > +		return -EINVAL;
> > > +
> > > +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > > +	if (!cluster)
> > > +		return -ENOMEM;
> > > +	INIT_LIST_HEAD(cluster);
> > > +
> > > +	ret = devm_of_platform_populate(dev);
> > > +	if (ret) {
> > > +		dev_err(dev, "devm_of_platform_populate failed, ret =
> > %d\n",
> > > +			ret);
> > > +		return ret;
> > > +	}
> > > +
> > > +	/* probe each individual r5 core's remoteproc-related info */
> > > +	for_each_available_child_of_node(dev->of_node, nc) {
> > > +		child_pdev = of_find_device_by_node(nc);
> > > +		if (!child_pdev) {
> > > +			dev_err(dev, "could not get R5 core platform
> > device\n");
> > > +			ret = -ENODEV;
> > > +			goto out;
> > > +		}
> > > +
> > > +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> > > +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> > > +			ret ? "Failed" : "Able",
> > > +			nc);
> > > +		if (IS_ERR(z_rproc)) {
> > > +			ret = PTR_ERR(z_rproc);
> > > +			goto out;
> > > +		}
> > > +		list_add_tail(&z_rproc->elem, cluster);
> > > +	}
> > > +	/* wire in so each core can be cleaned up at driver remove */
> > > +	platform_set_drvdata(pdev, cluster);
> > > +	return 0;
> > > +out:
> > > +	/*
> > > +	 * undo core0 upon any failures on core1 in split-mode
> > > +	 *
> > > +	 * in zynqmp_r5_probe z_rproc is set to null
> > > +	 * and ret to non-zero value if error
> > > +	 */
> > > +	list_for_each(pos, cluster) {
> > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > +	}
> > > +	return ret;
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_remoteproc_remove()
> > > + *
> > > + * @pdev: domain platform device for R5 cluster
> > > + *
> > > + * When the driver is unloaded, clean up the mailboxes for each
> > > + * remoteproc that was initially probed.
> > > + */
> > > +static int zynqmp_r5_remoteproc_remove(struct platform_device *pdev)
> > > +{
> > > +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> > > +						 platform_get_drvdata(pdev);
> > > +	struct zynqmp_r5_rproc *z_rproc = NULL;
> > > +
> > > +	list_for_each_safe(pos, temp, cluster) {
> > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > +		list_del(pos);
> > > +	}
> > > +	return 0;
> > > +}
> > > +
> > > +/* Match table for OF platform binding */
> > > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > > +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> > > +	{ /* end of list */ },
> > > +};
> > > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > > +
> > > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > > +	.probe = zynqmp_r5_remoteproc_probe,
> > > +	.remove = zynqmp_r5_remoteproc_remove,
> > > +	.driver = {
> > > +		.name = "zynqmp_r5_remoteproc",
> > > +		.of_match_table = zynqmp_r5_remoteproc_match,
> > > +	},
> > > +};
> > > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > > +
> > > +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> > > +MODULE_LICENSE("GPL v2");
> > > --
> > > 2.17.1
> > >

^ permalink raw reply	[flat|nested] 28+ messages in thread

* RE: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
@ 2021-02-17 19:11         ` Ben Levinsky
  0 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-02-17 19:11 UTC (permalink / raw)
  To: Mathieu Poirier
  Cc: devicetree, Michal Simek, linux-remoteproc, linux-kernel,
	linux-arm-kernel

Hi Mathieu,

I worked on this further and had 1 more comment on the prepare/unprepare for TCM banks below

> -----Original Message-----
> From: Ben Levinsky
> Sent: Wednesday, February 17, 2021 8:47 AM
> To: Mathieu Poirier <mathieu.poirier@linaro.org>
> Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal Simek
> <michals@xilinx.com>
> Subject: RE: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc
> driver
> 
> Hi Mathieu,
> 
> Please see my replies to your questions inline
> 
> Again thanks for the continued review of this set
> 
> Cheers
> Ben
> 
> > -----Original Message-----
> > From: Mathieu Poirier <mathieu.poirier@linaro.org>
> > Sent: Tuesday, February 16, 2021 10:48 AM
> > To: Ben Levinsky <BLEVINSK@xilinx.com>
> > Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal Simek
> > <michals@xilinx.com>
> > Subject: Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc
> > driver
> >
> > On Sun, Jan 10, 2021 at 06:02:50PM -0800, Ben Levinsky wrote:
> > > R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
> > > remotproc driver, we can boot the R5 sub-system in two different
> > > configurations -
> > > 	* Split
> > > 	* Lockstep
> > >
> > > The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
> > > Platform Management Unit that handles the R5 configuration, memory
> > access
> > > and R5 lifecycle management. The interface to this manager is done in this
> > > driver via zynqmp_pm_* function calls.
> > >
> > > Signed-off-by: Wendy Liang <wendy.liang@xilinx.com>
> > > Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> > > Signed-off-by: Ed Mooring <ed.mooring@xilinx.com>
> > > Signed-off-by: Jason Wu <j.wu@xilinx.com>
> > > Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> > > ---
> > > v25:
> > > - reword error message for rpu configuration
> > > - change char name[15] to 16 chars to have null terminated string
> > >   in parse_mem_regions()
> > > - reword comments in tcm_mem_alloc
> > > - call xilinx platform cleanup in parse_tcm_banks() case of failure
> > > - check value of mbox_send_message in all instances within this driver
> > > - indentation in zynqmp_r5_remoteproc driver
> > > - update style of constructing carveouts to match convention in ST
> > >   remoteproc drivers. No longer iterate through memory-region property
> > >   using of_count_phandle_with_args. Instead use of_phandle_iterator
> > > - use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per
> > feedback
> > > - rework loop in zynqmp_r5_pm_request_sram as per feedback
> > > - fix comment and linebreak in tcm_mem_alloc description
> > > - add comments and remove extraneous devm_ioremap_wc call in
> > tcm_mem_alloc
> > > - in parse_tcm_banks remove check for bank not being available
> > > - rework order of locals and remove else in zynqmp_r5_rproc_kick
> > > - document and update logic for zynqmp_r5_parse_fw
> > > - add dev_dbg output in event_notified_idr_cb if
> > >   rproc_vq_interrupt(rproc, id) == IRQ_NONE
> > > - add comment for handle_event_notified mbox_send_message function
> > call
> > > - add comment for zynqmp_r5_mb_rx_cb
> > > - update zynqmp_r5_setup_mbox to match convention of mbox setup in
> ST
> > >   remoteproc drivers
> > > - change return in zynqmp_r5_setup_mbox to use PTR_ERR
> > > - add zynqmp_r5_cleanup_mbox
> > > - in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as
> arg.
> > > - in zynqmp_r5_probe use return of PTR_ERR and use
> > zynqmp_r5_cleanup_mbox
> > > - in zynqmp_r5_remoteproc_probe update use return value of
> > zynqmp_r5_probe as
> > >   either zynqmp_r5_rproc* or PTR_ERR
> > > - update loop that cleans up cluster and mboxes in
> > zynqmp_r5_remoteproc_probe
> > > - update loop that cleans up cluster and mboxes in
> > zynqmp_r5_remoteproc_remove
> > > ---
> > >  drivers/remoteproc/Kconfig                |   8 +
> > >  drivers/remoteproc/Makefile               |   1 +
> > >  drivers/remoteproc/zynqmp_r5_remoteproc.c | 898
> > ++++++++++++++++++++++
> > >  3 files changed, 907 insertions(+)
> > >  create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c
> > >
> > > diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> > > index c6659dfea7c7..c2fe54b1d94f 100644
> > > --- a/drivers/remoteproc/Kconfig
> > > +++ b/drivers/remoteproc/Kconfig
> > > @@ -275,6 +275,14 @@ config TI_K3_DSP_REMOTEPROC
> > >  	  It's safe to say N here if you're not interested in utilizing
> > >  	  the DSP slave processors.
> > >
> > > +config ZYNQMP_R5_REMOTEPROC
> > > +	tristate "ZynqMP R5 remoteproc support"
> > > +	depends on PM && ARCH_ZYNQMP
> > > +	select RPMSG_VIRTIO
> > > +	select ZYNQMP_IPI_MBOX
> > > +	help
> > > +	  Say y or m here to support ZynqMP R5 remote processors via the
> > remote
> > > +	  processor framework.
> > >  endif # REMOTEPROC
> > >
> > >  endmenu
> > > diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> > > index 3dfa28e6c701..ef1abff654c2 100644
> > > --- a/drivers/remoteproc/Makefile
> > > +++ b/drivers/remoteproc/Makefile
> > > @@ -33,3 +33,4 @@ obj-$(CONFIG_ST_REMOTEPROC)		+=
> > st_remoteproc.o
> > >  obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
> > >  obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
> > >  obj-$(CONFIG_TI_K3_DSP_REMOTEPROC)	+= ti_k3_dsp_remoteproc.o
> > > +obj-$(CONFIG_ZYNQMP_R5_REMOTEPROC)	+= zynqmp_r5_remoteproc.o
> > > diff --git a/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > > new file mode 100644
> > > index 000000000000..9e228c9ecd43
> > > --- /dev/null
> > > +++ b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > > @@ -0,0 +1,898 @@
> > > +// SPDX-License-Identifier: GPL-2.0
> > > +/*
> > > + * Zynq R5 Remote Processor driver
> > > + *
> > > + * Based on origin OMAP and Zynq Remote Processor driver
> > > + *
> > > + */
> > > +
> > > +#include <linux/firmware/xlnx-zynqmp.h>
> > > +#include <linux/interrupt.h>
> > > +#include <linux/kernel.h>
> > > +#include <linux/list.h>
> > > +#include <linux/mailbox_client.h>
> > > +#include <linux/mailbox/zynqmp-ipi-message.h>
> > > +#include <linux/module.h>
> > > +#include <linux/of_address.h>
> > > +#include <linux/of_platform.h>
> > > +#include <linux/of_reserved_mem.h>
> > > +#include <linux/platform_device.h>
> > > +#include <linux/remoteproc.h>
> > > +#include <linux/skbuff.h>
> > > +#include <linux/sysfs.h>
> > > +
> > > +#include "remoteproc_internal.h"
> > > +
> > > +#define MAX_RPROCS	2 /* Support up to 2 RPU */
> > > +#define MAX_MEM_PNODES	4 /* Max power nodes for one RPU
> memory
> > instance */
> > > +
> > > +#define BANK_LIST_PROP	"sram"
> > > +#define DDR_LIST_PROP	"memory-region"
> > > +
> > > +/* IPI buffer MAX length */
> > > +#define IPI_BUF_LEN_MAX	32U
> > > +/* RX mailbox client buffer max length */
> > > +#define RX_MBOX_CLIENT_BUF_MAX	(IPI_BUF_LEN_MAX + \
> > > +				 sizeof(struct zynqmp_ipi_message))
> > > +
> > > +/*
> > > + * Map each Xilinx on-chip SRAM  Bank address to their own respective
> > > + * pm_node_id.
> > > + */
> > > +struct sram_addr_data {
> > > +	phys_addr_t addr;
> > > +	enum pm_node_id id;
> > > +};
> > > +
> > > +#define NUM_SRAMS 4U
> > > +static const struct sram_addr_data zynqmp_banks[NUM_SRAMS] = {
> > > +	{0xffe00000UL, NODE_TCM_0_A},
> > > +	{0xffe20000UL, NODE_TCM_0_B},
> > > +	{0xffe90000UL, NODE_TCM_1_A},
> > > +	{0xffeb0000UL, NODE_TCM_1_B},
> > > +};
> > > +
> > > +/**
> > > + * struct zynqmp_r5_rproc - ZynqMP R5 core structure
> > > + *
> > > + * @rx_mc_buf: rx mailbox client buffer to save the rx message
> > > + * @tx_mc: tx mailbox client
> > > + * @rx_mc: rx mailbox client
> > > + * @mbox_work: mbox_work for the RPU remoteproc
> > > + * @tx_mc_skbs: socket buffers for tx mailbox client
> > > + * @dev: device of RPU instance
> > > + * @rproc: rproc handle
> > > + * @tx_chan: tx mailbox channel
> > > + * @rx_chan: rx mailbox channel
> > > + * @pnode_id: RPU CPU power domain id
> > > + * @elem: linked list item
> > > + */
> > > +struct zynqmp_r5_rproc {
> > > +	unsigned char rx_mc_buf[RX_MBOX_CLIENT_BUF_MAX];
> > > +	struct mbox_client tx_mc;
> > > +	struct mbox_client rx_mc;
> > > +	struct work_struct mbox_work;
> > > +	struct sk_buff_head tx_mc_skbs;
> > > +	struct device *dev;
> > > +	struct rproc *rproc;
> > > +	struct mbox_chan *tx_chan;
> > > +	struct mbox_chan *rx_chan;
> > > +	u32 pnode_id;
> > > +	struct list_head elem;
> > > +};
> > > +
> > > +/*
> > > + * r5_set_mode - set RPU operation mode
> > > + * @z_rproc: Remote processor private data
> > > + * @rpu_mode: mode specified by device tree to configure the RPU to
> > > + *
> > > + * set RPU operation mode
> > > + *
> > > + * Return: 0 for success, negative value for failure
> > > + */
> > > +static int r5_set_mode(struct zynqmp_r5_rproc *z_rproc,
> > > +		       enum rpu_oper_mode rpu_mode)
> > > +{
> > > +	enum rpu_tcm_comb tcm_mode;
> > > +	enum rpu_oper_mode cur_rpu_mode;
> > > +	int ret;
> > > +
> > > +	ret = zynqmp_pm_get_rpu_mode(z_rproc->pnode_id,
> > &cur_rpu_mode);
> > > +	if (ret < 0)
> > > +		return ret;
> > > +
> > > +	if (rpu_mode != cur_rpu_mode) {
> > > +		ret = zynqmp_pm_set_rpu_mode(z_rproc->pnode_id,
> > rpu_mode);
> > > +		if (ret < 0)
> > > +			return ret;
> > > +	}
> > > +
> > > +	tcm_mode = (rpu_mode == PM_RPU_MODE_LOCKSTEP) ?
> > > +		    PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> > > +	return zynqmp_pm_set_tcm_config(z_rproc->pnode_id, tcm_mode);
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_rproc_mem_release
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + * @mem: mem entry to unmap
> > > + *
> > > + * Unmap TCM banks when powering down R5 core.
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int tcm_mem_release(struct rproc *rproc, struct rproc_mem_entry
> > *mem)
> > > +{
> > > +	u32 pnode_id = (u64)mem->priv;
> > > +
> > > +	iounmap(mem->va);
> > > +	return zynqmp_pm_release_node(pnode_id);
> >
> > Is the opposite of that call zynqmp_pm_request_node()?  If so doing the
> > power up
> > and down of the TCM banks at driver probe() and remove() is highly
> > inefficient.
> > Consider using rproc_ops::prepare() and rproc_ops::unprepare().
> >
If I understand your comment correctly this is about handling the request_node() and release_node() calls. While its true that these two functions can be moved to prepare and unprepared I think these are still occurring not at probe nor remove.

For request_node() of the TCM bank this is occurring in parse_fw() in remoteproc_core as in the zynqmp_r5 driver shows below:

	.parse_fw	= zynqmp_r5_parse_fw,
	...
	Zynqmp_parse_fw calls parse_tcm_banks

	static int parse_tcm_banks(struct rproc *rproc){
		...
		ret = zynqmp_r5_pm_request_sram(rsc.start);
		...
 		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
					   (size_t)size, rsc.start,
					   tcm_mem_alloc,
					   tcm_mem_release,
					   rsc.name);

For release_node() this is occurring not at remove but instead is managed by remoteproc_core and tied to the lifecyle of the remoteproc carveout since release_node() is within the function tcm_mem_release() and therefore rproc_resource_cleanup() will end up triggering 
TCM bank cleanup.


Still it is understandable to prefer prepare/unprepared to contain the 2 lines.


If you still prefer prepare/unprepared is the following implementation ok?

prepare: 
	Just move request_node() line here	
unprepare: 
	similarly this is only moving the 1 line of release_node().

If this suggestion satisfies what you are saying I will do so, otherwise please correct my understanding


Cheers
Ben 



> > > +}
> >
> > Please move this just after tcm_mem_alloc()
> >
> > > +
> > > +/*
> > > + * zynqmp_r5_rproc_start
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + *
> > > + * Start R5 Core from designated boot address.
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > +	enum rpu_boot_mem bootmem;
> > > +
> > > +	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
> > > +		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> > > +
> > > +	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
> > > +		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> > > +
> > > +	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
> > > +				      bootmem,
> > ZYNQMP_PM_REQUEST_ACK_NO);
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_rproc_stop
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + *
> > > + * Power down  R5 Core.
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > +
> > > +	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
> > > +				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_rproc_mem_alloc
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + * @mem: mem entry to map
> > > + *
> > > + * Callback to map va for memory-region's carveout.
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
> > > +				     struct rproc_mem_entry *mem)
> > > +{
> > > +	void *va;
> > > +
> > > +	va = ioremap_wc(mem->dma, mem->len);
> > > +	if (IS_ERR_OR_NULL(va))
> > > +		return -ENOMEM;
> > > +
> > > +	mem->va = va;
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_rproc_mem_release
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + * @mem: mem entry to unmap
> > > + *
> > > + * Unmap memory-region carveout
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
> > > +				       struct rproc_mem_entry *mem)
> > > +{
> > > +	iounmap(mem->va);
> > > +	return 0;
> > > +}
> > > +
> > > +/*
> > > + * parse_mem_regions
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + *
> > > + * Construct rproc mem carveouts from carveout provided in
> > > + * memory-region property
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int parse_mem_regions(struct rproc *rproc)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > +	struct device_node *np = z_rproc->dev->of_node;
> > > +	struct device *dev = &rproc->dev;
> > > +	struct of_phandle_iterator it;
> > > +	struct rproc_mem_entry *mem;
> > > +	struct reserved_mem *rmem;
> > > +	int index = 0;
> > > +
> > > +	/* Register associated reserved memory regions */
> > > +	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
> > > +	while (of_phandle_iterator_next(&it) == 0) {
> > > +		rmem = of_reserved_mem_lookup(it.node);
> > > +		if (!rmem) {
> > > +			dev_err(dev, "unable to acquire %s\n",
> > DDR_LIST_PROP);
> > > +			return -EINVAL;
> > > +		}
> > > +
> > > +		if (strstr(it.node->name, "vdev0buffer")) {
> > > +			mem = rproc_of_resm_mem_entry_init(dev, index,
> > > +							   rmem->size,
> > > +							   rmem->base,
> > > +							   "vdev0buffer");
> > > +		} else {
> > > +			/*
> > > +			 * The ensuing carveout is either for vring, which has
> > > +			 * device tree node name of length 15 characters, or
> > > +			 * firmware mem, where name is not used by
> > remoteproc
> > > +			 *  core later on. So default to vring length of 15.
> >
> > s/"*  core"/"* core"
> >
> > > +			 *
> > > +			 * Extra char for null-terminated string.
> > > +			 */
> > > +			char name[16];
> > > +
> > > +			if (strstr(it.node->name, "vdev0vring")) {
> > > +				int vring_id;
> > > +
> > > +				/*
> > > +				 * Expect "rpuXvdev0vringX" as documented
> > > +				 * in xilinx remoteproc device tree binding
> > > +				 */
> > > +				if (strlen(it.node->name) < 15) {
> > > +					dev_err(dev, "%pOF is less than 15
> > chars",
> > > +						it.node);
> > > +					return -EINVAL;
> > > +				}
> > > +
> > > +				/*
> > > +				 * can be 1 of multiple vring IDs per IPC
> > channel
> > > +				 * e.g. 'vdev0vring0' and 'vdev0vring1'
> > > +				 *
> > > +				 * Here get vring ID from 15th char
> > > +				 */
> > > +				vring_id = it.node->name[14] - '0';
> > > +				snprintf(name, sizeof(name), "vdev0vring%d",
> > vring_id);
> >
> > Function strstr() returns a pointer to the beginning of the string being
> > searched for.  As such you should be able to do something like:
> >
> >                         name = strstr(it.node->name, "vdev0vring");
> >                         if (name) {
> >                                 ...
> >
> > and simply use @name when calling rproc_mem_entry_init().
> >
> > > +			} else {
> > > +				strncpy(name, it.node->name, 16);
> > > +			}
> > > +
> > > +			mem = rproc_mem_entry_init(dev, NULL,
> > > +						   (dma_addr_t)rmem->base,
> > > +						   rmem->size, rmem->base,
> > > +
> > zynqmp_r5_rproc_mem_alloc,
> > > +
> > zynqmp_r5_rproc_mem_release,
> > > +						   name);
> > > +		}
> > > +
> > > +		if (!mem)
> > > +			return -ENOMEM;
> > > +
> > > +		rproc_add_carveout(rproc, mem);
> > > +		index++;
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_pm_request_tcm
> > > + * @addr: base address of mem provided in R5 core's sram property.
> > > + *
> > > + * Given sram base address, determine its corresponding Xilinx
> > > + * Platform Management ID and then request access to this node
> > > + * so that it can be power up.
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
> > > +{
> > > +	int i;
> > > +
> > > +	for (i = 0; i < NUM_SRAMS; i++) {
> > > +		if (zynqmp_banks[i].addr != addr)
> > > +			continue;
> > > +
> > > +		return zynqmp_pm_request_node(zynqmp_banks[i].id,
> > > +					      ZYNQMP_PM_CAPABILITY_ACCESS,
> > 0,
> > > +
> > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > +	}
> > > +
> > > +	return -EINVAL;
> > > +}
> > > +
> > > +/*
> > > + * tcm_mem_alloc
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + * @mem: mem entry to initialize the va and da fields of
> > > + *
> > > + * Given TCM bank entry, this callback will set device address for R5
> > > + * running on TCM and also setup virtual address for TCM bank
> > > + * remoteproc carveout.
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int tcm_mem_alloc(struct rproc *rproc,
> > > +			 struct rproc_mem_entry *mem)
> > > +{
> > > +	void *va;
> > > +	struct device *dev = rproc->dev.parent;
> > > +
> > > +	va = ioremap_wc(mem->dma, mem->len);
> > > +	if (IS_ERR_OR_NULL(va))
> > > +		return -ENOMEM;
> > > +
> > > +	/* Update memory entry va */
> > > +	mem->va = va;
> > > +
> > > +	/*
> > > +	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> > > +	 * while on the Linux side they are at 0xffexxxxx.
> > > +	 *
> > > +	 * Zero out the high 12 bits of the address. This will give
> > > +	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> > > +	 */
> > > +	mem->da &= 0x000fffff;
> > > +
> > > +	/*
> > > +	 * TCM Banks 1A and 1B still have to be translated.
> > > +	 *
> > > +	 * Below handle these two banks' absolute addresses (0xffe90000 and
> > > +	 * 0xffeb0000) and convert to the expected relative addresses
> > > +	 * (0x0 and 0x20000).
> > > +	 */
> > > +	if (mem->da == 0x90000 || mem->da == 0xB0000)
> > > +		mem->da -= 0x90000;
> > > +
> > > +	/* if translated TCM bank address is not valid report error */
> > > +	if (mem->da != 0x0 && mem->da != 0x20000) {
> > > +		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
> > > +		return -EINVAL;
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +/*
> > > + * parse_tcm_banks()
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + *
> > > + * Given R5 node in remoteproc instance
> > > + * allocate remoteproc carveout for TCM memory
> > > + * needed for firmware to be loaded
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int parse_tcm_banks(struct rproc *rproc)
> > > +{
> > > +	int i, num_banks;
> > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > +	struct device *dev = &rproc->dev;
> > > +	struct device_node *r5_node = z_rproc->dev->of_node;
> > > +
> > > +	/* go through TCM banks for r5 node */
> > > +	num_banks = of_count_phandle_with_args(r5_node,
> > BANK_LIST_PROP, NULL);
> > > +	if (num_banks <= 0) {
> > > +		dev_err(dev, "need to specify TCM banks\n");
> > > +		return -EINVAL;
> > > +	}
> > > +	for (i = 0; i < num_banks; i++) {
> > > +		struct resource rsc;
> > > +		resource_size_t size;
> > > +		struct device_node *dt_node;
> > > +		struct rproc_mem_entry *mem;
> > > +		int ret;
> > > +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> > > +
> > > +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> > > +		if (!dt_node)
> > > +			return -EINVAL;
> > > +
> > > +		ret = of_address_to_resource(dt_node, 0, &rsc);
> > > +		if (ret < 0) {
> > > +			of_node_put(dt_node);
> > > +			return ret;
> > > +		}
> > > +		of_node_put(dt_node);
> > > +
> > > +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> > > +		if (ret < 0)
> > > +			return ret;
> > > +
> > > +		/* add carveout */
> > > +		size = resource_size(&rsc);
> > > +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> > > +					   (size_t)size, rsc.start,
> > > +					   tcm_mem_alloc,
> > > +					   tcm_mem_release,
> > > +					   rsc.name);
> > > +		if (!mem) {
> > > +			ret = zynqmp_pm_release_node(pnode_id);
> >
> > Where is @pnode_id initialised?
> 
> Good catch. Previously this was value set in an earlier iteration. I will fix this
> as follows:
> - release the R5 cluster by instead passing z_rproc->pnode_id
> - release the TCM bank via call to release_sram
> 
> This will also remove the variable pnode_id.
> 
> >
> > > +			if (ret)
> > > +				dev_warn(dev,
> > > +					 "fail to release node: %x ret: %x\n",
> > > +					 pnode_id, ret);
> > > +			return -ENOMEM;
> > > +		}
> > > +
> > > +		mem->priv = (void *)(u64)pnode_id;
> >
> >
> > And here too - how does this work when you test things out on your side?
> >
> > Please note that I will need a reply to these questions before I review
> another
> > set.
> >
> Here I will update as follows:
> - set mem->priv to value from zynqmp_banks lookup table that contains
> power node IDs
> 
> 
> > > +		rproc_add_carveout(rproc, mem);
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_parse_fw()
> > > + * @rproc: single R5 core's corresponding rproc instance
> > > + * @fw: ptr to firmware to be loaded onto r5 core
> > > + *
> > > + * When loading firmware, ensure the necessary carveouts are in
> > remoteproc
> > > + *
> > > + * return 0 on success, otherwise non-zero value on failure
> > > + */
> > > +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware
> > *fw)
> > > +{
> > > +	int ret;
> > > +
> > > +	ret = parse_tcm_banks(rproc);
> > > +	if (ret)
> > > +		return ret;
> > > +
> > > +	ret = parse_mem_regions(rproc);
> > > +	if (ret)
> > > +		return ret;
> > > +
> > > +	ret = rproc_elf_load_rsc_table(rproc, fw);
> > > +	if (ret == -EINVAL) {
> > > +		/*
> > > +		 * resource table only required for IPC.
> > > +		 * if not present, this is not necessarily an error;
> > > +		 * for example, loading r5 hello world application
> > > +		 * so simply inform user and keep going.
> > > +		 */
> > > +		dev_info(&rproc->dev, "no resource table found.\n");
> > > +		ret = 0;
> > > +	}
> > > +	return ret;
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> > > + * @rproc: r5 core's corresponding rproc structure
> > > + * @vqid: virtqueue ID
> > > + */
> > > +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > +	struct device *dev = rproc->dev.parent;
> > > +	struct zynqmp_ipi_message *mb_msg = NULL;
> > > +	unsigned int skb_len;
> > > +	struct sk_buff *skb;
> > > +	int ret;
> > > +
> > > +	/* If vqid is negative, do not pass the vqid to
> > > +	 * mailbox as vqid is supposed to be 0 or positive.
> > > +	 * It also gives a way to just kick instead but
> > > +	 * not use the IPI buffer.
> > > +	 *
> > > +	 * For now use negative vqid to assume no message will be
> > > +	 * passed with IPI buffer, but just raise interrupt.
> > > +	 * This will be faster as it doesn't need to copy the
> > > +	 * message to the ZynqMP/Versal IPI message buffer.
> > > +	 */
> > > +	if (vqid >= 0) {
> > > +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> > > +		skb = alloc_skb(skb_len, GFP_KERNEL);
> > > +		if (!skb)
> > > +			return;
> > > +
> > > +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb,
> > skb_len);
> > > +		mb_msg->len = sizeof(vqid);
> > > +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> > > +
> > > +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> > > +	}
> > > +
> > > +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> > > +	if (ret < 0) {
> > > +		dev_warn(dev, "Failed to kick remote.\n");
> > > +		if (mb_msg) {
> > > +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> > > +			kfree_skb(skb);
> > > +		}
> > > +	}
> > > +}
> > > +
> >
> > I am done reviewing this set.
> >
> > Thanks,
> > Mathieu
> >
> > > +static struct rproc_ops zynqmp_r5_rproc_ops = {
> > > +	.start		= zynqmp_r5_rproc_start,
> > > +	.stop		= zynqmp_r5_rproc_stop,
> > > +	.load		= rproc_elf_load_segments,
> > > +	.parse_fw	= zynqmp_r5_parse_fw,
> > > +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > > +	.sanity_check	= rproc_elf_sanity_check,
> > > +	.get_boot_addr	= rproc_elf_get_boot_addr,
> > > +	.kick		= zynqmp_r5_rproc_kick,
> > > +};
> > > +
> > > +/**
> > > + * event_notified_idr_cb() - event notified idr callback
> > > + * @id: idr id
> > > + * @ptr: pointer to idr private data
> > > + * @data: data passed to idr_for_each callback
> > > + *
> > > + * Pass notification to remoteproc virtio
> > > + *
> > > + * Return: 0. having return is to satisfy the idr_for_each() function
> > > + *          pointer input argument requirement.
> > > + **/
> > > +static int event_notified_idr_cb(int id, void *ptr, void *data)
> > > +{
> > > +	struct rproc *rproc = data;
> > > +
> > > +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> > > +		dev_dbg(&rproc->dev, "no message was found in idr with id:
> > %d\n", id);
> > > +	return 0;
> > > +}
> > > +
> > > +/**
> > > + * handle_event_notified() - remoteproc notification work function
> > > + * @work: pointer to the work structure
> > > + *
> > > + * It checks each registered remoteproc notify IDs.
> > > + */
> > > +static void handle_event_notified(struct work_struct *work)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc;
> > > +	struct rproc *rproc;
> > > +
> > > +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> > > +	rproc = z_rproc->rproc;
> > > +
> > > +	/*
> > > +	 * This is to simply raise interrupt as ACK for remote.
> > > +	 *
> > > +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> > > +	 * mapping to remote processor.
> > > +	 */
> > > +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> > > +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> > > +
> > > +	/*
> > > +	 * We only use IPI for interrupt. The firmware side may or may
> > > +	 * not write the notifyid when it trigger IPI.
> > > +	 * And thus, we scan through all the registered notifyids.
> > > +	 */
> > > +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> > > +}
> > > +
> > > +/**
> > > + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> > > + * @cl: mailbox client
> > > + * @msg: message pointer
> > > + *
> > > + * It will schedule the R5 notification work.
> > > + */
> > > +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc;
> > > +
> > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> > > +	/*
> > > +	 * Notification can be received with message payload.
> > > +	 * To handle this, check for message contents.
> > > +	 * If there are contents, copy to ipi message buffer payload location.
> > > +	 *
> > > +	 * If message is empty, then that means that was simple raising
> > > +	 * of interrupt. no payload to process.
> > > +	 *
> > > +	 * Note: enqueue work regardless of msg as kick may imply that
> > remote
> > > +	 * is waiting for ack.
> > > +	 */
> > > +	if (msg) {
> > > +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> > > +		size_t len;
> > > +
> > > +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> > > +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> > > +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> > > +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> > > +		buf_msg->len = len;
> > > +		memcpy(buf_msg->data, ipi_msg->data, len);
> > > +	}
> > > +	schedule_work(&z_rproc->mbox_work);
> > > +}
> > > +
> > > +/**
> > > + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> > > + * @cl: mailbox client
> > > + * @msg: pointer to the message which has been sent
> > > + * @r: status of last TX - OK or error
> > > + *
> > > + * It will be called by the mailbox framework when the last TX has done.
> > > + */
> > > +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg,
> int
> > r)
> > > +{
> > > +	struct zynqmp_r5_rproc *z_rproc;
> > > +	struct sk_buff *skb;
> > > +
> > > +	if (!msg)
> > > +		return;
> > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> > > +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> > > +	kfree_skb(skb);
> > > +}
> > > +
> > > +/**
> > > + * zynqmp_r5_setup_mbox() - Setup mailboxes
> > > + *			    this is used for each individual R5 core
> > > + *
> > > + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> > > + * @node: pointer of the device node
> > > + *
> > > + * Function to setup mailboxes to talk to RPU.
> > > + *
> > > + * Return: 0 for success, negative value for failure.
> > > + */
> > > +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> > > +				struct device_node *node)
> > > +{
> > > +	struct mbox_client *mclient;
> > > +
> > > +	/* Setup TX mailbox channel client */
> > > +	mclient			= &z_rproc->tx_mc;
> > > +	mclient->dev		= z_rproc->dev;
> > > +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> > > +	mclient->tx_block	= false;
> > > +	mclient->knows_txdone	= false;
> > > +	mclient->rx_callback	= NULL;
> > > +
> > > +	/* Setup RX mailbox channel client */
> > > +	mclient			= &z_rproc->rx_mc;
> > > +	mclient->dev		= z_rproc->dev;
> > > +	mclient->tx_done	= NULL;
> > > +	mclient->tx_block	= false;
> > > +	mclient->knows_txdone	= false;
> > > +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> > > +
> > > +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> > > +
> > > +	/* Request TX and RX channels */
> > > +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc-
> > >tx_mc, "tx");
> > > +	if (IS_ERR(z_rproc->tx_chan)) {
> > > +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> > > +		return PTR_ERR(z_rproc->tx_chan);
> > > +	}
> > > +
> > > +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc-
> > >rx_mc, "rx");
> > > +	if (IS_ERR(z_rproc->rx_chan)) {
> > > +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> > > +		return PTR_ERR(z_rproc->rx_chan);
> > > +	}
> > > +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc *z_rproc)
> > > +{
> > > +	mbox_free_channel(z_rproc->tx_chan);
> > > +	mbox_free_channel(z_rproc->rx_chan);
> > > +}
> > > +
> > > +/**
> > > + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> > > + *		       this is called for each individual R5 core to
> > > + *		       set up mailbox, Xilinx platform manager unique ID,
> > > + *		       add to rproc core
> > > + *
> > > + * @pdev: domain platform device for current R5 core
> > > + * @node: pointer of the device node for current R5 core
> > > + * @rpu_mode: mode to configure RPU, split or lockstep
> > > + *
> > > + * Return: 0 for success, negative value for failure.
> > > + */
> > > +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device
> > *pdev,
> > > +					       struct device_node *node,
> > > +					       enum rpu_oper_mode
> > rpu_mode)
> > > +{
> > > +	int ret;
> > > +	struct device *dev = &pdev->dev;
> > > +	struct rproc *rproc_ptr;
> > > +	struct zynqmp_r5_rproc *z_rproc;
> > > +
> > > +	/* Allocate remoteproc instance */
> > > +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev),
> > &zynqmp_r5_rproc_ops,
> > > +				     NULL, sizeof(struct zynqmp_r5_rproc));
> > > +	if (!rproc_ptr) {
> > > +		ret = -ENOMEM;
> > > +		goto error;
> > > +	}
> > > +
> > > +	rproc_ptr->auto_boot = false;
> > > +	z_rproc = rproc_ptr->priv;
> > > +	z_rproc->rproc = rproc_ptr;
> > > +	z_rproc->dev = dev;
> > > +
> > > +	/* Set up DMA mask */
> > > +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> > > +	if (ret)
> > > +		goto error;
> > > +
> > > +	/* Get R5 power domain node */
> > > +	ret = of_property_read_u32(node, "power-domain", &z_rproc-
> > >pnode_id);
> > > +	if (ret)
> > > +		goto error;
> > > +
> > > +	ret = r5_set_mode(z_rproc, rpu_mode);
> > > +	if (ret)
> > > +		goto error;
> > > +
> > > +	if (of_property_read_bool(node, "mboxes")) {
> > > +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> > > +		if (ret)
> > > +			goto error;
> > > +	}
> > > +
> > > +	/* Add R5 remoteproc */
> > > +	ret = devm_rproc_add(dev, rproc_ptr);
> > > +	if (ret) {
> > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > +		goto error;
> > > +	}
> > > +
> > > +	return z_rproc;
> > > +error:
> > > +	return ERR_PTR(ret);
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_remoteproc_probe()
> > > + *
> > > + * @pdev: domain platform device for R5 cluster
> > > + *
> > > + * called when driver is probed, for each R5 core specified in DT,
> > > + * setup as needed to do remoteproc-related operations
> > > + *
> > > + * Return: 0 for success, negative value for failure.
> > > + */
> > > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > > +{
> > > +	int ret, core_count;
> > > +	struct device *dev = &pdev->dev;
> > > +	struct device_node *nc;
> > > +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> > > +	struct list_head *cluster; /* list to track each core's rproc */
> > > +	struct zynqmp_r5_rproc *z_rproc;
> > > +	struct platform_device *child_pdev;
> > > +	struct list_head *pos;
> > > +
> > > +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode",
> > &rpu_mode);
> > > +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> > > +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> > > +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> > > +			ret, rpu_mode);
> > > +		return ret;
> > > +	}
> > > +
> > > +	dev_dbg(dev, "RPU configuration: %s\n",
> > > +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" :
> > "split");
> > > +
> > > +	/*
> > > +	 * if 2 RPUs provided but one is lockstep, then we have an
> > > +	 * invalid configuration.
> > > +	 */
> > > +
> > > +	core_count = of_get_available_child_count(dev->of_node);
> > > +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> > > +	    core_count > MAX_RPROCS)
> > > +		return -EINVAL;
> > > +
> > > +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > > +	if (!cluster)
> > > +		return -ENOMEM;
> > > +	INIT_LIST_HEAD(cluster);
> > > +
> > > +	ret = devm_of_platform_populate(dev);
> > > +	if (ret) {
> > > +		dev_err(dev, "devm_of_platform_populate failed, ret =
> > %d\n",
> > > +			ret);
> > > +		return ret;
> > > +	}
> > > +
> > > +	/* probe each individual r5 core's remoteproc-related info */
> > > +	for_each_available_child_of_node(dev->of_node, nc) {
> > > +		child_pdev = of_find_device_by_node(nc);
> > > +		if (!child_pdev) {
> > > +			dev_err(dev, "could not get R5 core platform
> > device\n");
> > > +			ret = -ENODEV;
> > > +			goto out;
> > > +		}
> > > +
> > > +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> > > +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> > > +			ret ? "Failed" : "Able",
> > > +			nc);
> > > +		if (IS_ERR(z_rproc)) {
> > > +			ret = PTR_ERR(z_rproc);
> > > +			goto out;
> > > +		}
> > > +		list_add_tail(&z_rproc->elem, cluster);
> > > +	}
> > > +	/* wire in so each core can be cleaned up at driver remove */
> > > +	platform_set_drvdata(pdev, cluster);
> > > +	return 0;
> > > +out:
> > > +	/*
> > > +	 * undo core0 upon any failures on core1 in split-mode
> > > +	 *
> > > +	 * in zynqmp_r5_probe z_rproc is set to null
> > > +	 * and ret to non-zero value if error
> > > +	 */
> > > +	list_for_each(pos, cluster) {
> > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > +	}
> > > +	return ret;
> > > +}
> > > +
> > > +/*
> > > + * zynqmp_r5_remoteproc_remove()
> > > + *
> > > + * @pdev: domain platform device for R5 cluster
> > > + *
> > > + * When the driver is unloaded, clean up the mailboxes for each
> > > + * remoteproc that was initially probed.
> > > + */
> > > +static int zynqmp_r5_remoteproc_remove(struct platform_device *pdev)
> > > +{
> > > +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> > > +						 platform_get_drvdata(pdev);
> > > +	struct zynqmp_r5_rproc *z_rproc = NULL;
> > > +
> > > +	list_for_each_safe(pos, temp, cluster) {
> > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > +		list_del(pos);
> > > +	}
> > > +	return 0;
> > > +}
> > > +
> > > +/* Match table for OF platform binding */
> > > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > > +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> > > +	{ /* end of list */ },
> > > +};
> > > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > > +
> > > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > > +	.probe = zynqmp_r5_remoteproc_probe,
> > > +	.remove = zynqmp_r5_remoteproc_remove,
> > > +	.driver = {
> > > +		.name = "zynqmp_r5_remoteproc",
> > > +		.of_match_table = zynqmp_r5_remoteproc_match,
> > > +	},
> > > +};
> > > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > > +
> > > +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> > > +MODULE_LICENSE("GPL v2");
> > > --
> > > 2.17.1
> > >

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 28+ messages in thread

* RE: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
  2021-02-17 19:11         ` Ben Levinsky
@ 2021-02-17 21:34           ` Ben Levinsky
  -1 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-02-17 21:34 UTC (permalink / raw)
  To: Mathieu Poirier
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	Michal Simek

1 more for the pnode_id comment

> -----Original Message-----
> From: Ben Levinsky
> Sent: Wednesday, February 17, 2021 11:12 AM
> To: Mathieu Poirier <mathieu.poirier@linaro.org>
> Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal Simek
> <michals@xilinx.com>
> Subject: RE: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc
> driver
> 
> Hi Mathieu,
> 
> I worked on this further and had 1 more comment on the prepare/unprepare
> for TCM banks below
> 
> > -----Original Message-----
> > From: Ben Levinsky
> > Sent: Wednesday, February 17, 2021 8:47 AM
> > To: Mathieu Poirier <mathieu.poirier@linaro.org>
> > Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal Simek
> > <michals@xilinx.com>
> > Subject: RE: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc
> > driver
> >
> > Hi Mathieu,
> >
> > Please see my replies to your questions inline
> >
> > Again thanks for the continued review of this set
> >
> > Cheers
> > Ben
> >
> > > -----Original Message-----
> > > From: Mathieu Poirier <mathieu.poirier@linaro.org>
> > > Sent: Tuesday, February 16, 2021 10:48 AM
> > > To: Ben Levinsky <BLEVINSK@xilinx.com>
> > > Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> > > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal
> Simek
> > > <michals@xilinx.com>
> > > Subject: Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5
> remoteproc
> > > driver
> > >
> > > On Sun, Jan 10, 2021 at 06:02:50PM -0800, Ben Levinsky wrote:
> > > > R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
> > > > remotproc driver, we can boot the R5 sub-system in two different
> > > > configurations -
> > > > 	* Split
> > > > 	* Lockstep
> > > >
> > > > The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
> > > > Platform Management Unit that handles the R5 configuration, memory
> > > access
> > > > and R5 lifecycle management. The interface to this manager is done in
> this
> > > > driver via zynqmp_pm_* function calls.
> > > >
> > > > Signed-off-by: Wendy Liang <wendy.liang@xilinx.com>
> > > > Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> > > > Signed-off-by: Ed Mooring <ed.mooring@xilinx.com>
> > > > Signed-off-by: Jason Wu <j.wu@xilinx.com>
> > > > Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> > > > ---
> > > > v25:
> > > > - reword error message for rpu configuration
> > > > - change char name[15] to 16 chars to have null terminated string
> > > >   in parse_mem_regions()
> > > > - reword comments in tcm_mem_alloc
> > > > - call xilinx platform cleanup in parse_tcm_banks() case of failure
> > > > - check value of mbox_send_message in all instances within this driver
> > > > - indentation in zynqmp_r5_remoteproc driver
> > > > - update style of constructing carveouts to match convention in ST
> > > >   remoteproc drivers. No longer iterate through memory-region property
> > > >   using of_count_phandle_with_args. Instead use of_phandle_iterator
> > > > - use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per
> > > feedback
> > > > - rework loop in zynqmp_r5_pm_request_sram as per feedback
> > > > - fix comment and linebreak in tcm_mem_alloc description
> > > > - add comments and remove extraneous devm_ioremap_wc call in
> > > tcm_mem_alloc
> > > > - in parse_tcm_banks remove check for bank not being available
> > > > - rework order of locals and remove else in zynqmp_r5_rproc_kick
> > > > - document and update logic for zynqmp_r5_parse_fw
> > > > - add dev_dbg output in event_notified_idr_cb if
> > > >   rproc_vq_interrupt(rproc, id) == IRQ_NONE
> > > > - add comment for handle_event_notified mbox_send_message function
> > > call
> > > > - add comment for zynqmp_r5_mb_rx_cb
> > > > - update zynqmp_r5_setup_mbox to match convention of mbox setup in
> > ST
> > > >   remoteproc drivers
> > > > - change return in zynqmp_r5_setup_mbox to use PTR_ERR
> > > > - add zynqmp_r5_cleanup_mbox
> > > > - in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as
> > arg.
> > > > - in zynqmp_r5_probe use return of PTR_ERR and use
> > > zynqmp_r5_cleanup_mbox
> > > > - in zynqmp_r5_remoteproc_probe update use return value of
> > > zynqmp_r5_probe as
> > > >   either zynqmp_r5_rproc* or PTR_ERR
> > > > - update loop that cleans up cluster and mboxes in
> > > zynqmp_r5_remoteproc_probe
> > > > - update loop that cleans up cluster and mboxes in
> > > zynqmp_r5_remoteproc_remove
> > > > ---
> > > >  drivers/remoteproc/Kconfig                |   8 +
> > > >  drivers/remoteproc/Makefile               |   1 +
> > > >  drivers/remoteproc/zynqmp_r5_remoteproc.c | 898
> > > ++++++++++++++++++++++
> > > >  3 files changed, 907 insertions(+)
> > > >  create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c
> > > >
> > > > diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> > > > index c6659dfea7c7..c2fe54b1d94f 100644
> > > > --- a/drivers/remoteproc/Kconfig
> > > > +++ b/drivers/remoteproc/Kconfig
> > > > @@ -275,6 +275,14 @@ config TI_K3_DSP_REMOTEPROC
> > > >  	  It's safe to say N here if you're not interested in utilizing
> > > >  	  the DSP slave processors.
> > > >
> > > > +config ZYNQMP_R5_REMOTEPROC
> > > > +	tristate "ZynqMP R5 remoteproc support"
> > > > +	depends on PM && ARCH_ZYNQMP
> > > > +	select RPMSG_VIRTIO
> > > > +	select ZYNQMP_IPI_MBOX
> > > > +	help
> > > > +	  Say y or m here to support ZynqMP R5 remote processors via the
> > > remote
> > > > +	  processor framework.
> > > >  endif # REMOTEPROC
> > > >
> > > >  endmenu
> > > > diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> > > > index 3dfa28e6c701..ef1abff654c2 100644
> > > > --- a/drivers/remoteproc/Makefile
> > > > +++ b/drivers/remoteproc/Makefile
> > > > @@ -33,3 +33,4 @@ obj-$(CONFIG_ST_REMOTEPROC)		+=
> > > st_remoteproc.o
> > > >  obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
> > > >  obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
> > > >  obj-$(CONFIG_TI_K3_DSP_REMOTEPROC)	+= ti_k3_dsp_remoteproc.o
> > > > +obj-$(CONFIG_ZYNQMP_R5_REMOTEPROC)	+=
> zynqmp_r5_remoteproc.o
> > > > diff --git a/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > > b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > > > new file mode 100644
> > > > index 000000000000..9e228c9ecd43
> > > > --- /dev/null
> > > > +++ b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > > > @@ -0,0 +1,898 @@
> > > > +// SPDX-License-Identifier: GPL-2.0
> > > > +/*
> > > > + * Zynq R5 Remote Processor driver
> > > > + *
> > > > + * Based on origin OMAP and Zynq Remote Processor driver
> > > > + *
> > > > + */
> > > > +
> > > > +#include <linux/firmware/xlnx-zynqmp.h>
> > > > +#include <linux/interrupt.h>
> > > > +#include <linux/kernel.h>
> > > > +#include <linux/list.h>
> > > > +#include <linux/mailbox_client.h>
> > > > +#include <linux/mailbox/zynqmp-ipi-message.h>
> > > > +#include <linux/module.h>
> > > > +#include <linux/of_address.h>
> > > > +#include <linux/of_platform.h>
> > > > +#include <linux/of_reserved_mem.h>
> > > > +#include <linux/platform_device.h>
> > > > +#include <linux/remoteproc.h>
> > > > +#include <linux/skbuff.h>
> > > > +#include <linux/sysfs.h>
> > > > +
> > > > +#include "remoteproc_internal.h"
> > > > +
> > > > +#define MAX_RPROCS	2 /* Support up to 2 RPU */
> > > > +#define MAX_MEM_PNODES	4 /* Max power nodes for one RPU
> > memory
> > > instance */
> > > > +
> > > > +#define BANK_LIST_PROP	"sram"
> > > > +#define DDR_LIST_PROP	"memory-region"
> > > > +
> > > > +/* IPI buffer MAX length */
> > > > +#define IPI_BUF_LEN_MAX	32U
> > > > +/* RX mailbox client buffer max length */
> > > > +#define RX_MBOX_CLIENT_BUF_MAX	(IPI_BUF_LEN_MAX + \
> > > > +				 sizeof(struct zynqmp_ipi_message))
> > > > +
> > > > +/*
> > > > + * Map each Xilinx on-chip SRAM  Bank address to their own respective
> > > > + * pm_node_id.
> > > > + */
> > > > +struct sram_addr_data {
> > > > +	phys_addr_t addr;
> > > > +	enum pm_node_id id;
> > > > +};
> > > > +
> > > > +#define NUM_SRAMS 4U
> > > > +static const struct sram_addr_data zynqmp_banks[NUM_SRAMS] = {
> > > > +	{0xffe00000UL, NODE_TCM_0_A},
> > > > +	{0xffe20000UL, NODE_TCM_0_B},
> > > > +	{0xffe90000UL, NODE_TCM_1_A},
> > > > +	{0xffeb0000UL, NODE_TCM_1_B},
> > > > +};
> > > > +
> > > > +/**
> > > > + * struct zynqmp_r5_rproc - ZynqMP R5 core structure
> > > > + *
> > > > + * @rx_mc_buf: rx mailbox client buffer to save the rx message
> > > > + * @tx_mc: tx mailbox client
> > > > + * @rx_mc: rx mailbox client
> > > > + * @mbox_work: mbox_work for the RPU remoteproc
> > > > + * @tx_mc_skbs: socket buffers for tx mailbox client
> > > > + * @dev: device of RPU instance
> > > > + * @rproc: rproc handle
> > > > + * @tx_chan: tx mailbox channel
> > > > + * @rx_chan: rx mailbox channel
> > > > + * @pnode_id: RPU CPU power domain id
> > > > + * @elem: linked list item
> > > > + */
> > > > +struct zynqmp_r5_rproc {
> > > > +	unsigned char rx_mc_buf[RX_MBOX_CLIENT_BUF_MAX];
> > > > +	struct mbox_client tx_mc;
> > > > +	struct mbox_client rx_mc;
> > > > +	struct work_struct mbox_work;
> > > > +	struct sk_buff_head tx_mc_skbs;
> > > > +	struct device *dev;
> > > > +	struct rproc *rproc;
> > > > +	struct mbox_chan *tx_chan;
> > > > +	struct mbox_chan *rx_chan;
> > > > +	u32 pnode_id;
> > > > +	struct list_head elem;
> > > > +};
> > > > +
> > > > +/*
> > > > + * r5_set_mode - set RPU operation mode
> > > > + * @z_rproc: Remote processor private data
> > > > + * @rpu_mode: mode specified by device tree to configure the RPU to
> > > > + *
> > > > + * set RPU operation mode
> > > > + *
> > > > + * Return: 0 for success, negative value for failure
> > > > + */
> > > > +static int r5_set_mode(struct zynqmp_r5_rproc *z_rproc,
> > > > +		       enum rpu_oper_mode rpu_mode)
> > > > +{
> > > > +	enum rpu_tcm_comb tcm_mode;
> > > > +	enum rpu_oper_mode cur_rpu_mode;
> > > > +	int ret;
> > > > +
> > > > +	ret = zynqmp_pm_get_rpu_mode(z_rproc->pnode_id,
> > > &cur_rpu_mode);
> > > > +	if (ret < 0)
> > > > +		return ret;
> > > > +
> > > > +	if (rpu_mode != cur_rpu_mode) {
> > > > +		ret = zynqmp_pm_set_rpu_mode(z_rproc->pnode_id,
> > > rpu_mode);
> > > > +		if (ret < 0)
> > > > +			return ret;
> > > > +	}
> > > > +
> > > > +	tcm_mode = (rpu_mode == PM_RPU_MODE_LOCKSTEP) ?
> > > > +		    PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> > > > +	return zynqmp_pm_set_tcm_config(z_rproc->pnode_id, tcm_mode);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_mem_release
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to unmap
> > > > + *
> > > > + * Unmap TCM banks when powering down R5 core.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int tcm_mem_release(struct rproc *rproc, struct
> rproc_mem_entry
> > > *mem)
> > > > +{
> > > > +	u32 pnode_id = (u64)mem->priv;
> > > > +
> > > > +	iounmap(mem->va);
> > > > +	return zynqmp_pm_release_node(pnode_id);
> > >
> > > Is the opposite of that call zynqmp_pm_request_node()?  If so doing the
> > > power up
> > > and down of the TCM banks at driver probe() and remove() is highly
> > > inefficient.
> > > Consider using rproc_ops::prepare() and rproc_ops::unprepare().
> > >
> If I understand your comment correctly this is about handling the
> request_node() and release_node() calls. While its true that these two
> functions can be moved to prepare and unprepared I think these are still
> occurring not at probe nor remove.
> 
> For request_node() of the TCM bank this is occurring in parse_fw() in
> remoteproc_core as in the zynqmp_r5 driver shows below:
> 
> 	.parse_fw	= zynqmp_r5_parse_fw,
> 	...
> 	Zynqmp_parse_fw calls parse_tcm_banks
> 
> 	static int parse_tcm_banks(struct rproc *rproc){
> 		...
> 		ret = zynqmp_r5_pm_request_sram(rsc.start);
> 		...
>  		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> 					   (size_t)size, rsc.start,
> 					   tcm_mem_alloc,
> 					   tcm_mem_release,
> 					   rsc.name);
> 
> For release_node() this is occurring not at remove but instead is managed by
> remoteproc_core and tied to the lifecyle of the remoteproc carveout since
> release_node() is within the function tcm_mem_release() and therefore
> rproc_resource_cleanup() will end up triggering
> TCM bank cleanup.
> 
> 
> Still it is understandable to prefer prepare/unprepared to contain the 2 lines.
> 
> 
> If you still prefer prepare/unprepared is the following implementation ok?
> 
> prepare:
> 	Just move request_node() line here
> unprepare:
> 	similarly this is only moving the 1 line of release_node().
> 
> If this suggestion satisfies what you are saying I will do so, otherwise please
> correct my understanding
> 
> 
> Cheers
> Ben
> 
> 
> 
> > > > +}
> > >
> > > Please move this just after tcm_mem_alloc()
> > >
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_start
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Start R5 Core from designated boot address.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	enum rpu_boot_mem bootmem;
> > > > +
> > > > +	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
> > > > +		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> > > > +
> > > > +	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
> > > > +		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> > > > +
> > > > +	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
> > > > +				      bootmem,
> > > ZYNQMP_PM_REQUEST_ACK_NO);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_stop
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Power down  R5 Core.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +
> > > > +	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
> > > > +				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_mem_alloc
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to map
> > > > + *
> > > > + * Callback to map va for memory-region's carveout.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
> > > > +				     struct rproc_mem_entry *mem)
> > > > +{
> > > > +	void *va;
> > > > +
> > > > +	va = ioremap_wc(mem->dma, mem->len);
> > > > +	if (IS_ERR_OR_NULL(va))
> > > > +		return -ENOMEM;
> > > > +
> > > > +	mem->va = va;
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_mem_release
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to unmap
> > > > + *
> > > > + * Unmap memory-region carveout
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
> > > > +				       struct rproc_mem_entry *mem)
> > > > +{
> > > > +	iounmap(mem->va);
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * parse_mem_regions
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Construct rproc mem carveouts from carveout provided in
> > > > + * memory-region property
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int parse_mem_regions(struct rproc *rproc)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	struct device_node *np = z_rproc->dev->of_node;
> > > > +	struct device *dev = &rproc->dev;
> > > > +	struct of_phandle_iterator it;
> > > > +	struct rproc_mem_entry *mem;
> > > > +	struct reserved_mem *rmem;
> > > > +	int index = 0;
> > > > +
> > > > +	/* Register associated reserved memory regions */
> > > > +	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
> > > > +	while (of_phandle_iterator_next(&it) == 0) {
> > > > +		rmem = of_reserved_mem_lookup(it.node);
> > > > +		if (!rmem) {
> > > > +			dev_err(dev, "unable to acquire %s\n",
> > > DDR_LIST_PROP);
> > > > +			return -EINVAL;
> > > > +		}
> > > > +
> > > > +		if (strstr(it.node->name, "vdev0buffer")) {
> > > > +			mem = rproc_of_resm_mem_entry_init(dev, index,
> > > > +							   rmem->size,
> > > > +							   rmem->base,
> > > > +							   "vdev0buffer");
> > > > +		} else {
> > > > +			/*
> > > > +			 * The ensuing carveout is either for vring, which has
> > > > +			 * device tree node name of length 15 characters, or
> > > > +			 * firmware mem, where name is not used by
> > > remoteproc
> > > > +			 *  core later on. So default to vring length of 15.
> > >
> > > s/"*  core"/"* core"
> > >
> > > > +			 *
> > > > +			 * Extra char for null-terminated string.
> > > > +			 */
> > > > +			char name[16];
> > > > +
> > > > +			if (strstr(it.node->name, "vdev0vring")) {
> > > > +				int vring_id;
> > > > +
> > > > +				/*
> > > > +				 * Expect "rpuXvdev0vringX" as documented
> > > > +				 * in xilinx remoteproc device tree binding
> > > > +				 */
> > > > +				if (strlen(it.node->name) < 15) {
> > > > +					dev_err(dev, "%pOF is less than 15
> > > chars",
> > > > +						it.node);
> > > > +					return -EINVAL;
> > > > +				}
> > > > +
> > > > +				/*
> > > > +				 * can be 1 of multiple vring IDs per IPC
> > > channel
> > > > +				 * e.g. 'vdev0vring0' and 'vdev0vring1'
> > > > +				 *
> > > > +				 * Here get vring ID from 15th char
> > > > +				 */
> > > > +				vring_id = it.node->name[14] - '0';
> > > > +				snprintf(name, sizeof(name), "vdev0vring%d",
> > > vring_id);
> > >
> > > Function strstr() returns a pointer to the beginning of the string being
> > > searched for.  As such you should be able to do something like:
> > >
> > >                         name = strstr(it.node->name, "vdev0vring");
> > >                         if (name) {
> > >                                 ...
> > >
> > > and simply use @name when calling rproc_mem_entry_init().
> > >
> > > > +			} else {
> > > > +				strncpy(name, it.node->name, 16);
> > > > +			}
> > > > +
> > > > +			mem = rproc_mem_entry_init(dev, NULL,
> > > > +						   (dma_addr_t)rmem->base,
> > > > +						   rmem->size, rmem->base,
> > > > +
> > > zynqmp_r5_rproc_mem_alloc,
> > > > +
> > > zynqmp_r5_rproc_mem_release,
> > > > +						   name);
> > > > +		}
> > > > +
> > > > +		if (!mem)
> > > > +			return -ENOMEM;
> > > > +
> > > > +		rproc_add_carveout(rproc, mem);
> > > > +		index++;
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_pm_request_tcm
> > > > + * @addr: base address of mem provided in R5 core's sram property.
> > > > + *
> > > > + * Given sram base address, determine its corresponding Xilinx
> > > > + * Platform Management ID and then request access to this node
> > > > + * so that it can be power up.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
> > > > +{
> > > > +	int i;
> > > > +
> > > > +	for (i = 0; i < NUM_SRAMS; i++) {
> > > > +		if (zynqmp_banks[i].addr != addr)
> > > > +			continue;
> > > > +
> > > > +		return zynqmp_pm_request_node(zynqmp_banks[i].id,
> > > > +					      ZYNQMP_PM_CAPABILITY_ACCESS,
> > > 0,
> > > > +
> > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > +	}
> > > > +
> > > > +	return -EINVAL;
> > > > +}
> > > > +
> > > > +/*
> > > > + * tcm_mem_alloc
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to initialize the va and da fields of
> > > > + *
> > > > + * Given TCM bank entry, this callback will set device address for R5
> > > > + * running on TCM and also setup virtual address for TCM bank
> > > > + * remoteproc carveout.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int tcm_mem_alloc(struct rproc *rproc,
> > > > +			 struct rproc_mem_entry *mem)
> > > > +{
> > > > +	void *va;
> > > > +	struct device *dev = rproc->dev.parent;
> > > > +
> > > > +	va = ioremap_wc(mem->dma, mem->len);
> > > > +	if (IS_ERR_OR_NULL(va))
> > > > +		return -ENOMEM;
> > > > +
> > > > +	/* Update memory entry va */
> > > > +	mem->va = va;
> > > > +
> > > > +	/*
> > > > +	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> > > > +	 * while on the Linux side they are at 0xffexxxxx.
> > > > +	 *
> > > > +	 * Zero out the high 12 bits of the address. This will give
> > > > +	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> > > > +	 */
> > > > +	mem->da &= 0x000fffff;
> > > > +
> > > > +	/*
> > > > +	 * TCM Banks 1A and 1B still have to be translated.
> > > > +	 *
> > > > +	 * Below handle these two banks' absolute addresses (0xffe90000 and
> > > > +	 * 0xffeb0000) and convert to the expected relative addresses
> > > > +	 * (0x0 and 0x20000).
> > > > +	 */
> > > > +	if (mem->da == 0x90000 || mem->da == 0xB0000)
> > > > +		mem->da -= 0x90000;
> > > > +
> > > > +	/* if translated TCM bank address is not valid report error */
> > > > +	if (mem->da != 0x0 && mem->da != 0x20000) {
> > > > +		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
> > > > +		return -EINVAL;
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * parse_tcm_banks()
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Given R5 node in remoteproc instance
> > > > + * allocate remoteproc carveout for TCM memory
> > > > + * needed for firmware to be loaded
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int parse_tcm_banks(struct rproc *rproc)
> > > > +{
> > > > +	int i, num_banks;
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	struct device *dev = &rproc->dev;
> > > > +	struct device_node *r5_node = z_rproc->dev->of_node;
> > > > +
> > > > +	/* go through TCM banks for r5 node */
> > > > +	num_banks = of_count_phandle_with_args(r5_node,
> > > BANK_LIST_PROP, NULL);
> > > > +	if (num_banks <= 0) {
> > > > +		dev_err(dev, "need to specify TCM banks\n");
> > > > +		return -EINVAL;
> > > > +	}
> > > > +	for (i = 0; i < num_banks; i++) {
> > > > +		struct resource rsc;
> > > > +		resource_size_t size;
> > > > +		struct device_node *dt_node;
> > > > +		struct rproc_mem_entry *mem;
> > > > +		int ret;
> > > > +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> > > > +
> > > > +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> > > > +		if (!dt_node)
> > > > +			return -EINVAL;
> > > > +
> > > > +		ret = of_address_to_resource(dt_node, 0, &rsc);
> > > > +		if (ret < 0) {
> > > > +			of_node_put(dt_node);
> > > > +			return ret;
> > > > +		}
> > > > +		of_node_put(dt_node);
> > > > +
> > > > +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> > > > +		if (ret < 0)
> > > > +			return ret;
> > > > +
> > > > +		/* add carveout */
> > > > +		size = resource_size(&rsc);
> > > > +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> > > > +					   (size_t)size, rsc.start,
> > > > +					   tcm_mem_alloc,
> > > > +					   tcm_mem_release,
> > > > +					   rsc.name);
> > > > +		if (!mem) {
> > > > +			ret = zynqmp_pm_release_node(pnode_id);
> > >
> > > Where is @pnode_id initialised?
> >
> > Good catch. Previously this was value set in an earlier iteration. I will fix this
> > as follows:
> > - release the R5 cluster by instead passing z_rproc->pnode_id
> > - release the TCM bank via call to release_sram
> >
> > This will also remove the variable pnode_id.
> >


typo release_sram should be release_node()

also instead another, cleaner way might be as follows:
change zynqmp_r5_request_sram so that it takes 1 more arg to return by reference the TCM bank's Xilinx platform management ID and then set the TCM bank's ID with that information.

Thanks
Ben

> > >
> > > > +			if (ret)
> > > > +				dev_warn(dev,
> > > > +					 "fail to release node: %x ret: %x\n",
> > > > +					 pnode_id, ret);
> > > > +			return -ENOMEM;
> > > > +		}
> > > > +
> > > > +		mem->priv = (void *)(u64)pnode_id;
> > >
> > >
> > > And here too - how does this work when you test things out on your side?
> > >
> > > Please note that I will need a reply to these questions before I review
> > another
> > > set.
> > >
> > Here I will update as follows:
> > - set mem->priv to value from zynqmp_banks lookup table that contains
> > power node IDs
> >
> >
> > > > +		rproc_add_carveout(rproc, mem);
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_parse_fw()
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @fw: ptr to firmware to be loaded onto r5 core
> > > > + *
> > > > + * When loading firmware, ensure the necessary carveouts are in
> > > remoteproc
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct
> firmware
> > > *fw)
> > > > +{
> > > > +	int ret;
> > > > +
> > > > +	ret = parse_tcm_banks(rproc);
> > > > +	if (ret)
> > > > +		return ret;
> > > > +
> > > > +	ret = parse_mem_regions(rproc);
> > > > +	if (ret)
> > > > +		return ret;
> > > > +
> > > > +	ret = rproc_elf_load_rsc_table(rproc, fw);
> > > > +	if (ret == -EINVAL) {
> > > > +		/*
> > > > +		 * resource table only required for IPC.
> > > > +		 * if not present, this is not necessarily an error;
> > > > +		 * for example, loading r5 hello world application
> > > > +		 * so simply inform user and keep going.
> > > > +		 */
> > > > +		dev_info(&rproc->dev, "no resource table found.\n");
> > > > +		ret = 0;
> > > > +	}
> > > > +	return ret;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> > > > + * @rproc: r5 core's corresponding rproc structure
> > > > + * @vqid: virtqueue ID
> > > > + */
> > > > +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	struct device *dev = rproc->dev.parent;
> > > > +	struct zynqmp_ipi_message *mb_msg = NULL;
> > > > +	unsigned int skb_len;
> > > > +	struct sk_buff *skb;
> > > > +	int ret;
> > > > +
> > > > +	/* If vqid is negative, do not pass the vqid to
> > > > +	 * mailbox as vqid is supposed to be 0 or positive.
> > > > +	 * It also gives a way to just kick instead but
> > > > +	 * not use the IPI buffer.
> > > > +	 *
> > > > +	 * For now use negative vqid to assume no message will be
> > > > +	 * passed with IPI buffer, but just raise interrupt.
> > > > +	 * This will be faster as it doesn't need to copy the
> > > > +	 * message to the ZynqMP/Versal IPI message buffer.
> > > > +	 */
> > > > +	if (vqid >= 0) {
> > > > +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> > > > +		skb = alloc_skb(skb_len, GFP_KERNEL);
> > > > +		if (!skb)
> > > > +			return;
> > > > +
> > > > +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb,
> > > skb_len);
> > > > +		mb_msg->len = sizeof(vqid);
> > > > +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> > > > +
> > > > +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> > > > +	}
> > > > +
> > > > +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> > > > +	if (ret < 0) {
> > > > +		dev_warn(dev, "Failed to kick remote.\n");
> > > > +		if (mb_msg) {
> > > > +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> > > > +			kfree_skb(skb);
> > > > +		}
> > > > +	}
> > > > +}
> > > > +
> > >
> > > I am done reviewing this set.
> > >
> > > Thanks,
> > > Mathieu
> > >
> > > > +static struct rproc_ops zynqmp_r5_rproc_ops = {
> > > > +	.start		= zynqmp_r5_rproc_start,
> > > > +	.stop		= zynqmp_r5_rproc_stop,
> > > > +	.load		= rproc_elf_load_segments,
> > > > +	.parse_fw	= zynqmp_r5_parse_fw,
> > > > +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > > > +	.sanity_check	= rproc_elf_sanity_check,
> > > > +	.get_boot_addr	= rproc_elf_get_boot_addr,
> > > > +	.kick		= zynqmp_r5_rproc_kick,
> > > > +};
> > > > +
> > > > +/**
> > > > + * event_notified_idr_cb() - event notified idr callback
> > > > + * @id: idr id
> > > > + * @ptr: pointer to idr private data
> > > > + * @data: data passed to idr_for_each callback
> > > > + *
> > > > + * Pass notification to remoteproc virtio
> > > > + *
> > > > + * Return: 0. having return is to satisfy the idr_for_each() function
> > > > + *          pointer input argument requirement.
> > > > + **/
> > > > +static int event_notified_idr_cb(int id, void *ptr, void *data)
> > > > +{
> > > > +	struct rproc *rproc = data;
> > > > +
> > > > +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> > > > +		dev_dbg(&rproc->dev, "no message was found in idr with id:
> > > %d\n", id);
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/**
> > > > + * handle_event_notified() - remoteproc notification work function
> > > > + * @work: pointer to the work structure
> > > > + *
> > > > + * It checks each registered remoteproc notify IDs.
> > > > + */
> > > > +static void handle_event_notified(struct work_struct *work)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +	struct rproc *rproc;
> > > > +
> > > > +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> > > > +	rproc = z_rproc->rproc;
> > > > +
> > > > +	/*
> > > > +	 * This is to simply raise interrupt as ACK for remote.
> > > > +	 *
> > > > +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> > > > +	 * mapping to remote processor.
> > > > +	 */
> > > > +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> > > > +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> > > > +
> > > > +	/*
> > > > +	 * We only use IPI for interrupt. The firmware side may or may
> > > > +	 * not write the notifyid when it trigger IPI.
> > > > +	 * And thus, we scan through all the registered notifyids.
> > > > +	 */
> > > > +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> > > > + * @cl: mailbox client
> > > > + * @msg: message pointer
> > > > + *
> > > > + * It will schedule the R5 notification work.
> > > > + */
> > > > +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +
> > > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> > > > +	/*
> > > > +	 * Notification can be received with message payload.
> > > > +	 * To handle this, check for message contents.
> > > > +	 * If there are contents, copy to ipi message buffer payload location.
> > > > +	 *
> > > > +	 * If message is empty, then that means that was simple raising
> > > > +	 * of interrupt. no payload to process.
> > > > +	 *
> > > > +	 * Note: enqueue work regardless of msg as kick may imply that
> > > remote
> > > > +	 * is waiting for ack.
> > > > +	 */
> > > > +	if (msg) {
> > > > +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> > > > +		size_t len;
> > > > +
> > > > +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> > > > +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> > > > +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> > > > +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> > > > +		buf_msg->len = len;
> > > > +		memcpy(buf_msg->data, ipi_msg->data, len);
> > > > +	}
> > > > +	schedule_work(&z_rproc->mbox_work);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> > > > + * @cl: mailbox client
> > > > + * @msg: pointer to the message which has been sent
> > > > + * @r: status of last TX - OK or error
> > > > + *
> > > > + * It will be called by the mailbox framework when the last TX has
> done.
> > > > + */
> > > > +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg,
> > int
> > > r)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +	struct sk_buff *skb;
> > > > +
> > > > +	if (!msg)
> > > > +		return;
> > > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> > > > +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> > > > +	kfree_skb(skb);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_setup_mbox() - Setup mailboxes
> > > > + *			    this is used for each individual R5 core
> > > > + *
> > > > + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> > > > + * @node: pointer of the device node
> > > > + *
> > > > + * Function to setup mailboxes to talk to RPU.
> > > > + *
> > > > + * Return: 0 for success, negative value for failure.
> > > > + */
> > > > +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> > > > +				struct device_node *node)
> > > > +{
> > > > +	struct mbox_client *mclient;
> > > > +
> > > > +	/* Setup TX mailbox channel client */
> > > > +	mclient			= &z_rproc->tx_mc;
> > > > +	mclient->dev		= z_rproc->dev;
> > > > +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> > > > +	mclient->tx_block	= false;
> > > > +	mclient->knows_txdone	= false;
> > > > +	mclient->rx_callback	= NULL;
> > > > +
> > > > +	/* Setup RX mailbox channel client */
> > > > +	mclient			= &z_rproc->rx_mc;
> > > > +	mclient->dev		= z_rproc->dev;
> > > > +	mclient->tx_done	= NULL;
> > > > +	mclient->tx_block	= false;
> > > > +	mclient->knows_txdone	= false;
> > > > +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> > > > +
> > > > +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> > > > +
> > > > +	/* Request TX and RX channels */
> > > > +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc-
> > > >tx_mc, "tx");
> > > > +	if (IS_ERR(z_rproc->tx_chan)) {
> > > > +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> > > > +		return PTR_ERR(z_rproc->tx_chan);
> > > > +	}
> > > > +
> > > > +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc-
> > > >rx_mc, "rx");
> > > > +	if (IS_ERR(z_rproc->rx_chan)) {
> > > > +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> > > > +		return PTR_ERR(z_rproc->rx_chan);
> > > > +	}
> > > > +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc
> *z_rproc)
> > > > +{
> > > > +	mbox_free_channel(z_rproc->tx_chan);
> > > > +	mbox_free_channel(z_rproc->rx_chan);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> > > > + *		       this is called for each individual R5 core to
> > > > + *		       set up mailbox, Xilinx platform manager unique
> ID,
> > > > + *		       add to rproc core
> > > > + *
> > > > + * @pdev: domain platform device for current R5 core
> > > > + * @node: pointer of the device node for current R5 core
> > > > + * @rpu_mode: mode to configure RPU, split or lockstep
> > > > + *
> > > > + * Return: 0 for success, negative value for failure.
> > > > + */
> > > > +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct
> platform_device
> > > *pdev,
> > > > +					       struct device_node *node,
> > > > +					       enum rpu_oper_mode
> > > rpu_mode)
> > > > +{
> > > > +	int ret;
> > > > +	struct device *dev = &pdev->dev;
> > > > +	struct rproc *rproc_ptr;
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +
> > > > +	/* Allocate remoteproc instance */
> > > > +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev),
> > > &zynqmp_r5_rproc_ops,
> > > > +				     NULL, sizeof(struct zynqmp_r5_rproc));
> > > > +	if (!rproc_ptr) {
> > > > +		ret = -ENOMEM;
> > > > +		goto error;
> > > > +	}
> > > > +
> > > > +	rproc_ptr->auto_boot = false;
> > > > +	z_rproc = rproc_ptr->priv;
> > > > +	z_rproc->rproc = rproc_ptr;
> > > > +	z_rproc->dev = dev;
> > > > +
> > > > +	/* Set up DMA mask */
> > > > +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> > > > +	if (ret)
> > > > +		goto error;
> > > > +
> > > > +	/* Get R5 power domain node */
> > > > +	ret = of_property_read_u32(node, "power-domain", &z_rproc-
> > > >pnode_id);
> > > > +	if (ret)
> > > > +		goto error;
> > > > +
> > > > +	ret = r5_set_mode(z_rproc, rpu_mode);
> > > > +	if (ret)
> > > > +		goto error;
> > > > +
> > > > +	if (of_property_read_bool(node, "mboxes")) {
> > > > +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> > > > +		if (ret)
> > > > +			goto error;
> > > > +	}
> > > > +
> > > > +	/* Add R5 remoteproc */
> > > > +	ret = devm_rproc_add(dev, rproc_ptr);
> > > > +	if (ret) {
> > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > +		goto error;
> > > > +	}
> > > > +
> > > > +	return z_rproc;
> > > > +error:
> > > > +	return ERR_PTR(ret);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_remoteproc_probe()
> > > > + *
> > > > + * @pdev: domain platform device for R5 cluster
> > > > + *
> > > > + * called when driver is probed, for each R5 core specified in DT,
> > > > + * setup as needed to do remoteproc-related operations
> > > > + *
> > > > + * Return: 0 for success, negative value for failure.
> > > > + */
> > > > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > > > +{
> > > > +	int ret, core_count;
> > > > +	struct device *dev = &pdev->dev;
> > > > +	struct device_node *nc;
> > > > +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> > > > +	struct list_head *cluster; /* list to track each core's rproc */
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +	struct platform_device *child_pdev;
> > > > +	struct list_head *pos;
> > > > +
> > > > +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode",
> > > &rpu_mode);
> > > > +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> > > > +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> > > > +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> > > > +			ret, rpu_mode);
> > > > +		return ret;
> > > > +	}
> > > > +
> > > > +	dev_dbg(dev, "RPU configuration: %s\n",
> > > > +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" :
> > > "split");
> > > > +
> > > > +	/*
> > > > +	 * if 2 RPUs provided but one is lockstep, then we have an
> > > > +	 * invalid configuration.
> > > > +	 */
> > > > +
> > > > +	core_count = of_get_available_child_count(dev->of_node);
> > > > +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> > > > +	    core_count > MAX_RPROCS)
> > > > +		return -EINVAL;
> > > > +
> > > > +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > > > +	if (!cluster)
> > > > +		return -ENOMEM;
> > > > +	INIT_LIST_HEAD(cluster);
> > > > +
> > > > +	ret = devm_of_platform_populate(dev);
> > > > +	if (ret) {
> > > > +		dev_err(dev, "devm_of_platform_populate failed, ret =
> > > %d\n",
> > > > +			ret);
> > > > +		return ret;
> > > > +	}
> > > > +
> > > > +	/* probe each individual r5 core's remoteproc-related info */
> > > > +	for_each_available_child_of_node(dev->of_node, nc) {
> > > > +		child_pdev = of_find_device_by_node(nc);
> > > > +		if (!child_pdev) {
> > > > +			dev_err(dev, "could not get R5 core platform
> > > device\n");
> > > > +			ret = -ENODEV;
> > > > +			goto out;
> > > > +		}
> > > > +
> > > > +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> > > > +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> > > > +			ret ? "Failed" : "Able",
> > > > +			nc);
> > > > +		if (IS_ERR(z_rproc)) {
> > > > +			ret = PTR_ERR(z_rproc);
> > > > +			goto out;
> > > > +		}
> > > > +		list_add_tail(&z_rproc->elem, cluster);
> > > > +	}
> > > > +	/* wire in so each core can be cleaned up at driver remove */
> > > > +	platform_set_drvdata(pdev, cluster);
> > > > +	return 0;
> > > > +out:
> > > > +	/*
> > > > +	 * undo core0 upon any failures on core1 in split-mode
> > > > +	 *
> > > > +	 * in zynqmp_r5_probe z_rproc is set to null
> > > > +	 * and ret to non-zero value if error
> > > > +	 */
> > > > +	list_for_each(pos, cluster) {
> > > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > +	}
> > > > +	return ret;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_remoteproc_remove()
> > > > + *
> > > > + * @pdev: domain platform device for R5 cluster
> > > > + *
> > > > + * When the driver is unloaded, clean up the mailboxes for each
> > > > + * remoteproc that was initially probed.
> > > > + */
> > > > +static int zynqmp_r5_remoteproc_remove(struct platform_device
> *pdev)
> > > > +{
> > > > +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> > > > +						 platform_get_drvdata(pdev);
> > > > +	struct zynqmp_r5_rproc *z_rproc = NULL;
> > > > +
> > > > +	list_for_each_safe(pos, temp, cluster) {
> > > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > +		list_del(pos);
> > > > +	}
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/* Match table for OF platform binding */
> > > > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > > > +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> > > > +	{ /* end of list */ },
> > > > +};
> > > > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > > > +
> > > > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > > > +	.probe = zynqmp_r5_remoteproc_probe,
> > > > +	.remove = zynqmp_r5_remoteproc_remove,
> > > > +	.driver = {
> > > > +		.name = "zynqmp_r5_remoteproc",
> > > > +		.of_match_table = zynqmp_r5_remoteproc_match,
> > > > +	},
> > > > +};
> > > > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > > > +
> > > > +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> > > > +MODULE_LICENSE("GPL v2");
> > > > --
> > > > 2.17.1
> > > >

^ permalink raw reply	[flat|nested] 28+ messages in thread

* RE: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
@ 2021-02-17 21:34           ` Ben Levinsky
  0 siblings, 0 replies; 28+ messages in thread
From: Ben Levinsky @ 2021-02-17 21:34 UTC (permalink / raw)
  To: Mathieu Poirier
  Cc: devicetree, Michal Simek, linux-remoteproc, linux-kernel,
	linux-arm-kernel

1 more for the pnode_id comment

> -----Original Message-----
> From: Ben Levinsky
> Sent: Wednesday, February 17, 2021 11:12 AM
> To: Mathieu Poirier <mathieu.poirier@linaro.org>
> Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal Simek
> <michals@xilinx.com>
> Subject: RE: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc
> driver
> 
> Hi Mathieu,
> 
> I worked on this further and had 1 more comment on the prepare/unprepare
> for TCM banks below
> 
> > -----Original Message-----
> > From: Ben Levinsky
> > Sent: Wednesday, February 17, 2021 8:47 AM
> > To: Mathieu Poirier <mathieu.poirier@linaro.org>
> > Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal Simek
> > <michals@xilinx.com>
> > Subject: RE: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc
> > driver
> >
> > Hi Mathieu,
> >
> > Please see my replies to your questions inline
> >
> > Again thanks for the continued review of this set
> >
> > Cheers
> > Ben
> >
> > > -----Original Message-----
> > > From: Mathieu Poirier <mathieu.poirier@linaro.org>
> > > Sent: Tuesday, February 16, 2021 10:48 AM
> > > To: Ben Levinsky <BLEVINSK@xilinx.com>
> > > Cc: devicetree@vger.kernel.org; linux-remoteproc@vger.kernel.org; linux-
> > > kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Michal
> Simek
> > > <michals@xilinx.com>
> > > Subject: Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5
> remoteproc
> > > driver
> > >
> > > On Sun, Jan 10, 2021 at 06:02:50PM -0800, Ben Levinsky wrote:
> > > > R5 is included in Xilinx Zynq UltraScale MPSoC so by adding this
> > > > remotproc driver, we can boot the R5 sub-system in two different
> > > > configurations -
> > > > 	* Split
> > > > 	* Lockstep
> > > >
> > > > The Xilinx R5 Remoteproc Driver boots the R5's via calls to the Xilinx
> > > > Platform Management Unit that handles the R5 configuration, memory
> > > access
> > > > and R5 lifecycle management. The interface to this manager is done in
> this
> > > > driver via zynqmp_pm_* function calls.
> > > >
> > > > Signed-off-by: Wendy Liang <wendy.liang@xilinx.com>
> > > > Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> > > > Signed-off-by: Ed Mooring <ed.mooring@xilinx.com>
> > > > Signed-off-by: Jason Wu <j.wu@xilinx.com>
> > > > Signed-off-by: Ben Levinsky <ben.levinsky@xilinx.com>
> > > > ---
> > > > v25:
> > > > - reword error message for rpu configuration
> > > > - change char name[15] to 16 chars to have null terminated string
> > > >   in parse_mem_regions()
> > > > - reword comments in tcm_mem_alloc
> > > > - call xilinx platform cleanup in parse_tcm_banks() case of failure
> > > > - check value of mbox_send_message in all instances within this driver
> > > > - indentation in zynqmp_r5_remoteproc driver
> > > > - update style of constructing carveouts to match convention in ST
> > > >   remoteproc drivers. No longer iterate through memory-region property
> > > >   using of_count_phandle_with_args. Instead use of_phandle_iterator
> > > > - use rproc_of_resm_mem_entry_init for vdev0buffer carveout as per
> > > feedback
> > > > - rework loop in zynqmp_r5_pm_request_sram as per feedback
> > > > - fix comment and linebreak in tcm_mem_alloc description
> > > > - add comments and remove extraneous devm_ioremap_wc call in
> > > tcm_mem_alloc
> > > > - in parse_tcm_banks remove check for bank not being available
> > > > - rework order of locals and remove else in zynqmp_r5_rproc_kick
> > > > - document and update logic for zynqmp_r5_parse_fw
> > > > - add dev_dbg output in event_notified_idr_cb if
> > > >   rproc_vq_interrupt(rproc, id) == IRQ_NONE
> > > > - add comment for handle_event_notified mbox_send_message function
> > > call
> > > > - add comment for zynqmp_r5_mb_rx_cb
> > > > - update zynqmp_r5_setup_mbox to match convention of mbox setup in
> > ST
> > > >   remoteproc drivers
> > > > - change return in zynqmp_r5_setup_mbox to use PTR_ERR
> > > > - add zynqmp_r5_cleanup_mbox
> > > > - in zynqmp_r5_probe, reteurn zynqmp_r5_rproc instead of taking in as
> > arg.
> > > > - in zynqmp_r5_probe use return of PTR_ERR and use
> > > zynqmp_r5_cleanup_mbox
> > > > - in zynqmp_r5_remoteproc_probe update use return value of
> > > zynqmp_r5_probe as
> > > >   either zynqmp_r5_rproc* or PTR_ERR
> > > > - update loop that cleans up cluster and mboxes in
> > > zynqmp_r5_remoteproc_probe
> > > > - update loop that cleans up cluster and mboxes in
> > > zynqmp_r5_remoteproc_remove
> > > > ---
> > > >  drivers/remoteproc/Kconfig                |   8 +
> > > >  drivers/remoteproc/Makefile               |   1 +
> > > >  drivers/remoteproc/zynqmp_r5_remoteproc.c | 898
> > > ++++++++++++++++++++++
> > > >  3 files changed, 907 insertions(+)
> > > >  create mode 100644 drivers/remoteproc/zynqmp_r5_remoteproc.c
> > > >
> > > > diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
> > > > index c6659dfea7c7..c2fe54b1d94f 100644
> > > > --- a/drivers/remoteproc/Kconfig
> > > > +++ b/drivers/remoteproc/Kconfig
> > > > @@ -275,6 +275,14 @@ config TI_K3_DSP_REMOTEPROC
> > > >  	  It's safe to say N here if you're not interested in utilizing
> > > >  	  the DSP slave processors.
> > > >
> > > > +config ZYNQMP_R5_REMOTEPROC
> > > > +	tristate "ZynqMP R5 remoteproc support"
> > > > +	depends on PM && ARCH_ZYNQMP
> > > > +	select RPMSG_VIRTIO
> > > > +	select ZYNQMP_IPI_MBOX
> > > > +	help
> > > > +	  Say y or m here to support ZynqMP R5 remote processors via the
> > > remote
> > > > +	  processor framework.
> > > >  endif # REMOTEPROC
> > > >
> > > >  endmenu
> > > > diff --git a/drivers/remoteproc/Makefile b/drivers/remoteproc/Makefile
> > > > index 3dfa28e6c701..ef1abff654c2 100644
> > > > --- a/drivers/remoteproc/Makefile
> > > > +++ b/drivers/remoteproc/Makefile
> > > > @@ -33,3 +33,4 @@ obj-$(CONFIG_ST_REMOTEPROC)		+=
> > > st_remoteproc.o
> > > >  obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
> > > >  obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
> > > >  obj-$(CONFIG_TI_K3_DSP_REMOTEPROC)	+= ti_k3_dsp_remoteproc.o
> > > > +obj-$(CONFIG_ZYNQMP_R5_REMOTEPROC)	+=
> zynqmp_r5_remoteproc.o
> > > > diff --git a/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > > b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > > > new file mode 100644
> > > > index 000000000000..9e228c9ecd43
> > > > --- /dev/null
> > > > +++ b/drivers/remoteproc/zynqmp_r5_remoteproc.c
> > > > @@ -0,0 +1,898 @@
> > > > +// SPDX-License-Identifier: GPL-2.0
> > > > +/*
> > > > + * Zynq R5 Remote Processor driver
> > > > + *
> > > > + * Based on origin OMAP and Zynq Remote Processor driver
> > > > + *
> > > > + */
> > > > +
> > > > +#include <linux/firmware/xlnx-zynqmp.h>
> > > > +#include <linux/interrupt.h>
> > > > +#include <linux/kernel.h>
> > > > +#include <linux/list.h>
> > > > +#include <linux/mailbox_client.h>
> > > > +#include <linux/mailbox/zynqmp-ipi-message.h>
> > > > +#include <linux/module.h>
> > > > +#include <linux/of_address.h>
> > > > +#include <linux/of_platform.h>
> > > > +#include <linux/of_reserved_mem.h>
> > > > +#include <linux/platform_device.h>
> > > > +#include <linux/remoteproc.h>
> > > > +#include <linux/skbuff.h>
> > > > +#include <linux/sysfs.h>
> > > > +
> > > > +#include "remoteproc_internal.h"
> > > > +
> > > > +#define MAX_RPROCS	2 /* Support up to 2 RPU */
> > > > +#define MAX_MEM_PNODES	4 /* Max power nodes for one RPU
> > memory
> > > instance */
> > > > +
> > > > +#define BANK_LIST_PROP	"sram"
> > > > +#define DDR_LIST_PROP	"memory-region"
> > > > +
> > > > +/* IPI buffer MAX length */
> > > > +#define IPI_BUF_LEN_MAX	32U
> > > > +/* RX mailbox client buffer max length */
> > > > +#define RX_MBOX_CLIENT_BUF_MAX	(IPI_BUF_LEN_MAX + \
> > > > +				 sizeof(struct zynqmp_ipi_message))
> > > > +
> > > > +/*
> > > > + * Map each Xilinx on-chip SRAM  Bank address to their own respective
> > > > + * pm_node_id.
> > > > + */
> > > > +struct sram_addr_data {
> > > > +	phys_addr_t addr;
> > > > +	enum pm_node_id id;
> > > > +};
> > > > +
> > > > +#define NUM_SRAMS 4U
> > > > +static const struct sram_addr_data zynqmp_banks[NUM_SRAMS] = {
> > > > +	{0xffe00000UL, NODE_TCM_0_A},
> > > > +	{0xffe20000UL, NODE_TCM_0_B},
> > > > +	{0xffe90000UL, NODE_TCM_1_A},
> > > > +	{0xffeb0000UL, NODE_TCM_1_B},
> > > > +};
> > > > +
> > > > +/**
> > > > + * struct zynqmp_r5_rproc - ZynqMP R5 core structure
> > > > + *
> > > > + * @rx_mc_buf: rx mailbox client buffer to save the rx message
> > > > + * @tx_mc: tx mailbox client
> > > > + * @rx_mc: rx mailbox client
> > > > + * @mbox_work: mbox_work for the RPU remoteproc
> > > > + * @tx_mc_skbs: socket buffers for tx mailbox client
> > > > + * @dev: device of RPU instance
> > > > + * @rproc: rproc handle
> > > > + * @tx_chan: tx mailbox channel
> > > > + * @rx_chan: rx mailbox channel
> > > > + * @pnode_id: RPU CPU power domain id
> > > > + * @elem: linked list item
> > > > + */
> > > > +struct zynqmp_r5_rproc {
> > > > +	unsigned char rx_mc_buf[RX_MBOX_CLIENT_BUF_MAX];
> > > > +	struct mbox_client tx_mc;
> > > > +	struct mbox_client rx_mc;
> > > > +	struct work_struct mbox_work;
> > > > +	struct sk_buff_head tx_mc_skbs;
> > > > +	struct device *dev;
> > > > +	struct rproc *rproc;
> > > > +	struct mbox_chan *tx_chan;
> > > > +	struct mbox_chan *rx_chan;
> > > > +	u32 pnode_id;
> > > > +	struct list_head elem;
> > > > +};
> > > > +
> > > > +/*
> > > > + * r5_set_mode - set RPU operation mode
> > > > + * @z_rproc: Remote processor private data
> > > > + * @rpu_mode: mode specified by device tree to configure the RPU to
> > > > + *
> > > > + * set RPU operation mode
> > > > + *
> > > > + * Return: 0 for success, negative value for failure
> > > > + */
> > > > +static int r5_set_mode(struct zynqmp_r5_rproc *z_rproc,
> > > > +		       enum rpu_oper_mode rpu_mode)
> > > > +{
> > > > +	enum rpu_tcm_comb tcm_mode;
> > > > +	enum rpu_oper_mode cur_rpu_mode;
> > > > +	int ret;
> > > > +
> > > > +	ret = zynqmp_pm_get_rpu_mode(z_rproc->pnode_id,
> > > &cur_rpu_mode);
> > > > +	if (ret < 0)
> > > > +		return ret;
> > > > +
> > > > +	if (rpu_mode != cur_rpu_mode) {
> > > > +		ret = zynqmp_pm_set_rpu_mode(z_rproc->pnode_id,
> > > rpu_mode);
> > > > +		if (ret < 0)
> > > > +			return ret;
> > > > +	}
> > > > +
> > > > +	tcm_mode = (rpu_mode == PM_RPU_MODE_LOCKSTEP) ?
> > > > +		    PM_RPU_TCM_COMB : PM_RPU_TCM_SPLIT;
> > > > +	return zynqmp_pm_set_tcm_config(z_rproc->pnode_id, tcm_mode);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_mem_release
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to unmap
> > > > + *
> > > > + * Unmap TCM banks when powering down R5 core.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int tcm_mem_release(struct rproc *rproc, struct
> rproc_mem_entry
> > > *mem)
> > > > +{
> > > > +	u32 pnode_id = (u64)mem->priv;
> > > > +
> > > > +	iounmap(mem->va);
> > > > +	return zynqmp_pm_release_node(pnode_id);
> > >
> > > Is the opposite of that call zynqmp_pm_request_node()?  If so doing the
> > > power up
> > > and down of the TCM banks at driver probe() and remove() is highly
> > > inefficient.
> > > Consider using rproc_ops::prepare() and rproc_ops::unprepare().
> > >
> If I understand your comment correctly this is about handling the
> request_node() and release_node() calls. While its true that these two
> functions can be moved to prepare and unprepared I think these are still
> occurring not at probe nor remove.
> 
> For request_node() of the TCM bank this is occurring in parse_fw() in
> remoteproc_core as in the zynqmp_r5 driver shows below:
> 
> 	.parse_fw	= zynqmp_r5_parse_fw,
> 	...
> 	Zynqmp_parse_fw calls parse_tcm_banks
> 
> 	static int parse_tcm_banks(struct rproc *rproc){
> 		...
> 		ret = zynqmp_r5_pm_request_sram(rsc.start);
> 		...
>  		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> 					   (size_t)size, rsc.start,
> 					   tcm_mem_alloc,
> 					   tcm_mem_release,
> 					   rsc.name);
> 
> For release_node() this is occurring not at remove but instead is managed by
> remoteproc_core and tied to the lifecyle of the remoteproc carveout since
> release_node() is within the function tcm_mem_release() and therefore
> rproc_resource_cleanup() will end up triggering
> TCM bank cleanup.
> 
> 
> Still it is understandable to prefer prepare/unprepared to contain the 2 lines.
> 
> 
> If you still prefer prepare/unprepared is the following implementation ok?
> 
> prepare:
> 	Just move request_node() line here
> unprepare:
> 	similarly this is only moving the 1 line of release_node().
> 
> If this suggestion satisfies what you are saying I will do so, otherwise please
> correct my understanding
> 
> 
> Cheers
> Ben
> 
> 
> 
> > > > +}
> > >
> > > Please move this just after tcm_mem_alloc()
> > >
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_start
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Start R5 Core from designated boot address.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	enum rpu_boot_mem bootmem;
> > > > +
> > > > +	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
> > > > +		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> > > > +
> > > > +	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
> > > > +		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> > > > +
> > > > +	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
> > > > +				      bootmem,
> > > ZYNQMP_PM_REQUEST_ACK_NO);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_stop
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Power down  R5 Core.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +
> > > > +	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
> > > > +				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_mem_alloc
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to map
> > > > + *
> > > > + * Callback to map va for memory-region's carveout.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
> > > > +				     struct rproc_mem_entry *mem)
> > > > +{
> > > > +	void *va;
> > > > +
> > > > +	va = ioremap_wc(mem->dma, mem->len);
> > > > +	if (IS_ERR_OR_NULL(va))
> > > > +		return -ENOMEM;
> > > > +
> > > > +	mem->va = va;
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_mem_release
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to unmap
> > > > + *
> > > > + * Unmap memory-region carveout
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
> > > > +				       struct rproc_mem_entry *mem)
> > > > +{
> > > > +	iounmap(mem->va);
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * parse_mem_regions
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Construct rproc mem carveouts from carveout provided in
> > > > + * memory-region property
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int parse_mem_regions(struct rproc *rproc)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	struct device_node *np = z_rproc->dev->of_node;
> > > > +	struct device *dev = &rproc->dev;
> > > > +	struct of_phandle_iterator it;
> > > > +	struct rproc_mem_entry *mem;
> > > > +	struct reserved_mem *rmem;
> > > > +	int index = 0;
> > > > +
> > > > +	/* Register associated reserved memory regions */
> > > > +	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
> > > > +	while (of_phandle_iterator_next(&it) == 0) {
> > > > +		rmem = of_reserved_mem_lookup(it.node);
> > > > +		if (!rmem) {
> > > > +			dev_err(dev, "unable to acquire %s\n",
> > > DDR_LIST_PROP);
> > > > +			return -EINVAL;
> > > > +		}
> > > > +
> > > > +		if (strstr(it.node->name, "vdev0buffer")) {
> > > > +			mem = rproc_of_resm_mem_entry_init(dev, index,
> > > > +							   rmem->size,
> > > > +							   rmem->base,
> > > > +							   "vdev0buffer");
> > > > +		} else {
> > > > +			/*
> > > > +			 * The ensuing carveout is either for vring, which has
> > > > +			 * device tree node name of length 15 characters, or
> > > > +			 * firmware mem, where name is not used by
> > > remoteproc
> > > > +			 *  core later on. So default to vring length of 15.
> > >
> > > s/"*  core"/"* core"
> > >
> > > > +			 *
> > > > +			 * Extra char for null-terminated string.
> > > > +			 */
> > > > +			char name[16];
> > > > +
> > > > +			if (strstr(it.node->name, "vdev0vring")) {
> > > > +				int vring_id;
> > > > +
> > > > +				/*
> > > > +				 * Expect "rpuXvdev0vringX" as documented
> > > > +				 * in xilinx remoteproc device tree binding
> > > > +				 */
> > > > +				if (strlen(it.node->name) < 15) {
> > > > +					dev_err(dev, "%pOF is less than 15
> > > chars",
> > > > +						it.node);
> > > > +					return -EINVAL;
> > > > +				}
> > > > +
> > > > +				/*
> > > > +				 * can be 1 of multiple vring IDs per IPC
> > > channel
> > > > +				 * e.g. 'vdev0vring0' and 'vdev0vring1'
> > > > +				 *
> > > > +				 * Here get vring ID from 15th char
> > > > +				 */
> > > > +				vring_id = it.node->name[14] - '0';
> > > > +				snprintf(name, sizeof(name), "vdev0vring%d",
> > > vring_id);
> > >
> > > Function strstr() returns a pointer to the beginning of the string being
> > > searched for.  As such you should be able to do something like:
> > >
> > >                         name = strstr(it.node->name, "vdev0vring");
> > >                         if (name) {
> > >                                 ...
> > >
> > > and simply use @name when calling rproc_mem_entry_init().
> > >
> > > > +			} else {
> > > > +				strncpy(name, it.node->name, 16);
> > > > +			}
> > > > +
> > > > +			mem = rproc_mem_entry_init(dev, NULL,
> > > > +						   (dma_addr_t)rmem->base,
> > > > +						   rmem->size, rmem->base,
> > > > +
> > > zynqmp_r5_rproc_mem_alloc,
> > > > +
> > > zynqmp_r5_rproc_mem_release,
> > > > +						   name);
> > > > +		}
> > > > +
> > > > +		if (!mem)
> > > > +			return -ENOMEM;
> > > > +
> > > > +		rproc_add_carveout(rproc, mem);
> > > > +		index++;
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_pm_request_tcm
> > > > + * @addr: base address of mem provided in R5 core's sram property.
> > > > + *
> > > > + * Given sram base address, determine its corresponding Xilinx
> > > > + * Platform Management ID and then request access to this node
> > > > + * so that it can be power up.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
> > > > +{
> > > > +	int i;
> > > > +
> > > > +	for (i = 0; i < NUM_SRAMS; i++) {
> > > > +		if (zynqmp_banks[i].addr != addr)
> > > > +			continue;
> > > > +
> > > > +		return zynqmp_pm_request_node(zynqmp_banks[i].id,
> > > > +					      ZYNQMP_PM_CAPABILITY_ACCESS,
> > > 0,
> > > > +
> > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > +	}
> > > > +
> > > > +	return -EINVAL;
> > > > +}
> > > > +
> > > > +/*
> > > > + * tcm_mem_alloc
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to initialize the va and da fields of
> > > > + *
> > > > + * Given TCM bank entry, this callback will set device address for R5
> > > > + * running on TCM and also setup virtual address for TCM bank
> > > > + * remoteproc carveout.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int tcm_mem_alloc(struct rproc *rproc,
> > > > +			 struct rproc_mem_entry *mem)
> > > > +{
> > > > +	void *va;
> > > > +	struct device *dev = rproc->dev.parent;
> > > > +
> > > > +	va = ioremap_wc(mem->dma, mem->len);
> > > > +	if (IS_ERR_OR_NULL(va))
> > > > +		return -ENOMEM;
> > > > +
> > > > +	/* Update memory entry va */
> > > > +	mem->va = va;
> > > > +
> > > > +	/*
> > > > +	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> > > > +	 * while on the Linux side they are at 0xffexxxxx.
> > > > +	 *
> > > > +	 * Zero out the high 12 bits of the address. This will give
> > > > +	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> > > > +	 */
> > > > +	mem->da &= 0x000fffff;
> > > > +
> > > > +	/*
> > > > +	 * TCM Banks 1A and 1B still have to be translated.
> > > > +	 *
> > > > +	 * Below handle these two banks' absolute addresses (0xffe90000 and
> > > > +	 * 0xffeb0000) and convert to the expected relative addresses
> > > > +	 * (0x0 and 0x20000).
> > > > +	 */
> > > > +	if (mem->da == 0x90000 || mem->da == 0xB0000)
> > > > +		mem->da -= 0x90000;
> > > > +
> > > > +	/* if translated TCM bank address is not valid report error */
> > > > +	if (mem->da != 0x0 && mem->da != 0x20000) {
> > > > +		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
> > > > +		return -EINVAL;
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * parse_tcm_banks()
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Given R5 node in remoteproc instance
> > > > + * allocate remoteproc carveout for TCM memory
> > > > + * needed for firmware to be loaded
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int parse_tcm_banks(struct rproc *rproc)
> > > > +{
> > > > +	int i, num_banks;
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	struct device *dev = &rproc->dev;
> > > > +	struct device_node *r5_node = z_rproc->dev->of_node;
> > > > +
> > > > +	/* go through TCM banks for r5 node */
> > > > +	num_banks = of_count_phandle_with_args(r5_node,
> > > BANK_LIST_PROP, NULL);
> > > > +	if (num_banks <= 0) {
> > > > +		dev_err(dev, "need to specify TCM banks\n");
> > > > +		return -EINVAL;
> > > > +	}
> > > > +	for (i = 0; i < num_banks; i++) {
> > > > +		struct resource rsc;
> > > > +		resource_size_t size;
> > > > +		struct device_node *dt_node;
> > > > +		struct rproc_mem_entry *mem;
> > > > +		int ret;
> > > > +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> > > > +
> > > > +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> > > > +		if (!dt_node)
> > > > +			return -EINVAL;
> > > > +
> > > > +		ret = of_address_to_resource(dt_node, 0, &rsc);
> > > > +		if (ret < 0) {
> > > > +			of_node_put(dt_node);
> > > > +			return ret;
> > > > +		}
> > > > +		of_node_put(dt_node);
> > > > +
> > > > +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> > > > +		if (ret < 0)
> > > > +			return ret;
> > > > +
> > > > +		/* add carveout */
> > > > +		size = resource_size(&rsc);
> > > > +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> > > > +					   (size_t)size, rsc.start,
> > > > +					   tcm_mem_alloc,
> > > > +					   tcm_mem_release,
> > > > +					   rsc.name);
> > > > +		if (!mem) {
> > > > +			ret = zynqmp_pm_release_node(pnode_id);
> > >
> > > Where is @pnode_id initialised?
> >
> > Good catch. Previously this was value set in an earlier iteration. I will fix this
> > as follows:
> > - release the R5 cluster by instead passing z_rproc->pnode_id
> > - release the TCM bank via call to release_sram
> >
> > This will also remove the variable pnode_id.
> >


typo release_sram should be release_node()

also instead another, cleaner way might be as follows:
change zynqmp_r5_request_sram so that it takes 1 more arg to return by reference the TCM bank's Xilinx platform management ID and then set the TCM bank's ID with that information.

Thanks
Ben

> > >
> > > > +			if (ret)
> > > > +				dev_warn(dev,
> > > > +					 "fail to release node: %x ret: %x\n",
> > > > +					 pnode_id, ret);
> > > > +			return -ENOMEM;
> > > > +		}
> > > > +
> > > > +		mem->priv = (void *)(u64)pnode_id;
> > >
> > >
> > > And here too - how does this work when you test things out on your side?
> > >
> > > Please note that I will need a reply to these questions before I review
> > another
> > > set.
> > >
> > Here I will update as follows:
> > - set mem->priv to value from zynqmp_banks lookup table that contains
> > power node IDs
> >
> >
> > > > +		rproc_add_carveout(rproc, mem);
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_parse_fw()
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @fw: ptr to firmware to be loaded onto r5 core
> > > > + *
> > > > + * When loading firmware, ensure the necessary carveouts are in
> > > remoteproc
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct
> firmware
> > > *fw)
> > > > +{
> > > > +	int ret;
> > > > +
> > > > +	ret = parse_tcm_banks(rproc);
> > > > +	if (ret)
> > > > +		return ret;
> > > > +
> > > > +	ret = parse_mem_regions(rproc);
> > > > +	if (ret)
> > > > +		return ret;
> > > > +
> > > > +	ret = rproc_elf_load_rsc_table(rproc, fw);
> > > > +	if (ret == -EINVAL) {
> > > > +		/*
> > > > +		 * resource table only required for IPC.
> > > > +		 * if not present, this is not necessarily an error;
> > > > +		 * for example, loading r5 hello world application
> > > > +		 * so simply inform user and keep going.
> > > > +		 */
> > > > +		dev_info(&rproc->dev, "no resource table found.\n");
> > > > +		ret = 0;
> > > > +	}
> > > > +	return ret;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> > > > + * @rproc: r5 core's corresponding rproc structure
> > > > + * @vqid: virtqueue ID
> > > > + */
> > > > +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	struct device *dev = rproc->dev.parent;
> > > > +	struct zynqmp_ipi_message *mb_msg = NULL;
> > > > +	unsigned int skb_len;
> > > > +	struct sk_buff *skb;
> > > > +	int ret;
> > > > +
> > > > +	/* If vqid is negative, do not pass the vqid to
> > > > +	 * mailbox as vqid is supposed to be 0 or positive.
> > > > +	 * It also gives a way to just kick instead but
> > > > +	 * not use the IPI buffer.
> > > > +	 *
> > > > +	 * For now use negative vqid to assume no message will be
> > > > +	 * passed with IPI buffer, but just raise interrupt.
> > > > +	 * This will be faster as it doesn't need to copy the
> > > > +	 * message to the ZynqMP/Versal IPI message buffer.
> > > > +	 */
> > > > +	if (vqid >= 0) {
> > > > +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> > > > +		skb = alloc_skb(skb_len, GFP_KERNEL);
> > > > +		if (!skb)
> > > > +			return;
> > > > +
> > > > +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb,
> > > skb_len);
> > > > +		mb_msg->len = sizeof(vqid);
> > > > +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> > > > +
> > > > +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> > > > +	}
> > > > +
> > > > +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> > > > +	if (ret < 0) {
> > > > +		dev_warn(dev, "Failed to kick remote.\n");
> > > > +		if (mb_msg) {
> > > > +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> > > > +			kfree_skb(skb);
> > > > +		}
> > > > +	}
> > > > +}
> > > > +
> > >
> > > I am done reviewing this set.
> > >
> > > Thanks,
> > > Mathieu
> > >
> > > > +static struct rproc_ops zynqmp_r5_rproc_ops = {
> > > > +	.start		= zynqmp_r5_rproc_start,
> > > > +	.stop		= zynqmp_r5_rproc_stop,
> > > > +	.load		= rproc_elf_load_segments,
> > > > +	.parse_fw	= zynqmp_r5_parse_fw,
> > > > +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > > > +	.sanity_check	= rproc_elf_sanity_check,
> > > > +	.get_boot_addr	= rproc_elf_get_boot_addr,
> > > > +	.kick		= zynqmp_r5_rproc_kick,
> > > > +};
> > > > +
> > > > +/**
> > > > + * event_notified_idr_cb() - event notified idr callback
> > > > + * @id: idr id
> > > > + * @ptr: pointer to idr private data
> > > > + * @data: data passed to idr_for_each callback
> > > > + *
> > > > + * Pass notification to remoteproc virtio
> > > > + *
> > > > + * Return: 0. having return is to satisfy the idr_for_each() function
> > > > + *          pointer input argument requirement.
> > > > + **/
> > > > +static int event_notified_idr_cb(int id, void *ptr, void *data)
> > > > +{
> > > > +	struct rproc *rproc = data;
> > > > +
> > > > +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> > > > +		dev_dbg(&rproc->dev, "no message was found in idr with id:
> > > %d\n", id);
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/**
> > > > + * handle_event_notified() - remoteproc notification work function
> > > > + * @work: pointer to the work structure
> > > > + *
> > > > + * It checks each registered remoteproc notify IDs.
> > > > + */
> > > > +static void handle_event_notified(struct work_struct *work)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +	struct rproc *rproc;
> > > > +
> > > > +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> > > > +	rproc = z_rproc->rproc;
> > > > +
> > > > +	/*
> > > > +	 * This is to simply raise interrupt as ACK for remote.
> > > > +	 *
> > > > +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> > > > +	 * mapping to remote processor.
> > > > +	 */
> > > > +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> > > > +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> > > > +
> > > > +	/*
> > > > +	 * We only use IPI for interrupt. The firmware side may or may
> > > > +	 * not write the notifyid when it trigger IPI.
> > > > +	 * And thus, we scan through all the registered notifyids.
> > > > +	 */
> > > > +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> > > > + * @cl: mailbox client
> > > > + * @msg: message pointer
> > > > + *
> > > > + * It will schedule the R5 notification work.
> > > > + */
> > > > +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +
> > > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> > > > +	/*
> > > > +	 * Notification can be received with message payload.
> > > > +	 * To handle this, check for message contents.
> > > > +	 * If there are contents, copy to ipi message buffer payload location.
> > > > +	 *
> > > > +	 * If message is empty, then that means that was simple raising
> > > > +	 * of interrupt. no payload to process.
> > > > +	 *
> > > > +	 * Note: enqueue work regardless of msg as kick may imply that
> > > remote
> > > > +	 * is waiting for ack.
> > > > +	 */
> > > > +	if (msg) {
> > > > +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> > > > +		size_t len;
> > > > +
> > > > +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> > > > +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> > > > +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> > > > +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> > > > +		buf_msg->len = len;
> > > > +		memcpy(buf_msg->data, ipi_msg->data, len);
> > > > +	}
> > > > +	schedule_work(&z_rproc->mbox_work);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> > > > + * @cl: mailbox client
> > > > + * @msg: pointer to the message which has been sent
> > > > + * @r: status of last TX - OK or error
> > > > + *
> > > > + * It will be called by the mailbox framework when the last TX has
> done.
> > > > + */
> > > > +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg,
> > int
> > > r)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +	struct sk_buff *skb;
> > > > +
> > > > +	if (!msg)
> > > > +		return;
> > > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> > > > +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> > > > +	kfree_skb(skb);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_setup_mbox() - Setup mailboxes
> > > > + *			    this is used for each individual R5 core
> > > > + *
> > > > + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> > > > + * @node: pointer of the device node
> > > > + *
> > > > + * Function to setup mailboxes to talk to RPU.
> > > > + *
> > > > + * Return: 0 for success, negative value for failure.
> > > > + */
> > > > +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> > > > +				struct device_node *node)
> > > > +{
> > > > +	struct mbox_client *mclient;
> > > > +
> > > > +	/* Setup TX mailbox channel client */
> > > > +	mclient			= &z_rproc->tx_mc;
> > > > +	mclient->dev		= z_rproc->dev;
> > > > +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> > > > +	mclient->tx_block	= false;
> > > > +	mclient->knows_txdone	= false;
> > > > +	mclient->rx_callback	= NULL;
> > > > +
> > > > +	/* Setup RX mailbox channel client */
> > > > +	mclient			= &z_rproc->rx_mc;
> > > > +	mclient->dev		= z_rproc->dev;
> > > > +	mclient->tx_done	= NULL;
> > > > +	mclient->tx_block	= false;
> > > > +	mclient->knows_txdone	= false;
> > > > +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> > > > +
> > > > +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> > > > +
> > > > +	/* Request TX and RX channels */
> > > > +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc-
> > > >tx_mc, "tx");
> > > > +	if (IS_ERR(z_rproc->tx_chan)) {
> > > > +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> > > > +		return PTR_ERR(z_rproc->tx_chan);
> > > > +	}
> > > > +
> > > > +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc-
> > > >rx_mc, "rx");
> > > > +	if (IS_ERR(z_rproc->rx_chan)) {
> > > > +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> > > > +		return PTR_ERR(z_rproc->rx_chan);
> > > > +	}
> > > > +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc
> *z_rproc)
> > > > +{
> > > > +	mbox_free_channel(z_rproc->tx_chan);
> > > > +	mbox_free_channel(z_rproc->rx_chan);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> > > > + *		       this is called for each individual R5 core to
> > > > + *		       set up mailbox, Xilinx platform manager unique
> ID,
> > > > + *		       add to rproc core
> > > > + *
> > > > + * @pdev: domain platform device for current R5 core
> > > > + * @node: pointer of the device node for current R5 core
> > > > + * @rpu_mode: mode to configure RPU, split or lockstep
> > > > + *
> > > > + * Return: 0 for success, negative value for failure.
> > > > + */
> > > > +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct
> platform_device
> > > *pdev,
> > > > +					       struct device_node *node,
> > > > +					       enum rpu_oper_mode
> > > rpu_mode)
> > > > +{
> > > > +	int ret;
> > > > +	struct device *dev = &pdev->dev;
> > > > +	struct rproc *rproc_ptr;
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +
> > > > +	/* Allocate remoteproc instance */
> > > > +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev),
> > > &zynqmp_r5_rproc_ops,
> > > > +				     NULL, sizeof(struct zynqmp_r5_rproc));
> > > > +	if (!rproc_ptr) {
> > > > +		ret = -ENOMEM;
> > > > +		goto error;
> > > > +	}
> > > > +
> > > > +	rproc_ptr->auto_boot = false;
> > > > +	z_rproc = rproc_ptr->priv;
> > > > +	z_rproc->rproc = rproc_ptr;
> > > > +	z_rproc->dev = dev;
> > > > +
> > > > +	/* Set up DMA mask */
> > > > +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> > > > +	if (ret)
> > > > +		goto error;
> > > > +
> > > > +	/* Get R5 power domain node */
> > > > +	ret = of_property_read_u32(node, "power-domain", &z_rproc-
> > > >pnode_id);
> > > > +	if (ret)
> > > > +		goto error;
> > > > +
> > > > +	ret = r5_set_mode(z_rproc, rpu_mode);
> > > > +	if (ret)
> > > > +		goto error;
> > > > +
> > > > +	if (of_property_read_bool(node, "mboxes")) {
> > > > +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> > > > +		if (ret)
> > > > +			goto error;
> > > > +	}
> > > > +
> > > > +	/* Add R5 remoteproc */
> > > > +	ret = devm_rproc_add(dev, rproc_ptr);
> > > > +	if (ret) {
> > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > +		goto error;
> > > > +	}
> > > > +
> > > > +	return z_rproc;
> > > > +error:
> > > > +	return ERR_PTR(ret);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_remoteproc_probe()
> > > > + *
> > > > + * @pdev: domain platform device for R5 cluster
> > > > + *
> > > > + * called when driver is probed, for each R5 core specified in DT,
> > > > + * setup as needed to do remoteproc-related operations
> > > > + *
> > > > + * Return: 0 for success, negative value for failure.
> > > > + */
> > > > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > > > +{
> > > > +	int ret, core_count;
> > > > +	struct device *dev = &pdev->dev;
> > > > +	struct device_node *nc;
> > > > +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> > > > +	struct list_head *cluster; /* list to track each core's rproc */
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +	struct platform_device *child_pdev;
> > > > +	struct list_head *pos;
> > > > +
> > > > +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode",
> > > &rpu_mode);
> > > > +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> > > > +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> > > > +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> > > > +			ret, rpu_mode);
> > > > +		return ret;
> > > > +	}
> > > > +
> > > > +	dev_dbg(dev, "RPU configuration: %s\n",
> > > > +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" :
> > > "split");
> > > > +
> > > > +	/*
> > > > +	 * if 2 RPUs provided but one is lockstep, then we have an
> > > > +	 * invalid configuration.
> > > > +	 */
> > > > +
> > > > +	core_count = of_get_available_child_count(dev->of_node);
> > > > +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> > > > +	    core_count > MAX_RPROCS)
> > > > +		return -EINVAL;
> > > > +
> > > > +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > > > +	if (!cluster)
> > > > +		return -ENOMEM;
> > > > +	INIT_LIST_HEAD(cluster);
> > > > +
> > > > +	ret = devm_of_platform_populate(dev);
> > > > +	if (ret) {
> > > > +		dev_err(dev, "devm_of_platform_populate failed, ret =
> > > %d\n",
> > > > +			ret);
> > > > +		return ret;
> > > > +	}
> > > > +
> > > > +	/* probe each individual r5 core's remoteproc-related info */
> > > > +	for_each_available_child_of_node(dev->of_node, nc) {
> > > > +		child_pdev = of_find_device_by_node(nc);
> > > > +		if (!child_pdev) {
> > > > +			dev_err(dev, "could not get R5 core platform
> > > device\n");
> > > > +			ret = -ENODEV;
> > > > +			goto out;
> > > > +		}
> > > > +
> > > > +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> > > > +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> > > > +			ret ? "Failed" : "Able",
> > > > +			nc);
> > > > +		if (IS_ERR(z_rproc)) {
> > > > +			ret = PTR_ERR(z_rproc);
> > > > +			goto out;
> > > > +		}
> > > > +		list_add_tail(&z_rproc->elem, cluster);
> > > > +	}
> > > > +	/* wire in so each core can be cleaned up at driver remove */
> > > > +	platform_set_drvdata(pdev, cluster);
> > > > +	return 0;
> > > > +out:
> > > > +	/*
> > > > +	 * undo core0 upon any failures on core1 in split-mode
> > > > +	 *
> > > > +	 * in zynqmp_r5_probe z_rproc is set to null
> > > > +	 * and ret to non-zero value if error
> > > > +	 */
> > > > +	list_for_each(pos, cluster) {
> > > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > +	}
> > > > +	return ret;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_remoteproc_remove()
> > > > + *
> > > > + * @pdev: domain platform device for R5 cluster
> > > > + *
> > > > + * When the driver is unloaded, clean up the mailboxes for each
> > > > + * remoteproc that was initially probed.
> > > > + */
> > > > +static int zynqmp_r5_remoteproc_remove(struct platform_device
> *pdev)
> > > > +{
> > > > +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> > > > +						 platform_get_drvdata(pdev);
> > > > +	struct zynqmp_r5_rproc *z_rproc = NULL;
> > > > +
> > > > +	list_for_each_safe(pos, temp, cluster) {
> > > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > +		list_del(pos);
> > > > +	}
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/* Match table for OF platform binding */
> > > > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > > > +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> > > > +	{ /* end of list */ },
> > > > +};
> > > > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > > > +
> > > > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > > > +	.probe = zynqmp_r5_remoteproc_probe,
> > > > +	.remove = zynqmp_r5_remoteproc_remove,
> > > > +	.driver = {
> > > > +		.name = "zynqmp_r5_remoteproc",
> > > > +		.of_match_table = zynqmp_r5_remoteproc_match,
> > > > +	},
> > > > +};
> > > > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > > > +
> > > > +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> > > > +MODULE_LICENSE("GPL v2");
> > > > --
> > > > 2.17.1
> > > >

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
  2021-02-17 19:11         ` Ben Levinsky
@ 2021-02-17 21:50           ` Mathieu Poirier
  -1 siblings, 0 replies; 28+ messages in thread
From: Mathieu Poirier @ 2021-02-17 21:50 UTC (permalink / raw)
  To: Ben Levinsky
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	Michal Simek

On Wed, Feb 17, 2021 at 07:11:37PM +0000, Ben Levinsky wrote:
> Hi Mathieu,
> 
> I worked on this further and had 1 more comment on the prepare/unprepare for TCM banks below
>

[...]
 
> > > > + */
> > > > +static int tcm_mem_release(struct rproc *rproc, struct rproc_mem_entry
> > > *mem)
> > > > +{
> > > > +	u32 pnode_id = (u64)mem->priv;
> > > > +
> > > > +	iounmap(mem->va);
> > > > +	return zynqmp_pm_release_node(pnode_id);
> > >
> > > Is the opposite of that call zynqmp_pm_request_node()?  If so doing the
> > > power up
> > > and down of the TCM banks at driver probe() and remove() is highly
> > > inefficient.
> > > Consider using rproc_ops::prepare() and rproc_ops::unprepare().
> > >
> If I understand your comment correctly this is about handling the request_node() and release_node() calls. While its true that these two functions can be moved to prepare and unprepared I think these are still occurring not at probe nor remove.
>

Your assesment is corret and I am in the wrong.  Unfortunately that's what
happens when reviews are spread out over several days. 
 
> For request_node() of the TCM bank this is occurring in parse_fw() in remoteproc_core as in the zynqmp_r5 driver shows below:
> 
> 	.parse_fw	= zynqmp_r5_parse_fw,
> 	...
> 	Zynqmp_parse_fw calls parse_tcm_banks
> 
> 	static int parse_tcm_banks(struct rproc *rproc){
> 		...
> 		ret = zynqmp_r5_pm_request_sram(rsc.start);
> 		...
>  		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> 					   (size_t)size, rsc.start,
> 					   tcm_mem_alloc,
> 					   tcm_mem_release,
> 					   rsc.name);
> 
> For release_node() this is occurring not at remove but instead is managed by remoteproc_core and tied to the lifecyle of the remoteproc carveout since release_node() is within the function tcm_mem_release() and therefore rproc_resource_cleanup() will end up triggering 
> TCM bank cleanup.
> 
> 
> Still it is understandable to prefer prepare/unprepared to contain the 2 lines.
> 
> 
> If you still prefer prepare/unprepared is the following implementation ok?
>

The prepare/unprepare functions have been added to specifically handle cases
such as this one where miscellanous actions need to be taken before accessing
the HW.  Using them guarantees the same behavior as the remoteproc core
evolves, something that is less certain for parse_fw and resource cleanup.
 
> prepare: 
> 	Just move request_node() line here	
> unprepare: 
> 	similarly this is only moving the 1 line of release_node().

That will be just fine.

> 
> If this suggestion satisfies what you are saying I will do so, otherwise please correct my understanding
> 
> 
> Cheers
> Ben 
> 
> 
> 
> > > > +}
> > >
> > > Please move this just after tcm_mem_alloc()
> > >
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_start
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Start R5 Core from designated boot address.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	enum rpu_boot_mem bootmem;
> > > > +
> > > > +	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
> > > > +		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> > > > +
> > > > +	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
> > > > +		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> > > > +
> > > > +	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
> > > > +				      bootmem,
> > > ZYNQMP_PM_REQUEST_ACK_NO);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_stop
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Power down  R5 Core.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +
> > > > +	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
> > > > +				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_mem_alloc
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to map
> > > > + *
> > > > + * Callback to map va for memory-region's carveout.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
> > > > +				     struct rproc_mem_entry *mem)
> > > > +{
> > > > +	void *va;
> > > > +
> > > > +	va = ioremap_wc(mem->dma, mem->len);
> > > > +	if (IS_ERR_OR_NULL(va))
> > > > +		return -ENOMEM;
> > > > +
> > > > +	mem->va = va;
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_mem_release
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to unmap
> > > > + *
> > > > + * Unmap memory-region carveout
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
> > > > +				       struct rproc_mem_entry *mem)
> > > > +{
> > > > +	iounmap(mem->va);
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * parse_mem_regions
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Construct rproc mem carveouts from carveout provided in
> > > > + * memory-region property
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int parse_mem_regions(struct rproc *rproc)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	struct device_node *np = z_rproc->dev->of_node;
> > > > +	struct device *dev = &rproc->dev;
> > > > +	struct of_phandle_iterator it;
> > > > +	struct rproc_mem_entry *mem;
> > > > +	struct reserved_mem *rmem;
> > > > +	int index = 0;
> > > > +
> > > > +	/* Register associated reserved memory regions */
> > > > +	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
> > > > +	while (of_phandle_iterator_next(&it) == 0) {
> > > > +		rmem = of_reserved_mem_lookup(it.node);
> > > > +		if (!rmem) {
> > > > +			dev_err(dev, "unable to acquire %s\n",
> > > DDR_LIST_PROP);
> > > > +			return -EINVAL;
> > > > +		}
> > > > +
> > > > +		if (strstr(it.node->name, "vdev0buffer")) {
> > > > +			mem = rproc_of_resm_mem_entry_init(dev, index,
> > > > +							   rmem->size,
> > > > +							   rmem->base,
> > > > +							   "vdev0buffer");
> > > > +		} else {
> > > > +			/*
> > > > +			 * The ensuing carveout is either for vring, which has
> > > > +			 * device tree node name of length 15 characters, or
> > > > +			 * firmware mem, where name is not used by
> > > remoteproc
> > > > +			 *  core later on. So default to vring length of 15.
> > >
> > > s/"*  core"/"* core"
> > >
> > > > +			 *
> > > > +			 * Extra char for null-terminated string.
> > > > +			 */
> > > > +			char name[16];
> > > > +
> > > > +			if (strstr(it.node->name, "vdev0vring")) {
> > > > +				int vring_id;
> > > > +
> > > > +				/*
> > > > +				 * Expect "rpuXvdev0vringX" as documented
> > > > +				 * in xilinx remoteproc device tree binding
> > > > +				 */
> > > > +				if (strlen(it.node->name) < 15) {
> > > > +					dev_err(dev, "%pOF is less than 15
> > > chars",
> > > > +						it.node);
> > > > +					return -EINVAL;
> > > > +				}
> > > > +
> > > > +				/*
> > > > +				 * can be 1 of multiple vring IDs per IPC
> > > channel
> > > > +				 * e.g. 'vdev0vring0' and 'vdev0vring1'
> > > > +				 *
> > > > +				 * Here get vring ID from 15th char
> > > > +				 */
> > > > +				vring_id = it.node->name[14] - '0';
> > > > +				snprintf(name, sizeof(name), "vdev0vring%d",
> > > vring_id);
> > >
> > > Function strstr() returns a pointer to the beginning of the string being
> > > searched for.  As such you should be able to do something like:
> > >
> > >                         name = strstr(it.node->name, "vdev0vring");
> > >                         if (name) {
> > >                                 ...
> > >
> > > and simply use @name when calling rproc_mem_entry_init().
> > >
> > > > +			} else {
> > > > +				strncpy(name, it.node->name, 16);
> > > > +			}
> > > > +
> > > > +			mem = rproc_mem_entry_init(dev, NULL,
> > > > +						   (dma_addr_t)rmem->base,
> > > > +						   rmem->size, rmem->base,
> > > > +
> > > zynqmp_r5_rproc_mem_alloc,
> > > > +
> > > zynqmp_r5_rproc_mem_release,
> > > > +						   name);
> > > > +		}
> > > > +
> > > > +		if (!mem)
> > > > +			return -ENOMEM;
> > > > +
> > > > +		rproc_add_carveout(rproc, mem);
> > > > +		index++;
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_pm_request_tcm
> > > > + * @addr: base address of mem provided in R5 core's sram property.
> > > > + *
> > > > + * Given sram base address, determine its corresponding Xilinx
> > > > + * Platform Management ID and then request access to this node
> > > > + * so that it can be power up.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
> > > > +{
> > > > +	int i;
> > > > +
> > > > +	for (i = 0; i < NUM_SRAMS; i++) {
> > > > +		if (zynqmp_banks[i].addr != addr)
> > > > +			continue;
> > > > +
> > > > +		return zynqmp_pm_request_node(zynqmp_banks[i].id,
> > > > +					      ZYNQMP_PM_CAPABILITY_ACCESS,
> > > 0,
> > > > +
> > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > +	}
> > > > +
> > > > +	return -EINVAL;
> > > > +}
> > > > +
> > > > +/*
> > > > + * tcm_mem_alloc
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to initialize the va and da fields of
> > > > + *
> > > > + * Given TCM bank entry, this callback will set device address for R5
> > > > + * running on TCM and also setup virtual address for TCM bank
> > > > + * remoteproc carveout.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int tcm_mem_alloc(struct rproc *rproc,
> > > > +			 struct rproc_mem_entry *mem)
> > > > +{
> > > > +	void *va;
> > > > +	struct device *dev = rproc->dev.parent;
> > > > +
> > > > +	va = ioremap_wc(mem->dma, mem->len);
> > > > +	if (IS_ERR_OR_NULL(va))
> > > > +		return -ENOMEM;
> > > > +
> > > > +	/* Update memory entry va */
> > > > +	mem->va = va;
> > > > +
> > > > +	/*
> > > > +	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> > > > +	 * while on the Linux side they are at 0xffexxxxx.
> > > > +	 *
> > > > +	 * Zero out the high 12 bits of the address. This will give
> > > > +	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> > > > +	 */
> > > > +	mem->da &= 0x000fffff;
> > > > +
> > > > +	/*
> > > > +	 * TCM Banks 1A and 1B still have to be translated.
> > > > +	 *
> > > > +	 * Below handle these two banks' absolute addresses (0xffe90000 and
> > > > +	 * 0xffeb0000) and convert to the expected relative addresses
> > > > +	 * (0x0 and 0x20000).
> > > > +	 */
> > > > +	if (mem->da == 0x90000 || mem->da == 0xB0000)
> > > > +		mem->da -= 0x90000;
> > > > +
> > > > +	/* if translated TCM bank address is not valid report error */
> > > > +	if (mem->da != 0x0 && mem->da != 0x20000) {
> > > > +		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
> > > > +		return -EINVAL;
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * parse_tcm_banks()
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Given R5 node in remoteproc instance
> > > > + * allocate remoteproc carveout for TCM memory
> > > > + * needed for firmware to be loaded
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int parse_tcm_banks(struct rproc *rproc)
> > > > +{
> > > > +	int i, num_banks;
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	struct device *dev = &rproc->dev;
> > > > +	struct device_node *r5_node = z_rproc->dev->of_node;
> > > > +
> > > > +	/* go through TCM banks for r5 node */
> > > > +	num_banks = of_count_phandle_with_args(r5_node,
> > > BANK_LIST_PROP, NULL);
> > > > +	if (num_banks <= 0) {
> > > > +		dev_err(dev, "need to specify TCM banks\n");
> > > > +		return -EINVAL;
> > > > +	}
> > > > +	for (i = 0; i < num_banks; i++) {
> > > > +		struct resource rsc;
> > > > +		resource_size_t size;
> > > > +		struct device_node *dt_node;
> > > > +		struct rproc_mem_entry *mem;
> > > > +		int ret;
> > > > +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> > > > +
> > > > +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> > > > +		if (!dt_node)
> > > > +			return -EINVAL;
> > > > +
> > > > +		ret = of_address_to_resource(dt_node, 0, &rsc);
> > > > +		if (ret < 0) {
> > > > +			of_node_put(dt_node);
> > > > +			return ret;
> > > > +		}
> > > > +		of_node_put(dt_node);
> > > > +
> > > > +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> > > > +		if (ret < 0)
> > > > +			return ret;
> > > > +
> > > > +		/* add carveout */
> > > > +		size = resource_size(&rsc);
> > > > +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> > > > +					   (size_t)size, rsc.start,
> > > > +					   tcm_mem_alloc,
> > > > +					   tcm_mem_release,
> > > > +					   rsc.name);
> > > > +		if (!mem) {
> > > > +			ret = zynqmp_pm_release_node(pnode_id);
> > >
> > > Where is @pnode_id initialised?
> > 
> > Good catch. Previously this was value set in an earlier iteration. I will fix this
> > as follows:
> > - release the R5 cluster by instead passing z_rproc->pnode_id
> > - release the TCM bank via call to release_sram
> > 
> > This will also remove the variable pnode_id.
> > 
> > >
> > > > +			if (ret)
> > > > +				dev_warn(dev,
> > > > +					 "fail to release node: %x ret: %x\n",
> > > > +					 pnode_id, ret);
> > > > +			return -ENOMEM;
> > > > +		}
> > > > +
> > > > +		mem->priv = (void *)(u64)pnode_id;
> > >
> > >
> > > And here too - how does this work when you test things out on your side?
> > >
> > > Please note that I will need a reply to these questions before I review
> > another
> > > set.
> > >
> > Here I will update as follows:
> > - set mem->priv to value from zynqmp_banks lookup table that contains
> > power node IDs
> > 
> > 
> > > > +		rproc_add_carveout(rproc, mem);
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_parse_fw()
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @fw: ptr to firmware to be loaded onto r5 core
> > > > + *
> > > > + * When loading firmware, ensure the necessary carveouts are in
> > > remoteproc
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware
> > > *fw)
> > > > +{
> > > > +	int ret;
> > > > +
> > > > +	ret = parse_tcm_banks(rproc);
> > > > +	if (ret)
> > > > +		return ret;
> > > > +
> > > > +	ret = parse_mem_regions(rproc);
> > > > +	if (ret)
> > > > +		return ret;
> > > > +
> > > > +	ret = rproc_elf_load_rsc_table(rproc, fw);
> > > > +	if (ret == -EINVAL) {
> > > > +		/*
> > > > +		 * resource table only required for IPC.
> > > > +		 * if not present, this is not necessarily an error;
> > > > +		 * for example, loading r5 hello world application
> > > > +		 * so simply inform user and keep going.
> > > > +		 */
> > > > +		dev_info(&rproc->dev, "no resource table found.\n");
> > > > +		ret = 0;
> > > > +	}
> > > > +	return ret;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> > > > + * @rproc: r5 core's corresponding rproc structure
> > > > + * @vqid: virtqueue ID
> > > > + */
> > > > +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	struct device *dev = rproc->dev.parent;
> > > > +	struct zynqmp_ipi_message *mb_msg = NULL;
> > > > +	unsigned int skb_len;
> > > > +	struct sk_buff *skb;
> > > > +	int ret;
> > > > +
> > > > +	/* If vqid is negative, do not pass the vqid to
> > > > +	 * mailbox as vqid is supposed to be 0 or positive.
> > > > +	 * It also gives a way to just kick instead but
> > > > +	 * not use the IPI buffer.
> > > > +	 *
> > > > +	 * For now use negative vqid to assume no message will be
> > > > +	 * passed with IPI buffer, but just raise interrupt.
> > > > +	 * This will be faster as it doesn't need to copy the
> > > > +	 * message to the ZynqMP/Versal IPI message buffer.
> > > > +	 */
> > > > +	if (vqid >= 0) {
> > > > +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> > > > +		skb = alloc_skb(skb_len, GFP_KERNEL);
> > > > +		if (!skb)
> > > > +			return;
> > > > +
> > > > +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb,
> > > skb_len);
> > > > +		mb_msg->len = sizeof(vqid);
> > > > +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> > > > +
> > > > +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> > > > +	}
> > > > +
> > > > +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> > > > +	if (ret < 0) {
> > > > +		dev_warn(dev, "Failed to kick remote.\n");
> > > > +		if (mb_msg) {
> > > > +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> > > > +			kfree_skb(skb);
> > > > +		}
> > > > +	}
> > > > +}
> > > > +
> > >
> > > I am done reviewing this set.
> > >
> > > Thanks,
> > > Mathieu
> > >
> > > > +static struct rproc_ops zynqmp_r5_rproc_ops = {
> > > > +	.start		= zynqmp_r5_rproc_start,
> > > > +	.stop		= zynqmp_r5_rproc_stop,
> > > > +	.load		= rproc_elf_load_segments,
> > > > +	.parse_fw	= zynqmp_r5_parse_fw,
> > > > +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > > > +	.sanity_check	= rproc_elf_sanity_check,
> > > > +	.get_boot_addr	= rproc_elf_get_boot_addr,
> > > > +	.kick		= zynqmp_r5_rproc_kick,
> > > > +};
> > > > +
> > > > +/**
> > > > + * event_notified_idr_cb() - event notified idr callback
> > > > + * @id: idr id
> > > > + * @ptr: pointer to idr private data
> > > > + * @data: data passed to idr_for_each callback
> > > > + *
> > > > + * Pass notification to remoteproc virtio
> > > > + *
> > > > + * Return: 0. having return is to satisfy the idr_for_each() function
> > > > + *          pointer input argument requirement.
> > > > + **/
> > > > +static int event_notified_idr_cb(int id, void *ptr, void *data)
> > > > +{
> > > > +	struct rproc *rproc = data;
> > > > +
> > > > +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> > > > +		dev_dbg(&rproc->dev, "no message was found in idr with id:
> > > %d\n", id);
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/**
> > > > + * handle_event_notified() - remoteproc notification work function
> > > > + * @work: pointer to the work structure
> > > > + *
> > > > + * It checks each registered remoteproc notify IDs.
> > > > + */
> > > > +static void handle_event_notified(struct work_struct *work)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +	struct rproc *rproc;
> > > > +
> > > > +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> > > > +	rproc = z_rproc->rproc;
> > > > +
> > > > +	/*
> > > > +	 * This is to simply raise interrupt as ACK for remote.
> > > > +	 *
> > > > +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> > > > +	 * mapping to remote processor.
> > > > +	 */
> > > > +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> > > > +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> > > > +
> > > > +	/*
> > > > +	 * We only use IPI for interrupt. The firmware side may or may
> > > > +	 * not write the notifyid when it trigger IPI.
> > > > +	 * And thus, we scan through all the registered notifyids.
> > > > +	 */
> > > > +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> > > > + * @cl: mailbox client
> > > > + * @msg: message pointer
> > > > + *
> > > > + * It will schedule the R5 notification work.
> > > > + */
> > > > +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +
> > > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> > > > +	/*
> > > > +	 * Notification can be received with message payload.
> > > > +	 * To handle this, check for message contents.
> > > > +	 * If there are contents, copy to ipi message buffer payload location.
> > > > +	 *
> > > > +	 * If message is empty, then that means that was simple raising
> > > > +	 * of interrupt. no payload to process.
> > > > +	 *
> > > > +	 * Note: enqueue work regardless of msg as kick may imply that
> > > remote
> > > > +	 * is waiting for ack.
> > > > +	 */
> > > > +	if (msg) {
> > > > +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> > > > +		size_t len;
> > > > +
> > > > +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> > > > +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> > > > +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> > > > +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> > > > +		buf_msg->len = len;
> > > > +		memcpy(buf_msg->data, ipi_msg->data, len);
> > > > +	}
> > > > +	schedule_work(&z_rproc->mbox_work);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> > > > + * @cl: mailbox client
> > > > + * @msg: pointer to the message which has been sent
> > > > + * @r: status of last TX - OK or error
> > > > + *
> > > > + * It will be called by the mailbox framework when the last TX has done.
> > > > + */
> > > > +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg,
> > int
> > > r)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +	struct sk_buff *skb;
> > > > +
> > > > +	if (!msg)
> > > > +		return;
> > > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> > > > +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> > > > +	kfree_skb(skb);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_setup_mbox() - Setup mailboxes
> > > > + *			    this is used for each individual R5 core
> > > > + *
> > > > + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> > > > + * @node: pointer of the device node
> > > > + *
> > > > + * Function to setup mailboxes to talk to RPU.
> > > > + *
> > > > + * Return: 0 for success, negative value for failure.
> > > > + */
> > > > +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> > > > +				struct device_node *node)
> > > > +{
> > > > +	struct mbox_client *mclient;
> > > > +
> > > > +	/* Setup TX mailbox channel client */
> > > > +	mclient			= &z_rproc->tx_mc;
> > > > +	mclient->dev		= z_rproc->dev;
> > > > +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> > > > +	mclient->tx_block	= false;
> > > > +	mclient->knows_txdone	= false;
> > > > +	mclient->rx_callback	= NULL;
> > > > +
> > > > +	/* Setup RX mailbox channel client */
> > > > +	mclient			= &z_rproc->rx_mc;
> > > > +	mclient->dev		= z_rproc->dev;
> > > > +	mclient->tx_done	= NULL;
> > > > +	mclient->tx_block	= false;
> > > > +	mclient->knows_txdone	= false;
> > > > +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> > > > +
> > > > +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> > > > +
> > > > +	/* Request TX and RX channels */
> > > > +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc-
> > > >tx_mc, "tx");
> > > > +	if (IS_ERR(z_rproc->tx_chan)) {
> > > > +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> > > > +		return PTR_ERR(z_rproc->tx_chan);
> > > > +	}
> > > > +
> > > > +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc-
> > > >rx_mc, "rx");
> > > > +	if (IS_ERR(z_rproc->rx_chan)) {
> > > > +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> > > > +		return PTR_ERR(z_rproc->rx_chan);
> > > > +	}
> > > > +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc *z_rproc)
> > > > +{
> > > > +	mbox_free_channel(z_rproc->tx_chan);
> > > > +	mbox_free_channel(z_rproc->rx_chan);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> > > > + *		       this is called for each individual R5 core to
> > > > + *		       set up mailbox, Xilinx platform manager unique ID,
> > > > + *		       add to rproc core
> > > > + *
> > > > + * @pdev: domain platform device for current R5 core
> > > > + * @node: pointer of the device node for current R5 core
> > > > + * @rpu_mode: mode to configure RPU, split or lockstep
> > > > + *
> > > > + * Return: 0 for success, negative value for failure.
> > > > + */
> > > > +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device
> > > *pdev,
> > > > +					       struct device_node *node,
> > > > +					       enum rpu_oper_mode
> > > rpu_mode)
> > > > +{
> > > > +	int ret;
> > > > +	struct device *dev = &pdev->dev;
> > > > +	struct rproc *rproc_ptr;
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +
> > > > +	/* Allocate remoteproc instance */
> > > > +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev),
> > > &zynqmp_r5_rproc_ops,
> > > > +				     NULL, sizeof(struct zynqmp_r5_rproc));
> > > > +	if (!rproc_ptr) {
> > > > +		ret = -ENOMEM;
> > > > +		goto error;
> > > > +	}
> > > > +
> > > > +	rproc_ptr->auto_boot = false;
> > > > +	z_rproc = rproc_ptr->priv;
> > > > +	z_rproc->rproc = rproc_ptr;
> > > > +	z_rproc->dev = dev;
> > > > +
> > > > +	/* Set up DMA mask */
> > > > +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> > > > +	if (ret)
> > > > +		goto error;
> > > > +
> > > > +	/* Get R5 power domain node */
> > > > +	ret = of_property_read_u32(node, "power-domain", &z_rproc-
> > > >pnode_id);
> > > > +	if (ret)
> > > > +		goto error;
> > > > +
> > > > +	ret = r5_set_mode(z_rproc, rpu_mode);
> > > > +	if (ret)
> > > > +		goto error;
> > > > +
> > > > +	if (of_property_read_bool(node, "mboxes")) {
> > > > +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> > > > +		if (ret)
> > > > +			goto error;
> > > > +	}
> > > > +
> > > > +	/* Add R5 remoteproc */
> > > > +	ret = devm_rproc_add(dev, rproc_ptr);
> > > > +	if (ret) {
> > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > +		goto error;
> > > > +	}
> > > > +
> > > > +	return z_rproc;
> > > > +error:
> > > > +	return ERR_PTR(ret);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_remoteproc_probe()
> > > > + *
> > > > + * @pdev: domain platform device for R5 cluster
> > > > + *
> > > > + * called when driver is probed, for each R5 core specified in DT,
> > > > + * setup as needed to do remoteproc-related operations
> > > > + *
> > > > + * Return: 0 for success, negative value for failure.
> > > > + */
> > > > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > > > +{
> > > > +	int ret, core_count;
> > > > +	struct device *dev = &pdev->dev;
> > > > +	struct device_node *nc;
> > > > +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> > > > +	struct list_head *cluster; /* list to track each core's rproc */
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +	struct platform_device *child_pdev;
> > > > +	struct list_head *pos;
> > > > +
> > > > +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode",
> > > &rpu_mode);
> > > > +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> > > > +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> > > > +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> > > > +			ret, rpu_mode);
> > > > +		return ret;
> > > > +	}
> > > > +
> > > > +	dev_dbg(dev, "RPU configuration: %s\n",
> > > > +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" :
> > > "split");
> > > > +
> > > > +	/*
> > > > +	 * if 2 RPUs provided but one is lockstep, then we have an
> > > > +	 * invalid configuration.
> > > > +	 */
> > > > +
> > > > +	core_count = of_get_available_child_count(dev->of_node);
> > > > +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> > > > +	    core_count > MAX_RPROCS)
> > > > +		return -EINVAL;
> > > > +
> > > > +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > > > +	if (!cluster)
> > > > +		return -ENOMEM;
> > > > +	INIT_LIST_HEAD(cluster);
> > > > +
> > > > +	ret = devm_of_platform_populate(dev);
> > > > +	if (ret) {
> > > > +		dev_err(dev, "devm_of_platform_populate failed, ret =
> > > %d\n",
> > > > +			ret);
> > > > +		return ret;
> > > > +	}
> > > > +
> > > > +	/* probe each individual r5 core's remoteproc-related info */
> > > > +	for_each_available_child_of_node(dev->of_node, nc) {
> > > > +		child_pdev = of_find_device_by_node(nc);
> > > > +		if (!child_pdev) {
> > > > +			dev_err(dev, "could not get R5 core platform
> > > device\n");
> > > > +			ret = -ENODEV;
> > > > +			goto out;
> > > > +		}
> > > > +
> > > > +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> > > > +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> > > > +			ret ? "Failed" : "Able",
> > > > +			nc);
> > > > +		if (IS_ERR(z_rproc)) {
> > > > +			ret = PTR_ERR(z_rproc);
> > > > +			goto out;
> > > > +		}
> > > > +		list_add_tail(&z_rproc->elem, cluster);
> > > > +	}
> > > > +	/* wire in so each core can be cleaned up at driver remove */
> > > > +	platform_set_drvdata(pdev, cluster);
> > > > +	return 0;
> > > > +out:
> > > > +	/*
> > > > +	 * undo core0 upon any failures on core1 in split-mode
> > > > +	 *
> > > > +	 * in zynqmp_r5_probe z_rproc is set to null
> > > > +	 * and ret to non-zero value if error
> > > > +	 */
> > > > +	list_for_each(pos, cluster) {
> > > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > +	}
> > > > +	return ret;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_remoteproc_remove()
> > > > + *
> > > > + * @pdev: domain platform device for R5 cluster
> > > > + *
> > > > + * When the driver is unloaded, clean up the mailboxes for each
> > > > + * remoteproc that was initially probed.
> > > > + */
> > > > +static int zynqmp_r5_remoteproc_remove(struct platform_device *pdev)
> > > > +{
> > > > +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> > > > +						 platform_get_drvdata(pdev);
> > > > +	struct zynqmp_r5_rproc *z_rproc = NULL;
> > > > +
> > > > +	list_for_each_safe(pos, temp, cluster) {
> > > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > +		list_del(pos);
> > > > +	}
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/* Match table for OF platform binding */
> > > > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > > > +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> > > > +	{ /* end of list */ },
> > > > +};
> > > > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > > > +
> > > > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > > > +	.probe = zynqmp_r5_remoteproc_probe,
> > > > +	.remove = zynqmp_r5_remoteproc_remove,
> > > > +	.driver = {
> > > > +		.name = "zynqmp_r5_remoteproc",
> > > > +		.of_match_table = zynqmp_r5_remoteproc_match,
> > > > +	},
> > > > +};
> > > > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > > > +
> > > > +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> > > > +MODULE_LICENSE("GPL v2");
> > > > --
> > > > 2.17.1
> > > >

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
@ 2021-02-17 21:50           ` Mathieu Poirier
  0 siblings, 0 replies; 28+ messages in thread
From: Mathieu Poirier @ 2021-02-17 21:50 UTC (permalink / raw)
  To: Ben Levinsky
  Cc: devicetree, Michal Simek, linux-remoteproc, linux-kernel,
	linux-arm-kernel

On Wed, Feb 17, 2021 at 07:11:37PM +0000, Ben Levinsky wrote:
> Hi Mathieu,
> 
> I worked on this further and had 1 more comment on the prepare/unprepare for TCM banks below
>

[...]
 
> > > > + */
> > > > +static int tcm_mem_release(struct rproc *rproc, struct rproc_mem_entry
> > > *mem)
> > > > +{
> > > > +	u32 pnode_id = (u64)mem->priv;
> > > > +
> > > > +	iounmap(mem->va);
> > > > +	return zynqmp_pm_release_node(pnode_id);
> > >
> > > Is the opposite of that call zynqmp_pm_request_node()?  If so doing the
> > > power up
> > > and down of the TCM banks at driver probe() and remove() is highly
> > > inefficient.
> > > Consider using rproc_ops::prepare() and rproc_ops::unprepare().
> > >
> If I understand your comment correctly this is about handling the request_node() and release_node() calls. While its true that these two functions can be moved to prepare and unprepared I think these are still occurring not at probe nor remove.
>

Your assesment is corret and I am in the wrong.  Unfortunately that's what
happens when reviews are spread out over several days. 
 
> For request_node() of the TCM bank this is occurring in parse_fw() in remoteproc_core as in the zynqmp_r5 driver shows below:
> 
> 	.parse_fw	= zynqmp_r5_parse_fw,
> 	...
> 	Zynqmp_parse_fw calls parse_tcm_banks
> 
> 	static int parse_tcm_banks(struct rproc *rproc){
> 		...
> 		ret = zynqmp_r5_pm_request_sram(rsc.start);
> 		...
>  		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> 					   (size_t)size, rsc.start,
> 					   tcm_mem_alloc,
> 					   tcm_mem_release,
> 					   rsc.name);
> 
> For release_node() this is occurring not at remove but instead is managed by remoteproc_core and tied to the lifecyle of the remoteproc carveout since release_node() is within the function tcm_mem_release() and therefore rproc_resource_cleanup() will end up triggering 
> TCM bank cleanup.
> 
> 
> Still it is understandable to prefer prepare/unprepared to contain the 2 lines.
> 
> 
> If you still prefer prepare/unprepared is the following implementation ok?
>

The prepare/unprepare functions have been added to specifically handle cases
such as this one where miscellanous actions need to be taken before accessing
the HW.  Using them guarantees the same behavior as the remoteproc core
evolves, something that is less certain for parse_fw and resource cleanup.
 
> prepare: 
> 	Just move request_node() line here	
> unprepare: 
> 	similarly this is only moving the 1 line of release_node().

That will be just fine.

> 
> If this suggestion satisfies what you are saying I will do so, otherwise please correct my understanding
> 
> 
> Cheers
> Ben 
> 
> 
> 
> > > > +}
> > >
> > > Please move this just after tcm_mem_alloc()
> > >
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_start
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Start R5 Core from designated boot address.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_start(struct rproc *rproc)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	enum rpu_boot_mem bootmem;
> > > > +
> > > > +	bootmem = (rproc->bootaddr & 0xF0000000) == 0xF0000000 ?
> > > > +		   PM_RPU_BOOTMEM_HIVEC : PM_RPU_BOOTMEM_LOVEC;
> > > > +
> > > > +	dev_dbg(rproc->dev.parent, "RPU boot from %s.",
> > > > +		bootmem == PM_RPU_BOOTMEM_HIVEC ? "OCM" : "TCM");
> > > > +
> > > > +	return zynqmp_pm_request_wake(z_rproc->pnode_id, 1,
> > > > +				      bootmem,
> > > ZYNQMP_PM_REQUEST_ACK_NO);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_stop
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Power down  R5 Core.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_stop(struct rproc *rproc)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +
> > > > +	return zynqmp_pm_force_pwrdwn(z_rproc->pnode_id,
> > > > +				      ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_mem_alloc
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to map
> > > > + *
> > > > + * Callback to map va for memory-region's carveout.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_mem_alloc(struct rproc *rproc,
> > > > +				     struct rproc_mem_entry *mem)
> > > > +{
> > > > +	void *va;
> > > > +
> > > > +	va = ioremap_wc(mem->dma, mem->len);
> > > > +	if (IS_ERR_OR_NULL(va))
> > > > +		return -ENOMEM;
> > > > +
> > > > +	mem->va = va;
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_mem_release
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to unmap
> > > > + *
> > > > + * Unmap memory-region carveout
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_rproc_mem_release(struct rproc *rproc,
> > > > +				       struct rproc_mem_entry *mem)
> > > > +{
> > > > +	iounmap(mem->va);
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * parse_mem_regions
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Construct rproc mem carveouts from carveout provided in
> > > > + * memory-region property
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int parse_mem_regions(struct rproc *rproc)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	struct device_node *np = z_rproc->dev->of_node;
> > > > +	struct device *dev = &rproc->dev;
> > > > +	struct of_phandle_iterator it;
> > > > +	struct rproc_mem_entry *mem;
> > > > +	struct reserved_mem *rmem;
> > > > +	int index = 0;
> > > > +
> > > > +	/* Register associated reserved memory regions */
> > > > +	of_phandle_iterator_init(&it, np, DDR_LIST_PROP, NULL, 0);
> > > > +	while (of_phandle_iterator_next(&it) == 0) {
> > > > +		rmem = of_reserved_mem_lookup(it.node);
> > > > +		if (!rmem) {
> > > > +			dev_err(dev, "unable to acquire %s\n",
> > > DDR_LIST_PROP);
> > > > +			return -EINVAL;
> > > > +		}
> > > > +
> > > > +		if (strstr(it.node->name, "vdev0buffer")) {
> > > > +			mem = rproc_of_resm_mem_entry_init(dev, index,
> > > > +							   rmem->size,
> > > > +							   rmem->base,
> > > > +							   "vdev0buffer");
> > > > +		} else {
> > > > +			/*
> > > > +			 * The ensuing carveout is either for vring, which has
> > > > +			 * device tree node name of length 15 characters, or
> > > > +			 * firmware mem, where name is not used by
> > > remoteproc
> > > > +			 *  core later on. So default to vring length of 15.
> > >
> > > s/"*  core"/"* core"
> > >
> > > > +			 *
> > > > +			 * Extra char for null-terminated string.
> > > > +			 */
> > > > +			char name[16];
> > > > +
> > > > +			if (strstr(it.node->name, "vdev0vring")) {
> > > > +				int vring_id;
> > > > +
> > > > +				/*
> > > > +				 * Expect "rpuXvdev0vringX" as documented
> > > > +				 * in xilinx remoteproc device tree binding
> > > > +				 */
> > > > +				if (strlen(it.node->name) < 15) {
> > > > +					dev_err(dev, "%pOF is less than 15
> > > chars",
> > > > +						it.node);
> > > > +					return -EINVAL;
> > > > +				}
> > > > +
> > > > +				/*
> > > > +				 * can be 1 of multiple vring IDs per IPC
> > > channel
> > > > +				 * e.g. 'vdev0vring0' and 'vdev0vring1'
> > > > +				 *
> > > > +				 * Here get vring ID from 15th char
> > > > +				 */
> > > > +				vring_id = it.node->name[14] - '0';
> > > > +				snprintf(name, sizeof(name), "vdev0vring%d",
> > > vring_id);
> > >
> > > Function strstr() returns a pointer to the beginning of the string being
> > > searched for.  As such you should be able to do something like:
> > >
> > >                         name = strstr(it.node->name, "vdev0vring");
> > >                         if (name) {
> > >                                 ...
> > >
> > > and simply use @name when calling rproc_mem_entry_init().
> > >
> > > > +			} else {
> > > > +				strncpy(name, it.node->name, 16);
> > > > +			}
> > > > +
> > > > +			mem = rproc_mem_entry_init(dev, NULL,
> > > > +						   (dma_addr_t)rmem->base,
> > > > +						   rmem->size, rmem->base,
> > > > +
> > > zynqmp_r5_rproc_mem_alloc,
> > > > +
> > > zynqmp_r5_rproc_mem_release,
> > > > +						   name);
> > > > +		}
> > > > +
> > > > +		if (!mem)
> > > > +			return -ENOMEM;
> > > > +
> > > > +		rproc_add_carveout(rproc, mem);
> > > > +		index++;
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_pm_request_tcm
> > > > + * @addr: base address of mem provided in R5 core's sram property.
> > > > + *
> > > > + * Given sram base address, determine its corresponding Xilinx
> > > > + * Platform Management ID and then request access to this node
> > > > + * so that it can be power up.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_pm_request_sram(phys_addr_t addr)
> > > > +{
> > > > +	int i;
> > > > +
> > > > +	for (i = 0; i < NUM_SRAMS; i++) {
> > > > +		if (zynqmp_banks[i].addr != addr)
> > > > +			continue;
> > > > +
> > > > +		return zynqmp_pm_request_node(zynqmp_banks[i].id,
> > > > +					      ZYNQMP_PM_CAPABILITY_ACCESS,
> > > 0,
> > > > +
> > > ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > > +	}
> > > > +
> > > > +	return -EINVAL;
> > > > +}
> > > > +
> > > > +/*
> > > > + * tcm_mem_alloc
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @mem: mem entry to initialize the va and da fields of
> > > > + *
> > > > + * Given TCM bank entry, this callback will set device address for R5
> > > > + * running on TCM and also setup virtual address for TCM bank
> > > > + * remoteproc carveout.
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int tcm_mem_alloc(struct rproc *rproc,
> > > > +			 struct rproc_mem_entry *mem)
> > > > +{
> > > > +	void *va;
> > > > +	struct device *dev = rproc->dev.parent;
> > > > +
> > > > +	va = ioremap_wc(mem->dma, mem->len);
> > > > +	if (IS_ERR_OR_NULL(va))
> > > > +		return -ENOMEM;
> > > > +
> > > > +	/* Update memory entry va */
> > > > +	mem->va = va;
> > > > +
> > > > +	/*
> > > > +	 * The R5s expect their TCM banks to be at address 0x0 and 0x2000,
> > > > +	 * while on the Linux side they are at 0xffexxxxx.
> > > > +	 *
> > > > +	 * Zero out the high 12 bits of the address. This will give
> > > > +	 * expected values for TCM Banks 0A and 0B (0x0 and 0x20000).
> > > > +	 */
> > > > +	mem->da &= 0x000fffff;
> > > > +
> > > > +	/*
> > > > +	 * TCM Banks 1A and 1B still have to be translated.
> > > > +	 *
> > > > +	 * Below handle these two banks' absolute addresses (0xffe90000 and
> > > > +	 * 0xffeb0000) and convert to the expected relative addresses
> > > > +	 * (0x0 and 0x20000).
> > > > +	 */
> > > > +	if (mem->da == 0x90000 || mem->da == 0xB0000)
> > > > +		mem->da -= 0x90000;
> > > > +
> > > > +	/* if translated TCM bank address is not valid report error */
> > > > +	if (mem->da != 0x0 && mem->da != 0x20000) {
> > > > +		dev_err(dev, "invalid TCM bank address: %x\n", mem->da);
> > > > +		return -EINVAL;
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * parse_tcm_banks()
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + *
> > > > + * Given R5 node in remoteproc instance
> > > > + * allocate remoteproc carveout for TCM memory
> > > > + * needed for firmware to be loaded
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int parse_tcm_banks(struct rproc *rproc)
> > > > +{
> > > > +	int i, num_banks;
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	struct device *dev = &rproc->dev;
> > > > +	struct device_node *r5_node = z_rproc->dev->of_node;
> > > > +
> > > > +	/* go through TCM banks for r5 node */
> > > > +	num_banks = of_count_phandle_with_args(r5_node,
> > > BANK_LIST_PROP, NULL);
> > > > +	if (num_banks <= 0) {
> > > > +		dev_err(dev, "need to specify TCM banks\n");
> > > > +		return -EINVAL;
> > > > +	}
> > > > +	for (i = 0; i < num_banks; i++) {
> > > > +		struct resource rsc;
> > > > +		resource_size_t size;
> > > > +		struct device_node *dt_node;
> > > > +		struct rproc_mem_entry *mem;
> > > > +		int ret;
> > > > +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> > > > +
> > > > +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> > > > +		if (!dt_node)
> > > > +			return -EINVAL;
> > > > +
> > > > +		ret = of_address_to_resource(dt_node, 0, &rsc);
> > > > +		if (ret < 0) {
> > > > +			of_node_put(dt_node);
> > > > +			return ret;
> > > > +		}
> > > > +		of_node_put(dt_node);
> > > > +
> > > > +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> > > > +		if (ret < 0)
> > > > +			return ret;
> > > > +
> > > > +		/* add carveout */
> > > > +		size = resource_size(&rsc);
> > > > +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> > > > +					   (size_t)size, rsc.start,
> > > > +					   tcm_mem_alloc,
> > > > +					   tcm_mem_release,
> > > > +					   rsc.name);
> > > > +		if (!mem) {
> > > > +			ret = zynqmp_pm_release_node(pnode_id);
> > >
> > > Where is @pnode_id initialised?
> > 
> > Good catch. Previously this was value set in an earlier iteration. I will fix this
> > as follows:
> > - release the R5 cluster by instead passing z_rproc->pnode_id
> > - release the TCM bank via call to release_sram
> > 
> > This will also remove the variable pnode_id.
> > 
> > >
> > > > +			if (ret)
> > > > +				dev_warn(dev,
> > > > +					 "fail to release node: %x ret: %x\n",
> > > > +					 pnode_id, ret);
> > > > +			return -ENOMEM;
> > > > +		}
> > > > +
> > > > +		mem->priv = (void *)(u64)pnode_id;
> > >
> > >
> > > And here too - how does this work when you test things out on your side?
> > >
> > > Please note that I will need a reply to these questions before I review
> > another
> > > set.
> > >
> > Here I will update as follows:
> > - set mem->priv to value from zynqmp_banks lookup table that contains
> > power node IDs
> > 
> > 
> > > > +		rproc_add_carveout(rproc, mem);
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_parse_fw()
> > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > + * @fw: ptr to firmware to be loaded onto r5 core
> > > > + *
> > > > + * When loading firmware, ensure the necessary carveouts are in
> > > remoteproc
> > > > + *
> > > > + * return 0 on success, otherwise non-zero value on failure
> > > > + */
> > > > +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct firmware
> > > *fw)
> > > > +{
> > > > +	int ret;
> > > > +
> > > > +	ret = parse_tcm_banks(rproc);
> > > > +	if (ret)
> > > > +		return ret;
> > > > +
> > > > +	ret = parse_mem_regions(rproc);
> > > > +	if (ret)
> > > > +		return ret;
> > > > +
> > > > +	ret = rproc_elf_load_rsc_table(rproc, fw);
> > > > +	if (ret == -EINVAL) {
> > > > +		/*
> > > > +		 * resource table only required for IPC.
> > > > +		 * if not present, this is not necessarily an error;
> > > > +		 * for example, loading r5 hello world application
> > > > +		 * so simply inform user and keep going.
> > > > +		 */
> > > > +		dev_info(&rproc->dev, "no resource table found.\n");
> > > > +		ret = 0;
> > > > +	}
> > > > +	return ret;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> > > > + * @rproc: r5 core's corresponding rproc structure
> > > > + * @vqid: virtqueue ID
> > > > + */
> > > > +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > +	struct device *dev = rproc->dev.parent;
> > > > +	struct zynqmp_ipi_message *mb_msg = NULL;
> > > > +	unsigned int skb_len;
> > > > +	struct sk_buff *skb;
> > > > +	int ret;
> > > > +
> > > > +	/* If vqid is negative, do not pass the vqid to
> > > > +	 * mailbox as vqid is supposed to be 0 or positive.
> > > > +	 * It also gives a way to just kick instead but
> > > > +	 * not use the IPI buffer.
> > > > +	 *
> > > > +	 * For now use negative vqid to assume no message will be
> > > > +	 * passed with IPI buffer, but just raise interrupt.
> > > > +	 * This will be faster as it doesn't need to copy the
> > > > +	 * message to the ZynqMP/Versal IPI message buffer.
> > > > +	 */
> > > > +	if (vqid >= 0) {
> > > > +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> > > > +		skb = alloc_skb(skb_len, GFP_KERNEL);
> > > > +		if (!skb)
> > > > +			return;
> > > > +
> > > > +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb,
> > > skb_len);
> > > > +		mb_msg->len = sizeof(vqid);
> > > > +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> > > > +
> > > > +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> > > > +	}
> > > > +
> > > > +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> > > > +	if (ret < 0) {
> > > > +		dev_warn(dev, "Failed to kick remote.\n");
> > > > +		if (mb_msg) {
> > > > +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> > > > +			kfree_skb(skb);
> > > > +		}
> > > > +	}
> > > > +}
> > > > +
> > >
> > > I am done reviewing this set.
> > >
> > > Thanks,
> > > Mathieu
> > >
> > > > +static struct rproc_ops zynqmp_r5_rproc_ops = {
> > > > +	.start		= zynqmp_r5_rproc_start,
> > > > +	.stop		= zynqmp_r5_rproc_stop,
> > > > +	.load		= rproc_elf_load_segments,
> > > > +	.parse_fw	= zynqmp_r5_parse_fw,
> > > > +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > > > +	.sanity_check	= rproc_elf_sanity_check,
> > > > +	.get_boot_addr	= rproc_elf_get_boot_addr,
> > > > +	.kick		= zynqmp_r5_rproc_kick,
> > > > +};
> > > > +
> > > > +/**
> > > > + * event_notified_idr_cb() - event notified idr callback
> > > > + * @id: idr id
> > > > + * @ptr: pointer to idr private data
> > > > + * @data: data passed to idr_for_each callback
> > > > + *
> > > > + * Pass notification to remoteproc virtio
> > > > + *
> > > > + * Return: 0. having return is to satisfy the idr_for_each() function
> > > > + *          pointer input argument requirement.
> > > > + **/
> > > > +static int event_notified_idr_cb(int id, void *ptr, void *data)
> > > > +{
> > > > +	struct rproc *rproc = data;
> > > > +
> > > > +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> > > > +		dev_dbg(&rproc->dev, "no message was found in idr with id:
> > > %d\n", id);
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/**
> > > > + * handle_event_notified() - remoteproc notification work function
> > > > + * @work: pointer to the work structure
> > > > + *
> > > > + * It checks each registered remoteproc notify IDs.
> > > > + */
> > > > +static void handle_event_notified(struct work_struct *work)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +	struct rproc *rproc;
> > > > +
> > > > +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> > > > +	rproc = z_rproc->rproc;
> > > > +
> > > > +	/*
> > > > +	 * This is to simply raise interrupt as ACK for remote.
> > > > +	 *
> > > > +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> > > > +	 * mapping to remote processor.
> > > > +	 */
> > > > +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> > > > +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> > > > +
> > > > +	/*
> > > > +	 * We only use IPI for interrupt. The firmware side may or may
> > > > +	 * not write the notifyid when it trigger IPI.
> > > > +	 * And thus, we scan through all the registered notifyids.
> > > > +	 */
> > > > +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> > > > + * @cl: mailbox client
> > > > + * @msg: message pointer
> > > > + *
> > > > + * It will schedule the R5 notification work.
> > > > + */
> > > > +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +
> > > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> > > > +	/*
> > > > +	 * Notification can be received with message payload.
> > > > +	 * To handle this, check for message contents.
> > > > +	 * If there are contents, copy to ipi message buffer payload location.
> > > > +	 *
> > > > +	 * If message is empty, then that means that was simple raising
> > > > +	 * of interrupt. no payload to process.
> > > > +	 *
> > > > +	 * Note: enqueue work regardless of msg as kick may imply that
> > > remote
> > > > +	 * is waiting for ack.
> > > > +	 */
> > > > +	if (msg) {
> > > > +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> > > > +		size_t len;
> > > > +
> > > > +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> > > > +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> > > > +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> > > > +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> > > > +		buf_msg->len = len;
> > > > +		memcpy(buf_msg->data, ipi_msg->data, len);
> > > > +	}
> > > > +	schedule_work(&z_rproc->mbox_work);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> > > > + * @cl: mailbox client
> > > > + * @msg: pointer to the message which has been sent
> > > > + * @r: status of last TX - OK or error
> > > > + *
> > > > + * It will be called by the mailbox framework when the last TX has done.
> > > > + */
> > > > +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg,
> > int
> > > r)
> > > > +{
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +	struct sk_buff *skb;
> > > > +
> > > > +	if (!msg)
> > > > +		return;
> > > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> > > > +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> > > > +	kfree_skb(skb);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_setup_mbox() - Setup mailboxes
> > > > + *			    this is used for each individual R5 core
> > > > + *
> > > > + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> > > > + * @node: pointer of the device node
> > > > + *
> > > > + * Function to setup mailboxes to talk to RPU.
> > > > + *
> > > > + * Return: 0 for success, negative value for failure.
> > > > + */
> > > > +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> > > > +				struct device_node *node)
> > > > +{
> > > > +	struct mbox_client *mclient;
> > > > +
> > > > +	/* Setup TX mailbox channel client */
> > > > +	mclient			= &z_rproc->tx_mc;
> > > > +	mclient->dev		= z_rproc->dev;
> > > > +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> > > > +	mclient->tx_block	= false;
> > > > +	mclient->knows_txdone	= false;
> > > > +	mclient->rx_callback	= NULL;
> > > > +
> > > > +	/* Setup RX mailbox channel client */
> > > > +	mclient			= &z_rproc->rx_mc;
> > > > +	mclient->dev		= z_rproc->dev;
> > > > +	mclient->tx_done	= NULL;
> > > > +	mclient->tx_block	= false;
> > > > +	mclient->knows_txdone	= false;
> > > > +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> > > > +
> > > > +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> > > > +
> > > > +	/* Request TX and RX channels */
> > > > +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc-
> > > >tx_mc, "tx");
> > > > +	if (IS_ERR(z_rproc->tx_chan)) {
> > > > +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> > > > +		return PTR_ERR(z_rproc->tx_chan);
> > > > +	}
> > > > +
> > > > +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc-
> > > >rx_mc, "rx");
> > > > +	if (IS_ERR(z_rproc->rx_chan)) {
> > > > +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> > > > +		return PTR_ERR(z_rproc->rx_chan);
> > > > +	}
> > > > +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc *z_rproc)
> > > > +{
> > > > +	mbox_free_channel(z_rproc->tx_chan);
> > > > +	mbox_free_channel(z_rproc->rx_chan);
> > > > +}
> > > > +
> > > > +/**
> > > > + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> > > > + *		       this is called for each individual R5 core to
> > > > + *		       set up mailbox, Xilinx platform manager unique ID,
> > > > + *		       add to rproc core
> > > > + *
> > > > + * @pdev: domain platform device for current R5 core
> > > > + * @node: pointer of the device node for current R5 core
> > > > + * @rpu_mode: mode to configure RPU, split or lockstep
> > > > + *
> > > > + * Return: 0 for success, negative value for failure.
> > > > + */
> > > > +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device
> > > *pdev,
> > > > +					       struct device_node *node,
> > > > +					       enum rpu_oper_mode
> > > rpu_mode)
> > > > +{
> > > > +	int ret;
> > > > +	struct device *dev = &pdev->dev;
> > > > +	struct rproc *rproc_ptr;
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +
> > > > +	/* Allocate remoteproc instance */
> > > > +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev),
> > > &zynqmp_r5_rproc_ops,
> > > > +				     NULL, sizeof(struct zynqmp_r5_rproc));
> > > > +	if (!rproc_ptr) {
> > > > +		ret = -ENOMEM;
> > > > +		goto error;
> > > > +	}
> > > > +
> > > > +	rproc_ptr->auto_boot = false;
> > > > +	z_rproc = rproc_ptr->priv;
> > > > +	z_rproc->rproc = rproc_ptr;
> > > > +	z_rproc->dev = dev;
> > > > +
> > > > +	/* Set up DMA mask */
> > > > +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> > > > +	if (ret)
> > > > +		goto error;
> > > > +
> > > > +	/* Get R5 power domain node */
> > > > +	ret = of_property_read_u32(node, "power-domain", &z_rproc-
> > > >pnode_id);
> > > > +	if (ret)
> > > > +		goto error;
> > > > +
> > > > +	ret = r5_set_mode(z_rproc, rpu_mode);
> > > > +	if (ret)
> > > > +		goto error;
> > > > +
> > > > +	if (of_property_read_bool(node, "mboxes")) {
> > > > +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> > > > +		if (ret)
> > > > +			goto error;
> > > > +	}
> > > > +
> > > > +	/* Add R5 remoteproc */
> > > > +	ret = devm_rproc_add(dev, rproc_ptr);
> > > > +	if (ret) {
> > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > +		goto error;
> > > > +	}
> > > > +
> > > > +	return z_rproc;
> > > > +error:
> > > > +	return ERR_PTR(ret);
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_remoteproc_probe()
> > > > + *
> > > > + * @pdev: domain platform device for R5 cluster
> > > > + *
> > > > + * called when driver is probed, for each R5 core specified in DT,
> > > > + * setup as needed to do remoteproc-related operations
> > > > + *
> > > > + * Return: 0 for success, negative value for failure.
> > > > + */
> > > > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > > > +{
> > > > +	int ret, core_count;
> > > > +	struct device *dev = &pdev->dev;
> > > > +	struct device_node *nc;
> > > > +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> > > > +	struct list_head *cluster; /* list to track each core's rproc */
> > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > +	struct platform_device *child_pdev;
> > > > +	struct list_head *pos;
> > > > +
> > > > +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode",
> > > &rpu_mode);
> > > > +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> > > > +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> > > > +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> > > > +			ret, rpu_mode);
> > > > +		return ret;
> > > > +	}
> > > > +
> > > > +	dev_dbg(dev, "RPU configuration: %s\n",
> > > > +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" :
> > > "split");
> > > > +
> > > > +	/*
> > > > +	 * if 2 RPUs provided but one is lockstep, then we have an
> > > > +	 * invalid configuration.
> > > > +	 */
> > > > +
> > > > +	core_count = of_get_available_child_count(dev->of_node);
> > > > +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> > > > +	    core_count > MAX_RPROCS)
> > > > +		return -EINVAL;
> > > > +
> > > > +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > > > +	if (!cluster)
> > > > +		return -ENOMEM;
> > > > +	INIT_LIST_HEAD(cluster);
> > > > +
> > > > +	ret = devm_of_platform_populate(dev);
> > > > +	if (ret) {
> > > > +		dev_err(dev, "devm_of_platform_populate failed, ret =
> > > %d\n",
> > > > +			ret);
> > > > +		return ret;
> > > > +	}
> > > > +
> > > > +	/* probe each individual r5 core's remoteproc-related info */
> > > > +	for_each_available_child_of_node(dev->of_node, nc) {
> > > > +		child_pdev = of_find_device_by_node(nc);
> > > > +		if (!child_pdev) {
> > > > +			dev_err(dev, "could not get R5 core platform
> > > device\n");
> > > > +			ret = -ENODEV;
> > > > +			goto out;
> > > > +		}
> > > > +
> > > > +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> > > > +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> > > > +			ret ? "Failed" : "Able",
> > > > +			nc);
> > > > +		if (IS_ERR(z_rproc)) {
> > > > +			ret = PTR_ERR(z_rproc);
> > > > +			goto out;
> > > > +		}
> > > > +		list_add_tail(&z_rproc->elem, cluster);
> > > > +	}
> > > > +	/* wire in so each core can be cleaned up at driver remove */
> > > > +	platform_set_drvdata(pdev, cluster);
> > > > +	return 0;
> > > > +out:
> > > > +	/*
> > > > +	 * undo core0 upon any failures on core1 in split-mode
> > > > +	 *
> > > > +	 * in zynqmp_r5_probe z_rproc is set to null
> > > > +	 * and ret to non-zero value if error
> > > > +	 */
> > > > +	list_for_each(pos, cluster) {
> > > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > +	}
> > > > +	return ret;
> > > > +}
> > > > +
> > > > +/*
> > > > + * zynqmp_r5_remoteproc_remove()
> > > > + *
> > > > + * @pdev: domain platform device for R5 cluster
> > > > + *
> > > > + * When the driver is unloaded, clean up the mailboxes for each
> > > > + * remoteproc that was initially probed.
> > > > + */
> > > > +static int zynqmp_r5_remoteproc_remove(struct platform_device *pdev)
> > > > +{
> > > > +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> > > > +						 platform_get_drvdata(pdev);
> > > > +	struct zynqmp_r5_rproc *z_rproc = NULL;
> > > > +
> > > > +	list_for_each_safe(pos, temp, cluster) {
> > > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > +		list_del(pos);
> > > > +	}
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +/* Match table for OF platform binding */
> > > > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > > > +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> > > > +	{ /* end of list */ },
> > > > +};
> > > > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > > > +
> > > > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > > > +	.probe = zynqmp_r5_remoteproc_probe,
> > > > +	.remove = zynqmp_r5_remoteproc_remove,
> > > > +	.driver = {
> > > > +		.name = "zynqmp_r5_remoteproc",
> > > > +		.of_match_table = zynqmp_r5_remoteproc_match,
> > > > +	},
> > > > +};
> > > > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > > > +
> > > > +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> > > > +MODULE_LICENSE("GPL v2");
> > > > --
> > > > 2.17.1
> > > >

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
  2021-02-17 21:34           ` Ben Levinsky
@ 2021-02-17 21:58             ` Mathieu Poirier
  -1 siblings, 0 replies; 28+ messages in thread
From: Mathieu Poirier @ 2021-02-17 21:58 UTC (permalink / raw)
  To: Ben Levinsky
  Cc: devicetree, linux-remoteproc, linux-kernel, linux-arm-kernel,
	Michal Simek

On Wed, Feb 17, 2021 at 09:34:13PM +0000, Ben Levinsky wrote:
> 1 more for the pnode_id comment
>

[...]
 
> > > > > +static int parse_tcm_banks(struct rproc *rproc)
> > > > > +{
> > > > > +	int i, num_banks;
> > > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > > +	struct device *dev = &rproc->dev;
> > > > > +	struct device_node *r5_node = z_rproc->dev->of_node;
> > > > > +
> > > > > +	/* go through TCM banks for r5 node */
> > > > > +	num_banks = of_count_phandle_with_args(r5_node,
> > > > BANK_LIST_PROP, NULL);
> > > > > +	if (num_banks <= 0) {
> > > > > +		dev_err(dev, "need to specify TCM banks\n");
> > > > > +		return -EINVAL;
> > > > > +	}
> > > > > +	for (i = 0; i < num_banks; i++) {
> > > > > +		struct resource rsc;
> > > > > +		resource_size_t size;
> > > > > +		struct device_node *dt_node;
> > > > > +		struct rproc_mem_entry *mem;
> > > > > +		int ret;
> > > > > +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> > > > > +
> > > > > +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> > > > > +		if (!dt_node)
> > > > > +			return -EINVAL;
> > > > > +
> > > > > +		ret = of_address_to_resource(dt_node, 0, &rsc);
> > > > > +		if (ret < 0) {
> > > > > +			of_node_put(dt_node);
> > > > > +			return ret;
> > > > > +		}
> > > > > +		of_node_put(dt_node);
> > > > > +
> > > > > +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> > > > > +		if (ret < 0)
> > > > > +			return ret;
> > > > > +
> > > > > +		/* add carveout */
> > > > > +		size = resource_size(&rsc);
> > > > > +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> > > > > +					   (size_t)size, rsc.start,
> > > > > +					   tcm_mem_alloc,
> > > > > +					   tcm_mem_release,
> > > > > +					   rsc.name);
> > > > > +		if (!mem) {
> > > > > +			ret = zynqmp_pm_release_node(pnode_id);
> > > >
> > > > Where is @pnode_id initialised?
> > >
> > > Good catch. Previously this was value set in an earlier iteration. I will fix this
> > > as follows:
> > > - release the R5 cluster by instead passing z_rproc->pnode_id
> > > - release the TCM bank via call to release_sram
> > >
> > > This will also remove the variable pnode_id.
> > >
> 
> 
> typo release_sram should be release_node()
> 
> also instead another, cleaner way might be as follows:
> change zynqmp_r5_request_sram so that it takes 1 more arg to return by reference the TCM bank's Xilinx platform management ID and then set the TCM bank's ID with that information.
> 

That sounds more reasonable.

> Thanks
> Ben
> 
> > > >
> > > > > +			if (ret)
> > > > > +				dev_warn(dev,
> > > > > +					 "fail to release node: %x ret: %x\n",
> > > > > +					 pnode_id, ret);
> > > > > +			return -ENOMEM;
> > > > > +		}
> > > > > +
> > > > > +		mem->priv = (void *)(u64)pnode_id;
> > > >
> > > >
> > > > And here too - how does this work when you test things out on your side?
> > > >
> > > > Please note that I will need a reply to these questions before I review
> > > another
> > > > set.
> > > >
> > > Here I will update as follows:
> > > - set mem->priv to value from zynqmp_banks lookup table that contains
> > > power node IDs
> > >
> > >
> > > > > +		rproc_add_carveout(rproc, mem);
> > > > > +	}
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * zynqmp_r5_parse_fw()
> > > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > > + * @fw: ptr to firmware to be loaded onto r5 core
> > > > > + *
> > > > > + * When loading firmware, ensure the necessary carveouts are in
> > > > remoteproc
> > > > > + *
> > > > > + * return 0 on success, otherwise non-zero value on failure
> > > > > + */
> > > > > +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct
> > firmware
> > > > *fw)
> > > > > +{
> > > > > +	int ret;
> > > > > +
> > > > > +	ret = parse_tcm_banks(rproc);
> > > > > +	if (ret)
> > > > > +		return ret;
> > > > > +
> > > > > +	ret = parse_mem_regions(rproc);
> > > > > +	if (ret)
> > > > > +		return ret;
> > > > > +
> > > > > +	ret = rproc_elf_load_rsc_table(rproc, fw);
> > > > > +	if (ret == -EINVAL) {
> > > > > +		/*
> > > > > +		 * resource table only required for IPC.
> > > > > +		 * if not present, this is not necessarily an error;
> > > > > +		 * for example, loading r5 hello world application
> > > > > +		 * so simply inform user and keep going.
> > > > > +		 */
> > > > > +		dev_info(&rproc->dev, "no resource table found.\n");
> > > > > +		ret = 0;
> > > > > +	}
> > > > > +	return ret;
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> > > > > + * @rproc: r5 core's corresponding rproc structure
> > > > > + * @vqid: virtqueue ID
> > > > > + */
> > > > > +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> > > > > +{
> > > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > > +	struct device *dev = rproc->dev.parent;
> > > > > +	struct zynqmp_ipi_message *mb_msg = NULL;
> > > > > +	unsigned int skb_len;
> > > > > +	struct sk_buff *skb;
> > > > > +	int ret;
> > > > > +
> > > > > +	/* If vqid is negative, do not pass the vqid to
> > > > > +	 * mailbox as vqid is supposed to be 0 or positive.
> > > > > +	 * It also gives a way to just kick instead but
> > > > > +	 * not use the IPI buffer.
> > > > > +	 *
> > > > > +	 * For now use negative vqid to assume no message will be
> > > > > +	 * passed with IPI buffer, but just raise interrupt.
> > > > > +	 * This will be faster as it doesn't need to copy the
> > > > > +	 * message to the ZynqMP/Versal IPI message buffer.
> > > > > +	 */
> > > > > +	if (vqid >= 0) {
> > > > > +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> > > > > +		skb = alloc_skb(skb_len, GFP_KERNEL);
> > > > > +		if (!skb)
> > > > > +			return;
> > > > > +
> > > > > +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb,
> > > > skb_len);
> > > > > +		mb_msg->len = sizeof(vqid);
> > > > > +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> > > > > +
> > > > > +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> > > > > +	}
> > > > > +
> > > > > +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> > > > > +	if (ret < 0) {
> > > > > +		dev_warn(dev, "Failed to kick remote.\n");
> > > > > +		if (mb_msg) {
> > > > > +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> > > > > +			kfree_skb(skb);
> > > > > +		}
> > > > > +	}
> > > > > +}
> > > > > +
> > > >
> > > > I am done reviewing this set.
> > > >
> > > > Thanks,
> > > > Mathieu
> > > >
> > > > > +static struct rproc_ops zynqmp_r5_rproc_ops = {
> > > > > +	.start		= zynqmp_r5_rproc_start,
> > > > > +	.stop		= zynqmp_r5_rproc_stop,
> > > > > +	.load		= rproc_elf_load_segments,
> > > > > +	.parse_fw	= zynqmp_r5_parse_fw,
> > > > > +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > > > > +	.sanity_check	= rproc_elf_sanity_check,
> > > > > +	.get_boot_addr	= rproc_elf_get_boot_addr,
> > > > > +	.kick		= zynqmp_r5_rproc_kick,
> > > > > +};
> > > > > +
> > > > > +/**
> > > > > + * event_notified_idr_cb() - event notified idr callback
> > > > > + * @id: idr id
> > > > > + * @ptr: pointer to idr private data
> > > > > + * @data: data passed to idr_for_each callback
> > > > > + *
> > > > > + * Pass notification to remoteproc virtio
> > > > > + *
> > > > > + * Return: 0. having return is to satisfy the idr_for_each() function
> > > > > + *          pointer input argument requirement.
> > > > > + **/
> > > > > +static int event_notified_idr_cb(int id, void *ptr, void *data)
> > > > > +{
> > > > > +	struct rproc *rproc = data;
> > > > > +
> > > > > +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> > > > > +		dev_dbg(&rproc->dev, "no message was found in idr with id:
> > > > %d\n", id);
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * handle_event_notified() - remoteproc notification work function
> > > > > + * @work: pointer to the work structure
> > > > > + *
> > > > > + * It checks each registered remoteproc notify IDs.
> > > > > + */
> > > > > +static void handle_event_notified(struct work_struct *work)
> > > > > +{
> > > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > > +	struct rproc *rproc;
> > > > > +
> > > > > +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> > > > > +	rproc = z_rproc->rproc;
> > > > > +
> > > > > +	/*
> > > > > +	 * This is to simply raise interrupt as ACK for remote.
> > > > > +	 *
> > > > > +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> > > > > +	 * mapping to remote processor.
> > > > > +	 */
> > > > > +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> > > > > +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> > > > > +
> > > > > +	/*
> > > > > +	 * We only use IPI for interrupt. The firmware side may or may
> > > > > +	 * not write the notifyid when it trigger IPI.
> > > > > +	 * And thus, we scan through all the registered notifyids.
> > > > > +	 */
> > > > > +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> > > > > + * @cl: mailbox client
> > > > > + * @msg: message pointer
> > > > > + *
> > > > > + * It will schedule the R5 notification work.
> > > > > + */
> > > > > +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> > > > > +{
> > > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > > +
> > > > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> > > > > +	/*
> > > > > +	 * Notification can be received with message payload.
> > > > > +	 * To handle this, check for message contents.
> > > > > +	 * If there are contents, copy to ipi message buffer payload location.
> > > > > +	 *
> > > > > +	 * If message is empty, then that means that was simple raising
> > > > > +	 * of interrupt. no payload to process.
> > > > > +	 *
> > > > > +	 * Note: enqueue work regardless of msg as kick may imply that
> > > > remote
> > > > > +	 * is waiting for ack.
> > > > > +	 */
> > > > > +	if (msg) {
> > > > > +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> > > > > +		size_t len;
> > > > > +
> > > > > +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> > > > > +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> > > > > +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> > > > > +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> > > > > +		buf_msg->len = len;
> > > > > +		memcpy(buf_msg->data, ipi_msg->data, len);
> > > > > +	}
> > > > > +	schedule_work(&z_rproc->mbox_work);
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> > > > > + * @cl: mailbox client
> > > > > + * @msg: pointer to the message which has been sent
> > > > > + * @r: status of last TX - OK or error
> > > > > + *
> > > > > + * It will be called by the mailbox framework when the last TX has
> > done.
> > > > > + */
> > > > > +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg,
> > > int
> > > > r)
> > > > > +{
> > > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > > +	struct sk_buff *skb;
> > > > > +
> > > > > +	if (!msg)
> > > > > +		return;
> > > > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> > > > > +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> > > > > +	kfree_skb(skb);
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * zynqmp_r5_setup_mbox() - Setup mailboxes
> > > > > + *			    this is used for each individual R5 core
> > > > > + *
> > > > > + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> > > > > + * @node: pointer of the device node
> > > > > + *
> > > > > + * Function to setup mailboxes to talk to RPU.
> > > > > + *
> > > > > + * Return: 0 for success, negative value for failure.
> > > > > + */
> > > > > +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> > > > > +				struct device_node *node)
> > > > > +{
> > > > > +	struct mbox_client *mclient;
> > > > > +
> > > > > +	/* Setup TX mailbox channel client */
> > > > > +	mclient			= &z_rproc->tx_mc;
> > > > > +	mclient->dev		= z_rproc->dev;
> > > > > +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> > > > > +	mclient->tx_block	= false;
> > > > > +	mclient->knows_txdone	= false;
> > > > > +	mclient->rx_callback	= NULL;
> > > > > +
> > > > > +	/* Setup RX mailbox channel client */
> > > > > +	mclient			= &z_rproc->rx_mc;
> > > > > +	mclient->dev		= z_rproc->dev;
> > > > > +	mclient->tx_done	= NULL;
> > > > > +	mclient->tx_block	= false;
> > > > > +	mclient->knows_txdone	= false;
> > > > > +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> > > > > +
> > > > > +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> > > > > +
> > > > > +	/* Request TX and RX channels */
> > > > > +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc-
> > > > >tx_mc, "tx");
> > > > > +	if (IS_ERR(z_rproc->tx_chan)) {
> > > > > +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> > > > > +		return PTR_ERR(z_rproc->tx_chan);
> > > > > +	}
> > > > > +
> > > > > +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc-
> > > > >rx_mc, "rx");
> > > > > +	if (IS_ERR(z_rproc->rx_chan)) {
> > > > > +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> > > > > +		return PTR_ERR(z_rproc->rx_chan);
> > > > > +	}
> > > > > +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc
> > *z_rproc)
> > > > > +{
> > > > > +	mbox_free_channel(z_rproc->tx_chan);
> > > > > +	mbox_free_channel(z_rproc->rx_chan);
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> > > > > + *		       this is called for each individual R5 core to
> > > > > + *		       set up mailbox, Xilinx platform manager unique
> > ID,
> > > > > + *		       add to rproc core
> > > > > + *
> > > > > + * @pdev: domain platform device for current R5 core
> > > > > + * @node: pointer of the device node for current R5 core
> > > > > + * @rpu_mode: mode to configure RPU, split or lockstep
> > > > > + *
> > > > > + * Return: 0 for success, negative value for failure.
> > > > > + */
> > > > > +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct
> > platform_device
> > > > *pdev,
> > > > > +					       struct device_node *node,
> > > > > +					       enum rpu_oper_mode
> > > > rpu_mode)
> > > > > +{
> > > > > +	int ret;
> > > > > +	struct device *dev = &pdev->dev;
> > > > > +	struct rproc *rproc_ptr;
> > > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > > +
> > > > > +	/* Allocate remoteproc instance */
> > > > > +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev),
> > > > &zynqmp_r5_rproc_ops,
> > > > > +				     NULL, sizeof(struct zynqmp_r5_rproc));
> > > > > +	if (!rproc_ptr) {
> > > > > +		ret = -ENOMEM;
> > > > > +		goto error;
> > > > > +	}
> > > > > +
> > > > > +	rproc_ptr->auto_boot = false;
> > > > > +	z_rproc = rproc_ptr->priv;
> > > > > +	z_rproc->rproc = rproc_ptr;
> > > > > +	z_rproc->dev = dev;
> > > > > +
> > > > > +	/* Set up DMA mask */
> > > > > +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> > > > > +	if (ret)
> > > > > +		goto error;
> > > > > +
> > > > > +	/* Get R5 power domain node */
> > > > > +	ret = of_property_read_u32(node, "power-domain", &z_rproc-
> > > > >pnode_id);
> > > > > +	if (ret)
> > > > > +		goto error;
> > > > > +
> > > > > +	ret = r5_set_mode(z_rproc, rpu_mode);
> > > > > +	if (ret)
> > > > > +		goto error;
> > > > > +
> > > > > +	if (of_property_read_bool(node, "mboxes")) {
> > > > > +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> > > > > +		if (ret)
> > > > > +			goto error;
> > > > > +	}
> > > > > +
> > > > > +	/* Add R5 remoteproc */
> > > > > +	ret = devm_rproc_add(dev, rproc_ptr);
> > > > > +	if (ret) {
> > > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > > +		goto error;
> > > > > +	}
> > > > > +
> > > > > +	return z_rproc;
> > > > > +error:
> > > > > +	return ERR_PTR(ret);
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * zynqmp_r5_remoteproc_probe()
> > > > > + *
> > > > > + * @pdev: domain platform device for R5 cluster
> > > > > + *
> > > > > + * called when driver is probed, for each R5 core specified in DT,
> > > > > + * setup as needed to do remoteproc-related operations
> > > > > + *
> > > > > + * Return: 0 for success, negative value for failure.
> > > > > + */
> > > > > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > > > > +{
> > > > > +	int ret, core_count;
> > > > > +	struct device *dev = &pdev->dev;
> > > > > +	struct device_node *nc;
> > > > > +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> > > > > +	struct list_head *cluster; /* list to track each core's rproc */
> > > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > > +	struct platform_device *child_pdev;
> > > > > +	struct list_head *pos;
> > > > > +
> > > > > +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode",
> > > > &rpu_mode);
> > > > > +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> > > > > +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> > > > > +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> > > > > +			ret, rpu_mode);
> > > > > +		return ret;
> > > > > +	}
> > > > > +
> > > > > +	dev_dbg(dev, "RPU configuration: %s\n",
> > > > > +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" :
> > > > "split");
> > > > > +
> > > > > +	/*
> > > > > +	 * if 2 RPUs provided but one is lockstep, then we have an
> > > > > +	 * invalid configuration.
> > > > > +	 */
> > > > > +
> > > > > +	core_count = of_get_available_child_count(dev->of_node);
> > > > > +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> > > > > +	    core_count > MAX_RPROCS)
> > > > > +		return -EINVAL;
> > > > > +
> > > > > +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > > > > +	if (!cluster)
> > > > > +		return -ENOMEM;
> > > > > +	INIT_LIST_HEAD(cluster);
> > > > > +
> > > > > +	ret = devm_of_platform_populate(dev);
> > > > > +	if (ret) {
> > > > > +		dev_err(dev, "devm_of_platform_populate failed, ret =
> > > > %d\n",
> > > > > +			ret);
> > > > > +		return ret;
> > > > > +	}
> > > > > +
> > > > > +	/* probe each individual r5 core's remoteproc-related info */
> > > > > +	for_each_available_child_of_node(dev->of_node, nc) {
> > > > > +		child_pdev = of_find_device_by_node(nc);
> > > > > +		if (!child_pdev) {
> > > > > +			dev_err(dev, "could not get R5 core platform
> > > > device\n");
> > > > > +			ret = -ENODEV;
> > > > > +			goto out;
> > > > > +		}
> > > > > +
> > > > > +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> > > > > +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> > > > > +			ret ? "Failed" : "Able",
> > > > > +			nc);
> > > > > +		if (IS_ERR(z_rproc)) {
> > > > > +			ret = PTR_ERR(z_rproc);
> > > > > +			goto out;
> > > > > +		}
> > > > > +		list_add_tail(&z_rproc->elem, cluster);
> > > > > +	}
> > > > > +	/* wire in so each core can be cleaned up at driver remove */
> > > > > +	platform_set_drvdata(pdev, cluster);
> > > > > +	return 0;
> > > > > +out:
> > > > > +	/*
> > > > > +	 * undo core0 upon any failures on core1 in split-mode
> > > > > +	 *
> > > > > +	 * in zynqmp_r5_probe z_rproc is set to null
> > > > > +	 * and ret to non-zero value if error
> > > > > +	 */
> > > > > +	list_for_each(pos, cluster) {
> > > > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > > +	}
> > > > > +	return ret;
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * zynqmp_r5_remoteproc_remove()
> > > > > + *
> > > > > + * @pdev: domain platform device for R5 cluster
> > > > > + *
> > > > > + * When the driver is unloaded, clean up the mailboxes for each
> > > > > + * remoteproc that was initially probed.
> > > > > + */
> > > > > +static int zynqmp_r5_remoteproc_remove(struct platform_device
> > *pdev)
> > > > > +{
> > > > > +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> > > > > +						 platform_get_drvdata(pdev);
> > > > > +	struct zynqmp_r5_rproc *z_rproc = NULL;
> > > > > +
> > > > > +	list_for_each_safe(pos, temp, cluster) {
> > > > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > > +		list_del(pos);
> > > > > +	}
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +/* Match table for OF platform binding */
> > > > > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > > > > +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> > > > > +	{ /* end of list */ },
> > > > > +};
> > > > > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > > > > +
> > > > > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > > > > +	.probe = zynqmp_r5_remoteproc_probe,
> > > > > +	.remove = zynqmp_r5_remoteproc_remove,
> > > > > +	.driver = {
> > > > > +		.name = "zynqmp_r5_remoteproc",
> > > > > +		.of_match_table = zynqmp_r5_remoteproc_match,
> > > > > +	},
> > > > > +};
> > > > > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > > > > +
> > > > > +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> > > > > +MODULE_LICENSE("GPL v2");
> > > > > --
> > > > > 2.17.1
> > > > >

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
@ 2021-02-17 21:58             ` Mathieu Poirier
  0 siblings, 0 replies; 28+ messages in thread
From: Mathieu Poirier @ 2021-02-17 21:58 UTC (permalink / raw)
  To: Ben Levinsky
  Cc: devicetree, Michal Simek, linux-remoteproc, linux-kernel,
	linux-arm-kernel

On Wed, Feb 17, 2021 at 09:34:13PM +0000, Ben Levinsky wrote:
> 1 more for the pnode_id comment
>

[...]
 
> > > > > +static int parse_tcm_banks(struct rproc *rproc)
> > > > > +{
> > > > > +	int i, num_banks;
> > > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > > +	struct device *dev = &rproc->dev;
> > > > > +	struct device_node *r5_node = z_rproc->dev->of_node;
> > > > > +
> > > > > +	/* go through TCM banks for r5 node */
> > > > > +	num_banks = of_count_phandle_with_args(r5_node,
> > > > BANK_LIST_PROP, NULL);
> > > > > +	if (num_banks <= 0) {
> > > > > +		dev_err(dev, "need to specify TCM banks\n");
> > > > > +		return -EINVAL;
> > > > > +	}
> > > > > +	for (i = 0; i < num_banks; i++) {
> > > > > +		struct resource rsc;
> > > > > +		resource_size_t size;
> > > > > +		struct device_node *dt_node;
> > > > > +		struct rproc_mem_entry *mem;
> > > > > +		int ret;
> > > > > +		u32 pnode_id; /* zynqmp_pm* fn's expect u32 */
> > > > > +
> > > > > +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
> > > > > +		if (!dt_node)
> > > > > +			return -EINVAL;
> > > > > +
> > > > > +		ret = of_address_to_resource(dt_node, 0, &rsc);
> > > > > +		if (ret < 0) {
> > > > > +			of_node_put(dt_node);
> > > > > +			return ret;
> > > > > +		}
> > > > > +		of_node_put(dt_node);
> > > > > +
> > > > > +		ret = zynqmp_r5_pm_request_sram(rsc.start);
> > > > > +		if (ret < 0)
> > > > > +			return ret;
> > > > > +
> > > > > +		/* add carveout */
> > > > > +		size = resource_size(&rsc);
> > > > > +		mem = rproc_mem_entry_init(dev, NULL, rsc.start,
> > > > > +					   (size_t)size, rsc.start,
> > > > > +					   tcm_mem_alloc,
> > > > > +					   tcm_mem_release,
> > > > > +					   rsc.name);
> > > > > +		if (!mem) {
> > > > > +			ret = zynqmp_pm_release_node(pnode_id);
> > > >
> > > > Where is @pnode_id initialised?
> > >
> > > Good catch. Previously this was value set in an earlier iteration. I will fix this
> > > as follows:
> > > - release the R5 cluster by instead passing z_rproc->pnode_id
> > > - release the TCM bank via call to release_sram
> > >
> > > This will also remove the variable pnode_id.
> > >
> 
> 
> typo release_sram should be release_node()
> 
> also instead another, cleaner way might be as follows:
> change zynqmp_r5_request_sram so that it takes 1 more arg to return by reference the TCM bank's Xilinx platform management ID and then set the TCM bank's ID with that information.
> 

That sounds more reasonable.

> Thanks
> Ben
> 
> > > >
> > > > > +			if (ret)
> > > > > +				dev_warn(dev,
> > > > > +					 "fail to release node: %x ret: %x\n",
> > > > > +					 pnode_id, ret);
> > > > > +			return -ENOMEM;
> > > > > +		}
> > > > > +
> > > > > +		mem->priv = (void *)(u64)pnode_id;
> > > >
> > > >
> > > > And here too - how does this work when you test things out on your side?
> > > >
> > > > Please note that I will need a reply to these questions before I review
> > > another
> > > > set.
> > > >
> > > Here I will update as follows:
> > > - set mem->priv to value from zynqmp_banks lookup table that contains
> > > power node IDs
> > >
> > >
> > > > > +		rproc_add_carveout(rproc, mem);
> > > > > +	}
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * zynqmp_r5_parse_fw()
> > > > > + * @rproc: single R5 core's corresponding rproc instance
> > > > > + * @fw: ptr to firmware to be loaded onto r5 core
> > > > > + *
> > > > > + * When loading firmware, ensure the necessary carveouts are in
> > > > remoteproc
> > > > > + *
> > > > > + * return 0 on success, otherwise non-zero value on failure
> > > > > + */
> > > > > +static int zynqmp_r5_parse_fw(struct rproc *rproc, const struct
> > firmware
> > > > *fw)
> > > > > +{
> > > > > +	int ret;
> > > > > +
> > > > > +	ret = parse_tcm_banks(rproc);
> > > > > +	if (ret)
> > > > > +		return ret;
> > > > > +
> > > > > +	ret = parse_mem_regions(rproc);
> > > > > +	if (ret)
> > > > > +		return ret;
> > > > > +
> > > > > +	ret = rproc_elf_load_rsc_table(rproc, fw);
> > > > > +	if (ret == -EINVAL) {
> > > > > +		/*
> > > > > +		 * resource table only required for IPC.
> > > > > +		 * if not present, this is not necessarily an error;
> > > > > +		 * for example, loading r5 hello world application
> > > > > +		 * so simply inform user and keep going.
> > > > > +		 */
> > > > > +		dev_info(&rproc->dev, "no resource table found.\n");
> > > > > +		ret = 0;
> > > > > +	}
> > > > > +	return ret;
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * zynqmp_r5_rproc_kick() - kick a firmware if mbox is provided
> > > > > + * @rproc: r5 core's corresponding rproc structure
> > > > > + * @vqid: virtqueue ID
> > > > > + */
> > > > > +static void zynqmp_r5_rproc_kick(struct rproc *rproc, int vqid)
> > > > > +{
> > > > > +	struct zynqmp_r5_rproc *z_rproc = rproc->priv;
> > > > > +	struct device *dev = rproc->dev.parent;
> > > > > +	struct zynqmp_ipi_message *mb_msg = NULL;
> > > > > +	unsigned int skb_len;
> > > > > +	struct sk_buff *skb;
> > > > > +	int ret;
> > > > > +
> > > > > +	/* If vqid is negative, do not pass the vqid to
> > > > > +	 * mailbox as vqid is supposed to be 0 or positive.
> > > > > +	 * It also gives a way to just kick instead but
> > > > > +	 * not use the IPI buffer.
> > > > > +	 *
> > > > > +	 * For now use negative vqid to assume no message will be
> > > > > +	 * passed with IPI buffer, but just raise interrupt.
> > > > > +	 * This will be faster as it doesn't need to copy the
> > > > > +	 * message to the ZynqMP/Versal IPI message buffer.
> > > > > +	 */
> > > > > +	if (vqid >= 0) {
> > > > > +		skb_len = (unsigned int)(sizeof(vqid) + sizeof(mb_msg));
> > > > > +		skb = alloc_skb(skb_len, GFP_KERNEL);
> > > > > +		if (!skb)
> > > > > +			return;
> > > > > +
> > > > > +		mb_msg = (struct zynqmp_ipi_message *)skb_put(skb,
> > > > skb_len);
> > > > > +		mb_msg->len = sizeof(vqid);
> > > > > +		memcpy(mb_msg->data, &vqid, sizeof(vqid));
> > > > > +
> > > > > +		skb_queue_tail(&z_rproc->tx_mc_skbs, skb);
> > > > > +	}
> > > > > +
> > > > > +	ret = mbox_send_message(z_rproc->tx_chan, mb_msg);
> > > > > +	if (ret < 0) {
> > > > > +		dev_warn(dev, "Failed to kick remote.\n");
> > > > > +		if (mb_msg) {
> > > > > +			skb_dequeue_tail(&z_rproc->tx_mc_skbs);
> > > > > +			kfree_skb(skb);
> > > > > +		}
> > > > > +	}
> > > > > +}
> > > > > +
> > > >
> > > > I am done reviewing this set.
> > > >
> > > > Thanks,
> > > > Mathieu
> > > >
> > > > > +static struct rproc_ops zynqmp_r5_rproc_ops = {
> > > > > +	.start		= zynqmp_r5_rproc_start,
> > > > > +	.stop		= zynqmp_r5_rproc_stop,
> > > > > +	.load		= rproc_elf_load_segments,
> > > > > +	.parse_fw	= zynqmp_r5_parse_fw,
> > > > > +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > > > > +	.sanity_check	= rproc_elf_sanity_check,
> > > > > +	.get_boot_addr	= rproc_elf_get_boot_addr,
> > > > > +	.kick		= zynqmp_r5_rproc_kick,
> > > > > +};
> > > > > +
> > > > > +/**
> > > > > + * event_notified_idr_cb() - event notified idr callback
> > > > > + * @id: idr id
> > > > > + * @ptr: pointer to idr private data
> > > > > + * @data: data passed to idr_for_each callback
> > > > > + *
> > > > > + * Pass notification to remoteproc virtio
> > > > > + *
> > > > > + * Return: 0. having return is to satisfy the idr_for_each() function
> > > > > + *          pointer input argument requirement.
> > > > > + **/
> > > > > +static int event_notified_idr_cb(int id, void *ptr, void *data)
> > > > > +{
> > > > > +	struct rproc *rproc = data;
> > > > > +
> > > > > +	if (rproc_vq_interrupt(rproc, id) == IRQ_NONE)
> > > > > +		dev_dbg(&rproc->dev, "no message was found in idr with id:
> > > > %d\n", id);
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * handle_event_notified() - remoteproc notification work function
> > > > > + * @work: pointer to the work structure
> > > > > + *
> > > > > + * It checks each registered remoteproc notify IDs.
> > > > > + */
> > > > > +static void handle_event_notified(struct work_struct *work)
> > > > > +{
> > > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > > +	struct rproc *rproc;
> > > > > +
> > > > > +	z_rproc = container_of(work, struct zynqmp_r5_rproc, mbox_work);
> > > > > +	rproc = z_rproc->rproc;
> > > > > +
> > > > > +	/*
> > > > > +	 * This is to simply raise interrupt as ACK for remote.
> > > > > +	 *
> > > > > +	 * Xilinx mailbox, ATF and SoC specific IPI mapping will handle
> > > > > +	 * mapping to remote processor.
> > > > > +	 */
> > > > > +	if (mbox_send_message(z_rproc->rx_chan, NULL) < 0)
> > > > > +		dev_warn(rproc->dev.parent, "Failed to kick remote.\n");
> > > > > +
> > > > > +	/*
> > > > > +	 * We only use IPI for interrupt. The firmware side may or may
> > > > > +	 * not write the notifyid when it trigger IPI.
> > > > > +	 * And thus, we scan through all the registered notifyids.
> > > > > +	 */
> > > > > +	idr_for_each(&rproc->notifyids, event_notified_idr_cb, rproc);
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * zynqmp_r5_mb_rx_cb() - Receive channel mailbox callback
> > > > > + * @cl: mailbox client
> > > > > + * @msg: message pointer
> > > > > + *
> > > > > + * It will schedule the R5 notification work.
> > > > > + */
> > > > > +static void zynqmp_r5_mb_rx_cb(struct mbox_client *cl, void *msg)
> > > > > +{
> > > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > > +
> > > > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, rx_mc);
> > > > > +	/*
> > > > > +	 * Notification can be received with message payload.
> > > > > +	 * To handle this, check for message contents.
> > > > > +	 * If there are contents, copy to ipi message buffer payload location.
> > > > > +	 *
> > > > > +	 * If message is empty, then that means that was simple raising
> > > > > +	 * of interrupt. no payload to process.
> > > > > +	 *
> > > > > +	 * Note: enqueue work regardless of msg as kick may imply that
> > > > remote
> > > > > +	 * is waiting for ack.
> > > > > +	 */
> > > > > +	if (msg) {
> > > > > +		struct zynqmp_ipi_message *ipi_msg, *buf_msg;
> > > > > +		size_t len;
> > > > > +
> > > > > +		ipi_msg = (struct zynqmp_ipi_message *)msg;
> > > > > +		buf_msg = (struct zynqmp_ipi_message *)z_rproc->rx_mc_buf;
> > > > > +		len = (ipi_msg->len >= IPI_BUF_LEN_MAX) ?
> > > > > +		      IPI_BUF_LEN_MAX : ipi_msg->len;
> > > > > +		buf_msg->len = len;
> > > > > +		memcpy(buf_msg->data, ipi_msg->data, len);
> > > > > +	}
> > > > > +	schedule_work(&z_rproc->mbox_work);
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * zynqmp_r5_mb_tx_done() - Request has been sent to the remote
> > > > > + * @cl: mailbox client
> > > > > + * @msg: pointer to the message which has been sent
> > > > > + * @r: status of last TX - OK or error
> > > > > + *
> > > > > + * It will be called by the mailbox framework when the last TX has
> > done.
> > > > > + */
> > > > > +static void zynqmp_r5_mb_tx_done(struct mbox_client *cl, void *msg,
> > > int
> > > > r)
> > > > > +{
> > > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > > +	struct sk_buff *skb;
> > > > > +
> > > > > +	if (!msg)
> > > > > +		return;
> > > > > +	z_rproc = container_of(cl, struct zynqmp_r5_rproc, tx_mc);
> > > > > +	skb = skb_dequeue(&z_rproc->tx_mc_skbs);
> > > > > +	kfree_skb(skb);
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * zynqmp_r5_setup_mbox() - Setup mailboxes
> > > > > + *			    this is used for each individual R5 core
> > > > > + *
> > > > > + * @z_rproc: pointer to the ZynqMP R5 processor platform data
> > > > > + * @node: pointer of the device node
> > > > > + *
> > > > > + * Function to setup mailboxes to talk to RPU.
> > > > > + *
> > > > > + * Return: 0 for success, negative value for failure.
> > > > > + */
> > > > > +static int zynqmp_r5_setup_mbox(struct zynqmp_r5_rproc *z_rproc,
> > > > > +				struct device_node *node)
> > > > > +{
> > > > > +	struct mbox_client *mclient;
> > > > > +
> > > > > +	/* Setup TX mailbox channel client */
> > > > > +	mclient			= &z_rproc->tx_mc;
> > > > > +	mclient->dev		= z_rproc->dev;
> > > > > +	mclient->tx_done	= zynqmp_r5_mb_tx_done;
> > > > > +	mclient->tx_block	= false;
> > > > > +	mclient->knows_txdone	= false;
> > > > > +	mclient->rx_callback	= NULL;
> > > > > +
> > > > > +	/* Setup RX mailbox channel client */
> > > > > +	mclient			= &z_rproc->rx_mc;
> > > > > +	mclient->dev		= z_rproc->dev;
> > > > > +	mclient->tx_done	= NULL;
> > > > > +	mclient->tx_block	= false;
> > > > > +	mclient->knows_txdone	= false;
> > > > > +	mclient->rx_callback	= zynqmp_r5_mb_rx_cb;
> > > > > +
> > > > > +	INIT_WORK(&z_rproc->mbox_work, handle_event_notified);
> > > > > +
> > > > > +	/* Request TX and RX channels */
> > > > > +	z_rproc->tx_chan = mbox_request_channel_byname(&z_rproc-
> > > > >tx_mc, "tx");
> > > > > +	if (IS_ERR(z_rproc->tx_chan)) {
> > > > > +		dev_err(z_rproc->dev, "failed to request mbox tx channel.\n");
> > > > > +		return PTR_ERR(z_rproc->tx_chan);
> > > > > +	}
> > > > > +
> > > > > +	z_rproc->rx_chan = mbox_request_channel_byname(&z_rproc-
> > > > >rx_mc, "rx");
> > > > > +	if (IS_ERR(z_rproc->rx_chan)) {
> > > > > +		dev_err(z_rproc->dev, "failed to request mbox rx channel.\n");
> > > > > +		return PTR_ERR(z_rproc->rx_chan);
> > > > > +	}
> > > > > +	skb_queue_head_init(&z_rproc->tx_mc_skbs);
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +static void zynqmp_r5_cleanup_mbox(struct zynqmp_r5_rproc
> > *z_rproc)
> > > > > +{
> > > > > +	mbox_free_channel(z_rproc->tx_chan);
> > > > > +	mbox_free_channel(z_rproc->rx_chan);
> > > > > +}
> > > > > +
> > > > > +/**
> > > > > + * zynqmp_r5_probe() - Probes ZynqMP R5 processor device node
> > > > > + *		       this is called for each individual R5 core to
> > > > > + *		       set up mailbox, Xilinx platform manager unique
> > ID,
> > > > > + *		       add to rproc core
> > > > > + *
> > > > > + * @pdev: domain platform device for current R5 core
> > > > > + * @node: pointer of the device node for current R5 core
> > > > > + * @rpu_mode: mode to configure RPU, split or lockstep
> > > > > + *
> > > > > + * Return: 0 for success, negative value for failure.
> > > > > + */
> > > > > +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct
> > platform_device
> > > > *pdev,
> > > > > +					       struct device_node *node,
> > > > > +					       enum rpu_oper_mode
> > > > rpu_mode)
> > > > > +{
> > > > > +	int ret;
> > > > > +	struct device *dev = &pdev->dev;
> > > > > +	struct rproc *rproc_ptr;
> > > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > > +
> > > > > +	/* Allocate remoteproc instance */
> > > > > +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev),
> > > > &zynqmp_r5_rproc_ops,
> > > > > +				     NULL, sizeof(struct zynqmp_r5_rproc));
> > > > > +	if (!rproc_ptr) {
> > > > > +		ret = -ENOMEM;
> > > > > +		goto error;
> > > > > +	}
> > > > > +
> > > > > +	rproc_ptr->auto_boot = false;
> > > > > +	z_rproc = rproc_ptr->priv;
> > > > > +	z_rproc->rproc = rproc_ptr;
> > > > > +	z_rproc->dev = dev;
> > > > > +
> > > > > +	/* Set up DMA mask */
> > > > > +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> > > > > +	if (ret)
> > > > > +		goto error;
> > > > > +
> > > > > +	/* Get R5 power domain node */
> > > > > +	ret = of_property_read_u32(node, "power-domain", &z_rproc-
> > > > >pnode_id);
> > > > > +	if (ret)
> > > > > +		goto error;
> > > > > +
> > > > > +	ret = r5_set_mode(z_rproc, rpu_mode);
> > > > > +	if (ret)
> > > > > +		goto error;
> > > > > +
> > > > > +	if (of_property_read_bool(node, "mboxes")) {
> > > > > +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
> > > > > +		if (ret)
> > > > > +			goto error;
> > > > > +	}
> > > > > +
> > > > > +	/* Add R5 remoteproc */
> > > > > +	ret = devm_rproc_add(dev, rproc_ptr);
> > > > > +	if (ret) {
> > > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > > +		goto error;
> > > > > +	}
> > > > > +
> > > > > +	return z_rproc;
> > > > > +error:
> > > > > +	return ERR_PTR(ret);
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * zynqmp_r5_remoteproc_probe()
> > > > > + *
> > > > > + * @pdev: domain platform device for R5 cluster
> > > > > + *
> > > > > + * called when driver is probed, for each R5 core specified in DT,
> > > > > + * setup as needed to do remoteproc-related operations
> > > > > + *
> > > > > + * Return: 0 for success, negative value for failure.
> > > > > + */
> > > > > +static int zynqmp_r5_remoteproc_probe(struct platform_device *pdev)
> > > > > +{
> > > > > +	int ret, core_count;
> > > > > +	struct device *dev = &pdev->dev;
> > > > > +	struct device_node *nc;
> > > > > +	enum rpu_oper_mode rpu_mode = PM_RPU_MODE_LOCKSTEP;
> > > > > +	struct list_head *cluster; /* list to track each core's rproc */
> > > > > +	struct zynqmp_r5_rproc *z_rproc;
> > > > > +	struct platform_device *child_pdev;
> > > > > +	struct list_head *pos;
> > > > > +
> > > > > +	ret = of_property_read_u32(dev->of_node, "xlnx,cluster-mode",
> > > > &rpu_mode);
> > > > > +	if (ret < 0 || (rpu_mode != PM_RPU_MODE_LOCKSTEP &&
> > > > > +			rpu_mode != PM_RPU_MODE_SPLIT)) {
> > > > > +		dev_err(dev, "invalid cluster mode: ret %d mode %x\n",
> > > > > +			ret, rpu_mode);
> > > > > +		return ret;
> > > > > +	}
> > > > > +
> > > > > +	dev_dbg(dev, "RPU configuration: %s\n",
> > > > > +		rpu_mode == PM_RPU_MODE_LOCKSTEP ? "lockstep" :
> > > > "split");
> > > > > +
> > > > > +	/*
> > > > > +	 * if 2 RPUs provided but one is lockstep, then we have an
> > > > > +	 * invalid configuration.
> > > > > +	 */
> > > > > +
> > > > > +	core_count = of_get_available_child_count(dev->of_node);
> > > > > +	if ((rpu_mode == PM_RPU_MODE_LOCKSTEP && core_count != 1) ||
> > > > > +	    core_count > MAX_RPROCS)
> > > > > +		return -EINVAL;
> > > > > +
> > > > > +	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
> > > > > +	if (!cluster)
> > > > > +		return -ENOMEM;
> > > > > +	INIT_LIST_HEAD(cluster);
> > > > > +
> > > > > +	ret = devm_of_platform_populate(dev);
> > > > > +	if (ret) {
> > > > > +		dev_err(dev, "devm_of_platform_populate failed, ret =
> > > > %d\n",
> > > > > +			ret);
> > > > > +		return ret;
> > > > > +	}
> > > > > +
> > > > > +	/* probe each individual r5 core's remoteproc-related info */
> > > > > +	for_each_available_child_of_node(dev->of_node, nc) {
> > > > > +		child_pdev = of_find_device_by_node(nc);
> > > > > +		if (!child_pdev) {
> > > > > +			dev_err(dev, "could not get R5 core platform
> > > > device\n");
> > > > > +			ret = -ENODEV;
> > > > > +			goto out;
> > > > > +		}
> > > > > +
> > > > > +		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
> > > > > +		dev_dbg(dev, "%s to probe rpu %pOF\n",
> > > > > +			ret ? "Failed" : "Able",
> > > > > +			nc);
> > > > > +		if (IS_ERR(z_rproc)) {
> > > > > +			ret = PTR_ERR(z_rproc);
> > > > > +			goto out;
> > > > > +		}
> > > > > +		list_add_tail(&z_rproc->elem, cluster);
> > > > > +	}
> > > > > +	/* wire in so each core can be cleaned up at driver remove */
> > > > > +	platform_set_drvdata(pdev, cluster);
> > > > > +	return 0;
> > > > > +out:
> > > > > +	/*
> > > > > +	 * undo core0 upon any failures on core1 in split-mode
> > > > > +	 *
> > > > > +	 * in zynqmp_r5_probe z_rproc is set to null
> > > > > +	 * and ret to non-zero value if error
> > > > > +	 */
> > > > > +	list_for_each(pos, cluster) {
> > > > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > > +	}
> > > > > +	return ret;
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * zynqmp_r5_remoteproc_remove()
> > > > > + *
> > > > > + * @pdev: domain platform device for R5 cluster
> > > > > + *
> > > > > + * When the driver is unloaded, clean up the mailboxes for each
> > > > > + * remoteproc that was initially probed.
> > > > > + */
> > > > > +static int zynqmp_r5_remoteproc_remove(struct platform_device
> > *pdev)
> > > > > +{
> > > > > +	struct list_head *pos, *temp, *cluster = (struct list_head *)
> > > > > +						 platform_get_drvdata(pdev);
> > > > > +	struct zynqmp_r5_rproc *z_rproc = NULL;
> > > > > +
> > > > > +	list_for_each_safe(pos, temp, cluster) {
> > > > > +		z_rproc = list_entry(pos, struct zynqmp_r5_rproc, elem);
> > > > > +		zynqmp_r5_cleanup_mbox(z_rproc);
> > > > > +		list_del(pos);
> > > > > +	}
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +/* Match table for OF platform binding */
> > > > > +static const struct of_device_id zynqmp_r5_remoteproc_match[] = {
> > > > > +	{ .compatible = "xlnx,zynqmp-r5-remoteproc", },
> > > > > +	{ /* end of list */ },
> > > > > +};
> > > > > +MODULE_DEVICE_TABLE(of, zynqmp_r5_remoteproc_match);
> > > > > +
> > > > > +static struct platform_driver zynqmp_r5_remoteproc_driver = {
> > > > > +	.probe = zynqmp_r5_remoteproc_probe,
> > > > > +	.remove = zynqmp_r5_remoteproc_remove,
> > > > > +	.driver = {
> > > > > +		.name = "zynqmp_r5_remoteproc",
> > > > > +		.of_match_table = zynqmp_r5_remoteproc_match,
> > > > > +	},
> > > > > +};
> > > > > +module_platform_driver(zynqmp_r5_remoteproc_driver);
> > > > > +
> > > > > +MODULE_AUTHOR("Ben Levinsky <ben.levinsky@xilinx.com>");
> > > > > +MODULE_LICENSE("GPL v2");
> > > > > --
> > > > > 2.17.1
> > > > >

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 28+ messages in thread

end of thread, other threads:[~2021-02-17 22:00 UTC | newest]

Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-11  2:02 [PATCH v25 0/5] Provide basic driver to control Arm R5 co-processor found on Xilinx ZynqMP Ben Levinsky
2021-01-11  2:02 ` Ben Levinsky
2021-01-11  2:02 ` [PATCH v25 1/5] firmware: xilinx: Add ZynqMP firmware ioctl enums for RPU configuration Ben Levinsky
2021-01-11  2:02   ` Ben Levinsky
2021-01-11  2:02 ` [PATCH v25 2/5] firmware: xilinx: Add shutdown/wakeup APIs Ben Levinsky
2021-01-11  2:02   ` Ben Levinsky
2021-01-11  2:02 ` [PATCH v25 3/5] firmware: xilinx: Add RPU configuration APIs Ben Levinsky
2021-01-11  2:02   ` Ben Levinsky
2021-01-11  2:02 ` [PATCH v25 4/5] dt-bindings: remoteproc: Add documentation for ZynqMP R5 rproc bindings Ben Levinsky
2021-01-11  2:02   ` Ben Levinsky
2021-01-11  2:02 ` [PATCH v25 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver Ben Levinsky
2021-01-11  2:02   ` Ben Levinsky
2021-02-15 18:51   ` Mathieu Poirier
2021-02-15 18:51     ` Mathieu Poirier
2021-02-16 18:48   ` Mathieu Poirier
2021-02-16 18:48     ` Mathieu Poirier
2021-02-17 16:47     ` Ben Levinsky
2021-02-17 16:47       ` Ben Levinsky
2021-02-17 19:11       ` Ben Levinsky
2021-02-17 19:11         ` Ben Levinsky
2021-02-17 21:34         ` Ben Levinsky
2021-02-17 21:34           ` Ben Levinsky
2021-02-17 21:58           ` Mathieu Poirier
2021-02-17 21:58             ` Mathieu Poirier
2021-02-17 21:50         ` Mathieu Poirier
2021-02-17 21:50           ` Mathieu Poirier
2021-02-04 15:20 ` [PATCH v25 0/5] Provide basic driver to control Arm R5 co-processor found on Xilinx ZynqMP Ben Levinsky
2021-02-04 17:04   ` Mathieu Poirier

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.