Re: [PATCH v26 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver

[Ben Levinsky <BLEVINSK@xilinx.com>]

Hi Mathieu

-----Original Message-----
From: Mathieu Poirier <mathieu.poirier@linaro.org>
Date: Wednesday, March 17, 2021 at 9:27 AM
To: Ben Levinsky <BLEVINSK@xilinx.com>
Cc: "devicetree@vger.kernel.org" <devicetree@vger.kernel.org>, "linux-remoteproc@vger.kernel.org" <linux-remoteproc@vger.kernel.org>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>, Michal Simek <michals@xilinx.com>, "Ed T. Mooring" <emooring@xilinx.com>
Subject: Re: [PATCH v26 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver

    On Mon, Mar 15, 2021 at 09:32:40PM +0000, Ben Levinsky wrote:
    > 
    > 
    > -----Original Message-----
    > From: Mathieu Poirier <mathieu.poirier@linaro.org>
    > Date: Monday, March 15, 2021 at 10:37 AM
    > To: Ben Levinsky <BLEVINSK@xilinx.com>
    > Cc: "devicetree@vger.kernel.org" <devicetree@vger.kernel.org>, "linux-remoteproc@vger.kernel.org" <linux-remoteproc@vger.kernel.org>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>, Michal Simek <michals@xilinx.com>, "Ed T. Mooring" <emooring@xilinx.com>
    > Subject: Re: [PATCH v26 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
    > 
    >     On Thu, Mar 11, 2021 at 11:49:13PM +0000, Ben Levinsky wrote:
    >     > Hi Mathieu
    >     > 
    >     > -----Original Message-----
    >     > From: Mathieu Poirier <mathieu.poirier@linaro.org>
    >     > Date: Tuesday, March 9, 2021 at 8:53 AM
    >     > To: Ben Levinsky <BLEVINSK@xilinx.com>
    >     > Cc: "devicetree@vger.kernel.org" <devicetree@vger.kernel.org>, "linux-remoteproc@vger.kernel.org" <linux-remoteproc@vger.kernel.org>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>, Michal Simek <michals@xilinx.com>
    >     > Subject: Re: [PATCH v26 5/5] remoteproc: Add initial zynqmp R5 remoteproc driver
    >     > 
    >     >     [...]
    >     > 
    >     >     > +
    >     >     > +/**
    >     >     > + * zynqmp_r5_probe - Probes ZynqMP R5 processor device node
    >     >     > + *		       this is called for each individual R5 core to
    >     >     > + *		       set up mailbox, Xilinx platform manager unique ID,
    >     >     > + *		       add to rproc core
    >     >     > + *
    >     >     > + * @pdev: domain platform device for current R5 core
    >     >     > + * @node: pointer of the device node for current R5 core
    >     >     > + * @rpu_mode: mode to configure RPU, split or lockstep
    >     >     > + *
    >     >     > + * Return: 0 for success, negative value for failure.
    >     >     > + */
    >     >     > +static struct zynqmp_r5_rproc *zynqmp_r5_probe(struct platform_device *pdev,
    >     >     > +					       struct device_node *node,
    >     >     > +					       enum rpu_oper_mode rpu_mode)
    >     >     > +{
    >     >     > +	int ret, num_banks;
    >     >     > +	struct device *dev = &pdev->dev;
    >     >     > +	struct rproc *rproc_ptr;
    >     >     > +	struct zynqmp_r5_rproc *z_rproc;
    >     >     > +	struct device_node *r5_node;
    >     >     > +
    >     >     > +	/* Allocate remoteproc instance */
    >     >     > +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
    >     >     > +				     NULL, sizeof(struct zynqmp_r5_rproc));
    >     >     > +	if (!rproc_ptr) {
    >     >     > +		ret = -ENOMEM;
    >     >     > +		goto error;
    >     >     > +	}
    >     >     > +
    >     >     > +	rproc_ptr->auto_boot = false;
    >     >     > +	z_rproc = rproc_ptr->priv;
    >     >     > +	z_rproc->rproc = rproc_ptr;
    >     >     > +	r5_node = z_rproc->rproc->dev.parent->of_node;
    >     >     > +
    >     >     > +	/* Set up DMA mask */
    >     >     > +	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
    >     >     > +	if (ret)
    >     >     > +		goto error;
    >     >     > +
    >     >     > +	/* Get R5 power domain node */
    >     >     > +	ret = of_property_read_u32(node, "power-domain", &z_rproc->pnode_id);
    >     >     > +	if (ret)
    >     >     > +		goto error;
    >     >     > +
    >     >     > +	ret = r5_set_mode(z_rproc, rpu_mode);
    >     >     > +	if (ret)
    >     >     > +		goto error;
    >     >     > +
    >     >     > +	if (of_property_read_bool(node, "mboxes")) {
    >     >     > +		ret = zynqmp_r5_setup_mbox(z_rproc, node);
    >     >     > +		if (ret)
    >     >     > +			goto error;
    >     >     > +	}
    >     >     > +
    >     >     > +	/* go through TCM banks for r5 node */
    >     >     > +	num_banks = of_count_phandle_with_args(r5_node, BANK_LIST_PROP, NULL);
    >     >     > +	if (num_banks <= 0) {
    >     >     > +		dev_err(dev, "need to specify TCM banks\n");
    >     >     > +		ret = -EINVAL;
    >     >     > +		goto error;
    >     >     > +	}
    >     >     > +
    >     >     > +	if (num_banks > NUM_SRAMS) {
    >     >     > +		dev_err(dev, "max number of srams is %d. given: %d \r\n",
    >     >     > +			NUM_SRAMS, num_banks);
    >     >     > +		ret = -EINVAL;
    >     >     > +		goto error;
    >     >     > +	}
    >     >     > +
    >     >     > +	/* construct collection of srams used by the current R5 core */
    >     >     > +	for (; num_banks; num_banks--) {
    >     >     > +		struct resource rsc;
    >     >     > +		struct device_node *dt_node;
    >     >     > +		resource_size_t size;
    >     >     > +		int i;
    >     >     > +
    >     >     > +		dt_node = of_parse_phandle(r5_node, BANK_LIST_PROP, i);
    >     > 
    >     >     Variable @i is not initialised but it is used as an index to retrieve a handle
    >     >     to the sram banks.  That code _should_ have failed frequently or at least have
    >     >     yielded abnormal results often enough to be noticed.  Why wasn't it the case?
    >     > 
    >     >     I will stop here for the moment.
    >     > 
    >     > [Ben]
    >     > Yes this should be initialized. The reason this got through is that as i defaults to 0 and the 0th bank housed the required data. the case where SRAMS that can be written to, 0xFFE20000 in this case of split mode and on R5-0, was not caught.
    >     > 
    > 
    >     Here @i is a variable allocated on the stack and as such it is garanteed to be
    >     garbage on initialisation - it will do anything but default to 0.
    > 
    > Ok.
    > 
    >     > Instead of i I will use 
    >     > 
    >     >                 sram_node = of_parse_phandle(node, BANK_LIST_PROP,              
    >     >                                              num_banks - 1); 
    > 
    >     Do you have to start with the last bank?  If memory serves me well it isn't the
    >     case in the previous revisions.  Why not go back to the implementation you had
    >     in V25?  
    > 
    > Makes sense. Will revert as suggested.

    For your next revision, go back to V25 and fix only what I commented on.  I
    can't remember but you may also have to fix the put_device() problem we've been
    discussing. 

Ok will do. Few things:

Update per minor comments

1.
For SRAM parsing and usage, I would like to check if in v27 the SRAM collection and validation is done from a single function called by zynqmp_r5_probe() for each core with the loop cleaned up to match v25. Then if this SRAM information is only used in parse_fw and prepare/unprepared is ok?

This way in rproc core's parse_fw, the only thing that is occurring is the creation of rproc carveouts. As prepare occurs before parse_fw, the Xilinx-specific initialization occurs before parse_fw anyway.

I am envisioning this as:
1. zynqmp_r5_remoteproc_probe
2. zynqmp_r5_probe
3. zynqmp_r5_tcm_init 		- collect and validate SRAM information for each core
4. zynqmp_r5_prepare 		- call Xilinx request_node as initially suggested
5. zynqmp_r5_parse_fw 	- construct carveouts from DDR and SRAMs
6. zynqmp_r5_rproc_start
7. zynqmp_r5_unprepare	- unprepared does the release as initially suggested

2.
For the struct device pointer, in v25 you commented to remove it being stored:


	> +	/* Allocate remoteproc instance */
	> +	rproc_ptr = devm_rproc_alloc(dev, dev_name(dev), &zynqmp_r5_rproc_ops,
	> +				     NULL, sizeof(struct zynqmp_r5_rproc));
	> +	if (!rproc_ptr) {
	> +		ret = -ENOMEM;
	> +		goto error;
	> +	}
	> +
	> +	rproc_ptr->auto_boot = false;
	> +	z_rproc = rproc_ptr->priv;
	> +	z_rproc->rproc = rproc_ptr;
	> +	z_rproc->dev = dev;

	Here z_rproc->dev is the same as z_rproc->rproc->dev->parent and as far as I can
	tell z_rproc->rproc is available where z_rproc->dev is used.  As such
	zynqmpq_r5_rproc::dev can likely be removed.

will remove the ptr.

For the management of the device pointer and device tree node how is the following:
zynqmp_r5_remoteproc_probe
...
/* probe each individual r5 core's remoteproc-related info */
for_each_available_child_of_node(dev->of_node, nc) {
		child_pdev = of_find_device_by_node(nc);
		if (!child_pdev) {
			dev_err(dev, "could not get R5 core platform device\n");
			ret = -ENODEV;
			goto put_node;
		}

		z_rproc = zynqmp_r5_probe(child_pdev, nc, rpu_mode);
		dev_dbg(dev, "%s to probe rpu %pOF\n",
			ret ? "Failed" : "Able", nc);
		if (IS_ERR(z_rproc)) {
			put_device(dev);
			ret = PTR_ERR(z_rproc);
			goto put_device;
		}
		list_add_tail(&z_rproc->elem, cluster);
	}

put_device:
	put_device(dev);
put_node:
	of_node_put(nc);
out:
	list_for_each(pos, cluster) {
		....


Thanks
Ben

    > 
    >