linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v4 00/17] remoteproc: add fixed memory region support
@ 2018-07-27 13:14 Loic Pallardy
  2018-07-27 13:14 ` [PATCH v4 01/17] remoteproc: configure IOMMU only if device address requested Loic Pallardy
                   ` (18 more replies)
  0 siblings, 19 replies; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

The aim of the series is to implement carveout memory management as
discussed during OpenAMP weekly call and defined in proposed document [1]

This first series focus only on adding support of the different types of
carveout memories (dynamic, fixed, platform driver depend...).
64bit resource table will be addressed in a next series.
Cent
[1]: http://openamp.github.io/docs/mca/coprocessor-memory-definition-v6.pdf

---
Changes since V3:
- Rebase on top of Bjorn remoteproc next git (3d065621c921)
- Apply on top of "remoteproc: replace "%p" with "%pK"" [2]
- Lot of changes to centralize carveout allocation before rproc_start()
- Remove prepare/unprepare ops and rely on parse_fw instead of
- Add possibility to assign reserved memory region to vdev device
- Adapt TI da8xx and keystone remoteproc driver

As V3 series, V4 series takes some assumptions for carveout names
associated to vdev:
- For vring: "vdev%xvring%x" with vdev index from resource table and vring index
  in vdev.
- For vdev buffer: "vdev%xbuffer" with vdev index from resource table

[2] https://patchwork.kernel.org/patch/10511599

Changes since V2:
Reshuffle the series to: 
- Take into account Bjorn's comments.
- Add patch to check consistency between carveout resource request and IOMMU
  support.
- Introduce platform specific prepare and unprepare ops to enable HW like
  clock, bus, regulator, memory region... before loading co-processor firmware.
- Rely on memory carveout management for all remoteproc memory allocations.
- Lookup pre-registered carveout by name first.
- Create a subdevice for each vdev declared in firmware resource table that
  will be used by virtio based driver to retrieve specific memory pool.

This series takes some assumptions for carveout names associated to vdev:
- For vring: "vdev%xvring%x" with vdev index from resource table and vring index
  in vdev.
- For vdev buffer: "vdev%xbuffer" with vdev index from resource table

This will be changed in the future, adding names field in vdev resource in
next resource table version.


Changes since V1:
- Minor corrections on first 7 patches (error management)
- Add "memory device" support on the top of first 7 patches.
  Goal is to answer use case reported during OpenAMP weekly discussion:
  - "Be able to specify memory region for vring and buffer allocation, even
    if no specific request defined in firmware resource table."
  Patches offer the capability to create a "memory device" associated to a
  carveout with a dedicated DMA memory pool. Different resource handlers are
  modified to look-up for specific carveout by name. If match found and associated
  "memory device" present, device is used instead of rproc platform device for
  allocation.


Loic Pallardy (17):
  remoteproc: configure IOMMU only if device address requested
  remoteproc: add rproc_va_to_pa function
  remoteproc: add release ops in rproc_mem_entry struct
  remoteproc: add name in rproc_mem_entry struct
  remoteproc: add helper function to allocate and init rproc_mem_entry
    struct
  remoteproc: introduce rproc_add_carveout function
  remoteproc: introduce rproc_find_carveout_by_name function
  remoteproc: add alloc ops in rproc_mem_entry struct
  remoteproc: add helper function to allocate rproc_mem_entry from
    reserved memory
  remoteproc: add helper function to check carveout device address
  remoteproc: modify rproc_handle_carveout to support pre-registered
    region
  remoteproc: modify vring allocation to rely on centralized carveout
    allocator
  remoteproc: create vdev subdevice with specific dma memory pool
  remoteproc: keystone: declare reserved memory region for vdev device
  remoteproc: da8xx: declare reserved memory region for vdev device
  remoteproc: st: add reserved memory support
  rpmsg: virtio: allocate buffer from parent

 drivers/remoteproc/da8xx_remoteproc.c    |  38 ++
 drivers/remoteproc/keystone_remoteproc.c |  53 ++-
 drivers/remoteproc/remoteproc_core.c     | 572 +++++++++++++++++++++++++------
 drivers/remoteproc/remoteproc_debugfs.c  |   1 +
 drivers/remoteproc/remoteproc_internal.h |   3 +
 drivers/remoteproc/remoteproc_virtio.c   |  56 ++-
 drivers/remoteproc/st_remoteproc.c       |  96 +++++-
 drivers/rpmsg/virtio_rpmsg_bus.c         |   6 +-
 include/linux/remoteproc.h               |  35 +-
 9 files changed, 728 insertions(+), 132 deletions(-)

-- 
1.9.1


^ permalink raw reply	[flat|nested] 75+ messages in thread

* [PATCH v4 01/17] remoteproc: configure IOMMU only if device address requested
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-10-23 17:25   ` Suman Anna
  2018-07-27 13:14 ` [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function Loic Pallardy
                   ` (17 subsequent siblings)
  18 siblings, 1 reply; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

If there is no IOMMU associate to remote processor device,
remoteproc_core won't be able to satisfy device address requested
in firmware resource table.
Return an error as configuration won't be coherent.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/remoteproc/remoteproc_core.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index 4cd1a8e..437fabf 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -657,7 +657,15 @@ static int rproc_handle_carveout(struct rproc *rproc,
 	 * to use the iommu-based DMA API: we expect 'dma' to contain the
 	 * physical address in this case.
 	 */
-	if (rproc->domain) {
+
+	if (rsc->da != FW_RSC_ADDR_ANY && !rproc->domain) {
+		dev_err(dev->parent,
+			"Bad carveout rsc configuration\n");
+		ret = -ENOMEM;
+		goto dma_free;
+	}
+
+	if (rsc->da != FW_RSC_ADDR_ANY && rproc->domain) {
 		mapping = kzalloc(sizeof(*mapping), GFP_KERNEL);
 		if (!mapping) {
 			ret = -ENOMEM;
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
  2018-07-27 13:14 ` [PATCH v4 01/17] remoteproc: configure IOMMU only if device address requested Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-10-23 16:50   ` Suman Anna
  2018-07-27 13:14 ` [PATCH v4 03/17] remoteproc: add release ops in rproc_mem_entry struct Loic Pallardy
                   ` (16 subsequent siblings)
  18 siblings, 1 reply; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

This new function translates CPU virtual address in
CPU physical one according to virtual address location.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>
---
 drivers/remoteproc/remoteproc_core.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index 437fabf..8e5fe1e 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -140,6 +140,22 @@ static void rproc_disable_iommu(struct rproc *rproc)
 	iommu_domain_free(domain);
 }
 
+static phys_addr_t rproc_va_to_pa(void *cpu_addr)
+{
+	/*
+	 * Return physical address according to virtual address location
+	 * - in vmalloc: if region ioremapped or defined as dma_alloc_coherent
+	 * - in kernel: if region allocated in generic dma memory pool
+	 */
+	if (is_vmalloc_addr(cpu_addr)) {
+		return page_to_phys(vmalloc_to_page(cpu_addr)) +
+				    offset_in_page(cpu_addr);
+	}
+
+	WARN_ON(!virt_addr_valid(cpu_addr));
+	return virt_to_phys(cpu_addr);
+}
+
 /**
  * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc address
  * @rproc: handle of a remote processor
@@ -711,7 +727,7 @@ static int rproc_handle_carveout(struct rproc *rproc,
 	 * In this case, the device address and the physical address
 	 * are the same.
 	 */
-	rsc->pa = dma;
+	rsc->pa = (u32)rproc_va_to_pa(va);
 
 	carveout->va = va;
 	carveout->len = rsc->len;
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 03/17] remoteproc: add release ops in rproc_mem_entry struct
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
  2018-07-27 13:14 ` [PATCH v4 01/17] remoteproc: configure IOMMU only if device address requested Loic Pallardy
  2018-07-27 13:14 ` [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-10-23 16:53   ` Suman Anna
  2018-07-27 13:14 ` [PATCH v4 04/17] remoteproc: add name " Loic Pallardy
                   ` (15 subsequent siblings)
  18 siblings, 1 reply; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

Memory entry could be allocated in different ways (ioremap,
dma_alloc_coherent, internal RAM allocator...).
This patch introduces a release ops in rproc_mem_entry structure
to associate dedicated release mechanism to each memory entry descriptor
in order to keep remoteproc core generic.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>
---
 drivers/remoteproc/remoteproc_core.c | 23 +++++++++++++++++++++--
 include/linux/remoteproc.h           |  5 ++++-
 2 files changed, 25 insertions(+), 3 deletions(-)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index 8e5fe1e..24fe11f 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -600,6 +600,24 @@ static int rproc_handle_devmem(struct rproc *rproc, struct fw_rsc_devmem *rsc,
 }
 
 /**
+ * rproc_release_carveout() - release acquired carveout
+ * @rproc: rproc handle
+ * @mem: the memory entry to release
+ *
+ * This function releases specified memory entry @mem allocated via
+ * dma_alloc_coherent() function by @rproc.
+ */
+static int rproc_release_carveout(struct rproc *rproc,
+				  struct rproc_mem_entry *mem)
+{
+	struct device *dev = &rproc->dev;
+
+	/* clean up carveout allocations */
+	dma_free_coherent(dev->parent, mem->len, mem->va, mem->dma);
+	return 0;
+}
+
+/**
  * rproc_handle_carveout() - handle phys contig memory allocation requests
  * @rproc: rproc handle
  * @rsc: the resource entry
@@ -733,6 +751,7 @@ static int rproc_handle_carveout(struct rproc *rproc,
 	carveout->len = rsc->len;
 	carveout->dma = dma;
 	carveout->da = rsc->da;
+	carveout->release = rproc_release_carveout;
 
 	list_add_tail(&carveout->node, &rproc->carveouts);
 
@@ -920,8 +939,8 @@ static void rproc_resource_cleanup(struct rproc *rproc)
 
 	/* clean up carveout allocations */
 	list_for_each_entry_safe(entry, tmp, &rproc->carveouts, node) {
-		dma_free_coherent(dev->parent, entry->len, entry->va,
-				  entry->dma);
+		if (entry->release)
+			entry->release(rproc, entry);
 		list_del(&entry->node);
 		kfree(entry);
 	}
diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
index e3c5d85..bef2e83 100644
--- a/include/linux/remoteproc.h
+++ b/include/linux/remoteproc.h
@@ -305,12 +305,15 @@ struct fw_rsc_vdev {
 	struct fw_rsc_vdev_vring vring[0];
 } __packed;
 
+struct rproc;
+
 /**
  * struct rproc_mem_entry - memory entry descriptor
  * @va:	virtual address
  * @dma: dma address
  * @len: length, in bytes
  * @da: device address
+ * @release: release associated memory
  * @priv: associated data
  * @node: list node
  */
@@ -321,9 +324,9 @@ struct rproc_mem_entry {
 	u32 da;
 	void *priv;
 	struct list_head node;
+	int (*release)(struct rproc *rproc, struct rproc_mem_entry *mem);
 };
 
-struct rproc;
 struct firmware;
 
 /**
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 04/17] remoteproc: add name in rproc_mem_entry struct
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (2 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 03/17] remoteproc: add release ops in rproc_mem_entry struct Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-10-23 17:06   ` Suman Anna
  2018-07-27 13:14 ` [PATCH v4 05/17] remoteproc: add helper function to allocate and init " Loic Pallardy
                   ` (14 subsequent siblings)
  18 siblings, 1 reply; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

Add name field in struct rproc_mem_entry.
This new field will be used to match memory area
requested in resource table with pre-registered carveout.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>
---
 drivers/remoteproc/remoteproc_core.c    | 1 +
 drivers/remoteproc/remoteproc_debugfs.c | 1 +
 include/linux/remoteproc.h              | 2 ++
 3 files changed, 4 insertions(+)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index 24fe11f..d7e3138 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -752,6 +752,7 @@ static int rproc_handle_carveout(struct rproc *rproc,
 	carveout->dma = dma;
 	carveout->da = rsc->da;
 	carveout->release = rproc_release_carveout;
+	strlcpy(carveout->name, rsc->name, sizeof(carveout->name));
 
 	list_add_tail(&carveout->node, &rproc->carveouts);
 
diff --git a/drivers/remoteproc/remoteproc_debugfs.c b/drivers/remoteproc/remoteproc_debugfs.c
index a5c29f2..e90135c 100644
--- a/drivers/remoteproc/remoteproc_debugfs.c
+++ b/drivers/remoteproc/remoteproc_debugfs.c
@@ -260,6 +260,7 @@ static int rproc_carveouts_show(struct seq_file *seq, void *p)
 
 	list_for_each_entry(carveout, &rproc->carveouts, node) {
 		seq_puts(seq, "Carveout memory entry:\n");
+		seq_printf(seq, "\tName: %s\n", carveout->name);
 		seq_printf(seq, "\tVirtual address: %pK\n", carveout->va);
 		seq_printf(seq, "\tDMA address: %pad\n", &carveout->dma);
 		seq_printf(seq, "\tDevice address: 0x%x\n", carveout->da);
diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
index bef2e83..0e21098 100644
--- a/include/linux/remoteproc.h
+++ b/include/linux/remoteproc.h
@@ -315,6 +315,7 @@ struct fw_rsc_vdev {
  * @da: device address
  * @release: release associated memory
  * @priv: associated data
+ * @name: associated memory region name (optional)
  * @node: list node
  */
 struct rproc_mem_entry {
@@ -323,6 +324,7 @@ struct rproc_mem_entry {
 	int len;
 	u32 da;
 	void *priv;
+	char name[32];
 	struct list_head node;
 	int (*release)(struct rproc *rproc, struct rproc_mem_entry *mem);
 };
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 05/17] remoteproc: add helper function to allocate and init rproc_mem_entry struct
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (3 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 04/17] remoteproc: add name " Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-10-23 19:24   ` Suman Anna
  2018-07-27 13:14 ` [PATCH v4 06/17] remoteproc: introduce rproc_add_carveout function Loic Pallardy
                   ` (13 subsequent siblings)
  18 siblings, 1 reply; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

This patch introduces rproc_mem_entry_init helper function to
simplify rproc_mem_entry structure allocation and filling by
client.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/remoteproc/remoteproc_core.c | 65 +++++++++++++++++++++++++++---------
 include/linux/remoteproc.h           |  6 ++++
 2 files changed, 55 insertions(+), 16 deletions(-)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index d7e3138..b76760e 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -639,7 +639,7 @@ static int rproc_handle_carveout(struct rproc *rproc,
 				 struct fw_rsc_carveout *rsc,
 				 int offset, int avail)
 {
-	struct rproc_mem_entry *carveout, *mapping;
+	struct rproc_mem_entry *carveout, *mapping = NULL;
 	struct device *dev = &rproc->dev;
 	dma_addr_t dma;
 	void *va;
@@ -659,16 +659,11 @@ static int rproc_handle_carveout(struct rproc *rproc,
 	dev_dbg(dev, "carveout rsc: name: %s, da 0x%x, pa 0x%x, len 0x%x, flags 0x%x\n",
 		rsc->name, rsc->da, rsc->pa, rsc->len, rsc->flags);
 
-	carveout = kzalloc(sizeof(*carveout), GFP_KERNEL);
-	if (!carveout)
-		return -ENOMEM;
-
 	va = dma_alloc_coherent(dev->parent, rsc->len, &dma, GFP_KERNEL);
 	if (!va) {
 		dev_err(dev->parent,
 			"failed to allocate dma memory: len 0x%x\n", rsc->len);
-		ret = -ENOMEM;
-		goto free_carv;
+		return -ENOMEM;
 	}
 
 	dev_dbg(dev, "carveout va %pK, dma %pad, len 0x%x\n",
@@ -747,27 +742,65 @@ static int rproc_handle_carveout(struct rproc *rproc,
 	 */
 	rsc->pa = (u32)rproc_va_to_pa(va);
 
-	carveout->va = va;
-	carveout->len = rsc->len;
-	carveout->dma = dma;
-	carveout->da = rsc->da;
-	carveout->release = rproc_release_carveout;
-	strlcpy(carveout->name, rsc->name, sizeof(carveout->name));
+	carveout = rproc_mem_entry_init(dev, va, dma, rsc->len, rsc->da,
+					rproc_release_carveout, rsc->name);
+	if (!carveout)
+		goto free_carv;
 
 	list_add_tail(&carveout->node, &rproc->carveouts);
 
 	return 0;
 
+free_carv:
+	kfree(carveout);
 free_mapping:
 	kfree(mapping);
 dma_free:
 	dma_free_coherent(dev->parent, rsc->len, va, dma);
-free_carv:
-	kfree(carveout);
 	return ret;
 }
 
-/*
+/**
+ * rproc_mem_entry_init() - allocate and initialize rproc_mem_entry struct
+ * @dev: pointer on device struct
+ * @va: virtual address
+ * @dma: dma address
+ * @len: memory carveout length
+ * @da: device address
+ * @release: memory carveout function
+ * @name: carveout name
+ *
+ * This function allocates a rproc_mem_entry struct and fill it with parameters
+ * provided by client.
+ */
+struct rproc_mem_entry *
+rproc_mem_entry_init(struct device *dev,
+		     void *va, dma_addr_t dma, int len, u32 da,
+		     int (*release)(struct rproc *, struct rproc_mem_entry *),
+		     const char *name, ...)
+{
+	struct rproc_mem_entry *mem;
+	va_list args;
+
+	mem = kzalloc(sizeof(*mem), GFP_KERNEL);
+	if (!mem)
+		return mem;
+
+	mem->va = va;
+	mem->dma = dma;
+	mem->da = da;
+	mem->len = len;
+	mem->release = release;
+
+	va_start(args, name);
+	vsnprintf(mem->name, sizeof(mem->name), name, args);
+	va_end(args);
+
+	return mem;
+}
+EXPORT_SYMBOL(rproc_mem_entry_init);
+
+/**
  * A lookup table for resource handlers. The indices are defined in
  * enum fw_resource_type.
  */
diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
index 0e21098..4bc961f 100644
--- a/include/linux/remoteproc.h
+++ b/include/linux/remoteproc.h
@@ -558,6 +558,12 @@ struct rproc *rproc_alloc(struct device *dev, const char *name,
 int rproc_del(struct rproc *rproc);
 void rproc_free(struct rproc *rproc);
 
+struct rproc_mem_entry *
+rproc_mem_entry_init(struct device *dev,
+		     void *va, dma_addr_t dma, int len, u32 da,
+		     int (*release)(struct rproc *, struct rproc_mem_entry *),
+		     const char *name, ...);
+
 int rproc_boot(struct rproc *rproc);
 void rproc_shutdown(struct rproc *rproc);
 void rproc_report_crash(struct rproc *rproc, enum rproc_crash_type type);
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 06/17] remoteproc: introduce rproc_add_carveout function
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (4 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 05/17] remoteproc: add helper function to allocate and init " Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-10-23 17:05   ` Suman Anna
  2018-07-27 13:14 ` [PATCH v4 07/17] remoteproc: introduce rproc_find_carveout_by_name function Loic Pallardy
                   ` (12 subsequent siblings)
  18 siblings, 1 reply; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

This patch introduces a new API to allow platform driver to register
platform specific carveout regions.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>
---
 drivers/remoteproc/remoteproc_core.c | 16 +++++++++++++++-
 include/linux/remoteproc.h           |  2 ++
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index b76760e..fe6c4e4 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -747,7 +747,7 @@ static int rproc_handle_carveout(struct rproc *rproc,
 	if (!carveout)
 		goto free_carv;
 
-	list_add_tail(&carveout->node, &rproc->carveouts);
+	rproc_add_carveout(rproc, carveout);
 
 	return 0;
 
@@ -761,6 +761,20 @@ static int rproc_handle_carveout(struct rproc *rproc,
 }
 
 /**
+ * rproc_add_carveout() - register an allocated carveout region
+ * @rproc: rproc handle
+ * @mem: memory entry to register
+ *
+ * This function registers specified memory entry in @rproc carveouts list.
+ * Specified carveout should have been allocated before registering.
+ */
+void rproc_add_carveout(struct rproc *rproc, struct rproc_mem_entry *mem)
+{
+	list_add_tail(&mem->node, &rproc->carveouts);
+}
+EXPORT_SYMBOL(rproc_add_carveout);
+
+/**
  * rproc_mem_entry_init() - allocate and initialize rproc_mem_entry struct
  * @dev: pointer on device struct
  * @va: virtual address
diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
index 4bc961f..55f30fc 100644
--- a/include/linux/remoteproc.h
+++ b/include/linux/remoteproc.h
@@ -558,6 +558,8 @@ struct rproc *rproc_alloc(struct device *dev, const char *name,
 int rproc_del(struct rproc *rproc);
 void rproc_free(struct rproc *rproc);
 
+void rproc_add_carveout(struct rproc *rproc, struct rproc_mem_entry *mem);
+
 struct rproc_mem_entry *
 rproc_mem_entry_init(struct device *dev,
 		     void *va, dma_addr_t dma, int len, u32 da,
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 07/17] remoteproc: introduce rproc_find_carveout_by_name function
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (5 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 06/17] remoteproc: introduce rproc_add_carveout function Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-10-23 19:28   ` Suman Anna
  2018-07-27 13:14 ` [PATCH v4 08/17] remoteproc: add alloc ops in rproc_mem_entry struct Loic Pallardy
                   ` (11 subsequent siblings)
  18 siblings, 1 reply; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

This patch provides a new function to find a carveout according
to a name.
If match found, this function returns a pointer on the corresponding
carveout (rproc_mem_entry structure).

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/remoteproc/remoteproc_core.c | 42 ++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index fe6c4e4..77b39ba 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -217,6 +217,48 @@ void *rproc_da_to_va(struct rproc *rproc, u64 da, int len)
 }
 EXPORT_SYMBOL(rproc_da_to_va);
 
+/**
+ * rproc_find_carveout_by_name() - lookup the carveout region by a name
+ * @rproc: handle of a remote processor
+ * @name,..: carveout name to find (standard printf format)
+ *
+ * Platform driver has the capability to register some pre-allacoted carveout
+ * (physically contiguous memory regions) before rproc firmware loading and
+ * associated resource table analysis. These regions may be dedicated memory
+ * regions internal to the coprocessor or specified DDR region with specific
+ * attributes
+ *
+ * This function is a helper function with which we can go over the
+ * allocated carveouts and return associated region characteristics like
+ * coprocessor address, length or processor virtual address.
+ *
+ * Return: a valid pointer on carveout entry on success or NULL on failure.
+ */
+struct rproc_mem_entry *
+rproc_find_carveout_by_name(struct rproc *rproc, const char *name, ...)
+{
+	va_list args;
+	char _name[32];
+	struct rproc_mem_entry *carveout, *mem = NULL;
+
+	if (!name)
+		return NULL;
+
+	va_start(args, name);
+	vsnprintf(_name, sizeof(_name), name, args);
+	va_end(args);
+
+	list_for_each_entry(carveout, &rproc->carveouts, node) {
+		/* Compare carveout and requested names */
+		if (!strcmp(carveout->name, _name)) {
+			mem = carveout;
+			break;
+		}
+	}
+
+	return mem;
+}
+
 int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
 {
 	struct rproc *rproc = rvdev->rproc;
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 08/17] remoteproc: add alloc ops in rproc_mem_entry struct
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (6 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 07/17] remoteproc: introduce rproc_find_carveout_by_name function Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-10-23 21:20   ` Suman Anna
  2018-07-27 13:14 ` [PATCH v4 09/17] remoteproc: add helper function to allocate rproc_mem_entry from reserved memory Loic Pallardy
                   ` (10 subsequent siblings)
  18 siblings, 1 reply; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

Memory entry could be allocated in different ways (ioremap,
dma_alloc_coherent, internal RAM allocator...).
This patch introduces an alloc ops in rproc_mem_entry structure
to associate dedicated allocation mechanism to each memory entry
descriptor in order to do remote core agnostic from memory allocators.

The introduction of this ops allows to perform allocation of all registered
carveout at the same time, just before calling rproc_start().
It simplifies and makes uniform carveout management whatever origin.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/remoteproc/remoteproc_core.c | 261 ++++++++++++++++++++++-------------
 include/linux/remoteproc.h           |   7 +
 2 files changed, 175 insertions(+), 93 deletions(-)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index 77b39ba..2c51549 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -642,74 +642,31 @@ static int rproc_handle_devmem(struct rproc *rproc, struct fw_rsc_devmem *rsc,
 }
 
 /**
- * rproc_release_carveout() - release acquired carveout
+ * rproc_alloc_carveout() - allocated specified carveout
  * @rproc: rproc handle
- * @mem: the memory entry to release
- *
- * This function releases specified memory entry @mem allocated via
- * dma_alloc_coherent() function by @rproc.
- */
-static int rproc_release_carveout(struct rproc *rproc,
-				  struct rproc_mem_entry *mem)
-{
-	struct device *dev = &rproc->dev;
-
-	/* clean up carveout allocations */
-	dma_free_coherent(dev->parent, mem->len, mem->va, mem->dma);
-	return 0;
-}
-
-/**
- * rproc_handle_carveout() - handle phys contig memory allocation requests
- * @rproc: rproc handle
- * @rsc: the resource entry
- * @avail: size of available data (for image validation)
- *
- * This function will handle firmware requests for allocation of physically
- * contiguous memory regions.
- *
- * These request entries should come first in the firmware's resource table,
- * as other firmware entries might request placing other data objects inside
- * these memory regions (e.g. data/code segments, trace resource entries, ...).
+ * @mem: the memory entry to allocate
  *
- * Allocating memory this way helps utilizing the reserved physical memory
- * (e.g. CMA) more efficiently, and also minimizes the number of TLB entries
- * needed to map it (in case @rproc is using an IOMMU). Reducing the TLB
- * pressure is important; it may have a substantial impact on performance.
+ * This function allocate specified memory entry @mem using
+ * dma_alloc_coherent() as default allocator
  */
-static int rproc_handle_carveout(struct rproc *rproc,
-				 struct fw_rsc_carveout *rsc,
-				 int offset, int avail)
+static int rproc_alloc_carveout(struct rproc *rproc,
+				struct rproc_mem_entry *mem)
 {
-	struct rproc_mem_entry *carveout, *mapping = NULL;
+	struct rproc_mem_entry *mapping = NULL;
 	struct device *dev = &rproc->dev;
 	dma_addr_t dma;
 	void *va;
 	int ret;
 
-	if (sizeof(*rsc) > avail) {
-		dev_err(dev, "carveout rsc is truncated\n");
-		return -EINVAL;
-	}
-
-	/* make sure reserved bytes are zeroes */
-	if (rsc->reserved) {
-		dev_err(dev, "carveout rsc has non zero reserved bytes\n");
-		return -EINVAL;
-	}
-
-	dev_dbg(dev, "carveout rsc: name: %s, da 0x%x, pa 0x%x, len 0x%x, flags 0x%x\n",
-		rsc->name, rsc->da, rsc->pa, rsc->len, rsc->flags);
-
-	va = dma_alloc_coherent(dev->parent, rsc->len, &dma, GFP_KERNEL);
+	va = dma_alloc_coherent(dev->parent, mem->len, &dma, GFP_KERNEL);
 	if (!va) {
 		dev_err(dev->parent,
-			"failed to allocate dma memory: len 0x%x\n", rsc->len);
+			"failed to allocate dma memory: len 0x%x\n", mem->len);
 		return -ENOMEM;
 	}
 
 	dev_dbg(dev, "carveout va %pK, dma %pad, len 0x%x\n",
-		va, &dma, rsc->len);
+		va, &dma, mem->len);
 
 	/*
 	 * Ok, this is non-standard.
@@ -729,22 +686,22 @@ static int rproc_handle_carveout(struct rproc *rproc,
 	 * physical address in this case.
 	 */
 
-	if (rsc->da != FW_RSC_ADDR_ANY && !rproc->domain) {
-		dev_err(dev->parent,
-			"Bad carveout rsc configuration\n");
-		ret = -ENOMEM;
-		goto dma_free;
-	}
+	if (mem->da != FW_RSC_ADDR_ANY) {
+		if (!rproc->domain) {
+			dev_err(dev->parent,
+				"Bad carveout rsc configuration\n");
+			ret = -ENOMEM;
+			goto dma_free;
+		}
 
-	if (rsc->da != FW_RSC_ADDR_ANY && rproc->domain) {
 		mapping = kzalloc(sizeof(*mapping), GFP_KERNEL);
 		if (!mapping) {
 			ret = -ENOMEM;
 			goto dma_free;
 		}
 
-		ret = iommu_map(rproc->domain, rsc->da, dma, rsc->len,
-				rsc->flags);
+		ret = iommu_map(rproc->domain, mem->da, dma, mem->len,
+				mem->flags);
 		if (ret) {
 			dev_err(dev, "iommu_map failed: %d\n", ret);
 			goto free_mapping;
@@ -757,52 +714,102 @@ static int rproc_handle_carveout(struct rproc *rproc,
 		 * We can't trust the remote processor not to change the
 		 * resource table, so we must maintain this info independently.
 		 */
-		mapping->da = rsc->da;
-		mapping->len = rsc->len;
+		mapping->da = mem->da;
+		mapping->len = mem->len;
 		list_add_tail(&mapping->node, &rproc->mappings);
 
 		dev_dbg(dev, "carveout mapped 0x%x to %pad\n",
-			rsc->da, &dma);
+			mem->da, &dma);
+	} else {
+		mem->da = (u32)dma;
 	}
 
-	/*
-	 * Some remote processors might need to know the pa
-	 * even though they are behind an IOMMU. E.g., OMAP4's
-	 * remote M3 processor needs this so it can control
-	 * on-chip hardware accelerators that are not behind
-	 * the IOMMU, and therefor must know the pa.
-	 *
-	 * Generally we don't want to expose physical addresses
-	 * if we don't have to (remote processors are generally
-	 * _not_ trusted), so we might want to do this only for
-	 * remote processor that _must_ have this (e.g. OMAP4's
-	 * dual M3 subsystem).
-	 *
-	 * Non-IOMMU processors might also want to have this info.
-	 * In this case, the device address and the physical address
-	 * are the same.
-	 */
-	rsc->pa = (u32)rproc_va_to_pa(va);
-
-	carveout = rproc_mem_entry_init(dev, va, dma, rsc->len, rsc->da,
-					rproc_release_carveout, rsc->name);
-	if (!carveout)
-		goto free_carv;
-
-	rproc_add_carveout(rproc, carveout);
+	mem->dma = (u32)dma;
+	mem->va = va;
 
 	return 0;
 
-free_carv:
-	kfree(carveout);
 free_mapping:
 	kfree(mapping);
 dma_free:
-	dma_free_coherent(dev->parent, rsc->len, va, dma);
+	dma_free_coherent(dev->parent, mem->len, va, dma);
 	return ret;
 }
 
 /**
+ * rproc_release_carveout() - release acquired carveout
+ * @rproc: rproc handle
+ * @mem: the memory entry to release
+ *
+ * This function releases specified memory entry @mem allocated via
+ * rproc_alloc_carveout() function by @rproc.
+ */
+static int rproc_release_carveout(struct rproc *rproc,
+				  struct rproc_mem_entry *mem)
+{
+	struct device *dev = &rproc->dev;
+
+	/* clean up carveout allocations */
+	dma_free_coherent(dev->parent, mem->len, mem->va, mem->dma);
+	return 0;
+}
+
+/**
+ * rproc_handle_carveout() - handle phys contig memory allocation requests
+ * @rproc: rproc handle
+ * @rsc: the resource entry
+ * @avail: size of available data (for image validation)
+ *
+ * This function will handle firmware requests for allocation of physically
+ * contiguous memory regions.
+ *
+ * These request entries should come first in the firmware's resource table,
+ * as other firmware entries might request placing other data objects inside
+ * these memory regions (e.g. data/code segments, trace resource entries, ...).
+ *
+ * Allocating memory this way helps utilizing the reserved physical memory
+ * (e.g. CMA) more efficiently, and also minimizes the number of TLB entries
+ * needed to map it (in case @rproc is using an IOMMU). Reducing the TLB
+ * pressure is important; it may have a substantial impact on performance.
+ */
+static int rproc_handle_carveout(struct rproc *rproc,
+				 struct fw_rsc_carveout *rsc,
+				 int offset, int avail)
+{
+	struct rproc_mem_entry *carveout;
+	struct device *dev = &rproc->dev;
+
+	if (sizeof(*rsc) > avail) {
+		dev_err(dev, "carveout rsc is truncated\n");
+		return -EINVAL;
+	}
+
+	/* make sure reserved bytes are zeroes */
+	if (rsc->reserved) {
+		dev_err(dev, "carveout rsc has non zero reserved bytes\n");
+		return -EINVAL;
+	}
+
+	dev_dbg(dev, "carveout rsc: name: %s, da 0x%x, pa 0x%x, len 0x%x, flags 0x%x\n",
+		rsc->name, rsc->da, rsc->pa, rsc->len, rsc->flags);
+
+	/* Register carveout in in list */
+	carveout = rproc_mem_entry_init(dev, 0, 0, rsc->len, rsc->da,
+					rproc_alloc_carveout,
+					rproc_release_carveout, rsc->name);
+	if (!carveout) {
+		dev_err(dev, "Can't allocate memory entry structure\n");
+		return -ENOMEM;
+	}
+
+	carveout->flags = rsc->flags;
+	carveout->rsc_offset = offset;
+	rproc_add_carveout(rproc, carveout);
+
+	return 0;
+}
+
+/**
  * rproc_add_carveout() - register an allocated carveout region
  * @rproc: rproc handle
  * @mem: memory entry to register
@@ -832,6 +839,7 @@ void rproc_add_carveout(struct rproc *rproc, struct rproc_mem_entry *mem)
 struct rproc_mem_entry *
 rproc_mem_entry_init(struct device *dev,
 		     void *va, dma_addr_t dma, int len, u32 da,
+		     int (*alloc)(struct rproc *, struct rproc_mem_entry *),
 		     int (*release)(struct rproc *, struct rproc_mem_entry *),
 		     const char *name, ...)
 {
@@ -846,7 +854,9 @@ struct rproc_mem_entry *
 	mem->dma = dma;
 	mem->da = da;
 	mem->len = len;
+	mem->alloc = alloc;
 	mem->release = release;
+	mem->rsc_offset = FW_RSC_ADDR_ANY;
 
 	va_start(args, name);
 	vsnprintf(mem->name, sizeof(mem->name), name, args);
@@ -978,6 +988,63 @@ static void rproc_unprepare_subdevices(struct rproc *rproc)
 }
 
 /**
+ * rproc_alloc_registered_carveouts() - allocate all carveouts registered
+ * in the list
+ * @rproc: the remote processor handle
+ *
+ * This function parses registered carveout list, performs allocation
+ * if alloc() ops registered and updates resource table information
+ * if rsc_offset set.
+ *
+ * Return: 0 on success
+ */
+static int rproc_alloc_registered_carveouts(struct rproc *rproc)
+{
+	struct rproc_mem_entry *entry, *tmp;
+	struct fw_rsc_carveout *rsc;
+	struct device *dev = &rproc->dev;
+	int ret;
+
+	list_for_each_entry_safe(entry, tmp, &rproc->carveouts, node) {
+		if (entry->alloc) {
+			ret = entry->alloc(rproc, entry);
+			if (ret) {
+				dev_err(dev, "Unable to allocate carveout %s: %d\n",
+					entry->name, ret);
+				return -ENOMEM;
+			}
+		}
+
+		if (entry->rsc_offset != FW_RSC_ADDR_ANY) {
+			/* update resource table */
+			rsc = (void *)rproc->table_ptr + entry->rsc_offset;
+
+			/*
+			 * Some remote processors might need to know the pa
+			 * even though they are behind an IOMMU. E.g., OMAP4's
+			 * remote M3 processor needs this so it can control
+			 * on-chip hardware accelerators that are not behind
+			 * the IOMMU, and therefor must know the pa.
+			 *
+			 * Generally we don't want to expose physical addresses
+			 * if we don't have to (remote processors are generally
+			 * _not_ trusted), so we might want to do this only for
+			 * remote processor that _must_ have this (e.g. OMAP4's
+			 * dual M3 subsystem).
+			 *
+			 * Non-IOMMU processors might also want to have this info.
+			 * In this case, the device address and the physical address
+			 * are the same.
+			 */
+			if (entry->va)
+				rsc->pa = (u32)rproc_va_to_pa(entry->va);
+		}
+	}
+
+	return 0;
+}
+
+/**
  * rproc_coredump_cleanup() - clean up dump_segments list
  * @rproc: the remote processor handle
  */
@@ -1148,6 +1215,14 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
 		goto clean_up_resources;
 	}
 
+	/* Allocate carveout resources associated to rproc */
+	ret = rproc_alloc_registered_carveouts(rproc);
+	if (ret) {
+		dev_err(dev, "Failed to allocate associated carveouts: %d\n",
+			ret);
+		goto clean_up_resources;
+	}
+
 	ret = rproc_start(rproc, fw);
 	if (ret)
 		goto clean_up_resources;
diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
index 55f30fc..ea95b04 100644
--- a/include/linux/remoteproc.h
+++ b/include/linux/remoteproc.h
@@ -317,6 +317,9 @@ struct fw_rsc_vdev {
  * @priv: associated data
  * @name: associated memory region name (optional)
  * @node: list node
+ * @rsc_offset: offset in resource table
+ * @flags: iommu protection flags
+ * @alloc: specific memory allocator function
  */
 struct rproc_mem_entry {
 	void *va;
@@ -326,6 +329,9 @@ struct rproc_mem_entry {
 	void *priv;
 	char name[32];
 	struct list_head node;
+	u32 rsc_offset;
+	u32 flags;
+	int (*alloc)(struct rproc *rproc, struct rproc_mem_entry *mem);
 	int (*release)(struct rproc *rproc, struct rproc_mem_entry *mem);
 };
 
@@ -563,6 +569,7 @@ struct rproc *rproc_alloc(struct device *dev, const char *name,
 struct rproc_mem_entry *
 rproc_mem_entry_init(struct device *dev,
 		     void *va, dma_addr_t dma, int len, u32 da,
+		     int (*alloc)(struct rproc *, struct rproc_mem_entry *),
 		     int (*release)(struct rproc *, struct rproc_mem_entry *),
 		     const char *name, ...);
 
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 09/17] remoteproc: add helper function to allocate rproc_mem_entry from reserved memory
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (7 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 08/17] remoteproc: add alloc ops in rproc_mem_entry struct Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-07-27 13:14 ` [PATCH v4 10/17] remoteproc: add helper function to check carveout device address Loic Pallardy
                   ` (9 subsequent siblings)
  18 siblings, 0 replies; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

This patch introduces rproc_res_mem_entry_init() helper function to
allocate a rproc_mem_entry structure from a reserved memory region.
In that case, rproc_mem_entry structure has no alloc and release ops.
It will be used to assigned the specified reserved memory to any
rproc sub device.
Relation between rproc_mem_entry and rproc sub device will be done
by name.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/remoteproc/remoteproc_core.c | 37 ++++++++++++++++++++++++++++++++++++
 include/linux/remoteproc.h           |  6 ++++++
 2 files changed, 43 insertions(+)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index 2c51549..1e0fe3e 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -857,6 +857,7 @@ struct rproc_mem_entry *
 	mem->alloc = alloc;
 	mem->release = release;
 	mem->rsc_offset = FW_RSC_ADDR_ANY;
+	mem->of_resm_idx = -1;
 
 	va_start(args, name);
 	vsnprintf(mem->name, sizeof(mem->name), name, args);
@@ -867,6 +868,42 @@ struct rproc_mem_entry *
 EXPORT_SYMBOL(rproc_mem_entry_init);
 
 /**
+ * rproc_of_resm_mem_entry_init() - allocate and initialize rproc_mem_entry struct
+ * from a reserved memory phandle
+ * @dev: pointer on device struct
+ * @of_resm_idx: reserved memory phandle index in "memory-region"
+ * @len: memory carveout length
+ * @da: device address
+ * @name: carveout name
+ *
+ * This function allocates a rproc_mem_entry struct and fill it with parameters
+ * provided by client.
+ */
+struct rproc_mem_entry *
+rproc_of_resm_mem_entry_init(struct device *dev, u32 of_resm_idx, int len,
+			     u32 da, const char *name, ...)
+{
+	struct rproc_mem_entry *mem;
+	va_list args;
+
+	mem = kzalloc(sizeof(*mem), GFP_KERNEL);
+	if (!mem)
+		return mem;
+
+	mem->da = da;
+	mem->len = len;
+	mem->rsc_offset = FW_RSC_ADDR_ANY;
+	mem->of_resm_idx = of_resm_idx;
+
+	va_start(args, name);
+	vsnprintf(mem->name, sizeof(mem->name), name, args);
+	va_end(args);
+
+	return mem;
+}
+EXPORT_SYMBOL(rproc_of_resm_mem_entry_init);
+
+/**
  * A lookup table for resource handlers. The indices are defined in
  * enum fw_resource_type.
  */
diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
index ea95b04..4cdd0c6 100644
--- a/include/linux/remoteproc.h
+++ b/include/linux/remoteproc.h
@@ -319,6 +319,7 @@ struct fw_rsc_vdev {
  * @node: list node
  * @rsc_offset: offset in resource table
  * @flags: iommu protection flags
+ * @of_resm_idx: reserved memory phandle index
  * @alloc: specific memory allocator function
  */
 struct rproc_mem_entry {
@@ -331,6 +332,7 @@ struct rproc_mem_entry {
 	struct list_head node;
 	u32 rsc_offset;
 	u32 flags;
+	u32 of_resm_idx;
 	int (*alloc)(struct rproc *rproc, struct rproc_mem_entry *mem);
 	int (*release)(struct rproc *rproc, struct rproc_mem_entry *mem);
 };
@@ -573,6 +575,10 @@ struct rproc_mem_entry *
 		     int (*release)(struct rproc *, struct rproc_mem_entry *),
 		     const char *name, ...);
 
+struct rproc_mem_entry *
+rproc_of_resm_mem_entry_init(struct device *dev, u32 of_resm_idx, int len,
+			     u32 da, const char *name, ...);
+
 int rproc_boot(struct rproc *rproc);
 void rproc_shutdown(struct rproc *rproc);
 void rproc_report_crash(struct rproc *rproc, enum rproc_crash_type type);
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 10/17] remoteproc: add helper function to check carveout device address
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (8 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 09/17] remoteproc: add helper function to allocate rproc_mem_entry from reserved memory Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-10-23 22:14   ` Suman Anna
  2018-07-27 13:14 ` [PATCH v4 11/17] remoteproc: modify rproc_handle_carveout to support pre-registered region Loic Pallardy
                   ` (8 subsequent siblings)
  18 siblings, 1 reply; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

This patch introduces a function to verify that a specified carveout
is fitting request device address and associated length

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/remoteproc/remoteproc_core.c | 47 ++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index 1e0fe3e..5dd5edf 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -259,6 +259,53 @@ struct rproc_mem_entry *
 	return mem;
 }
 
+/**
+ * rproc_check_carveout_da() - Check specified carveout da configuration
+ * @rproc: handle of a remote processor
+ * @mem: pointer on carveout to check
+ * @da: area device address
+ * @len: associated area size
+ *
+ * This function is a helper function to verify requested device area (couple
+ * da, len) is part of specified carevout.
+ *
+ * Return: 0 if carveout matchs request else -ENOMEM
+ */
+int rproc_check_carveout_da(struct rproc *rproc, struct rproc_mem_entry *mem,
+			    u32 da, u32 len)
+{
+	struct device *dev = &rproc->dev;
+	int delta = 0;
+
+	/* Check requested resource length */
+	if (len > mem->len) {
+		dev_err(dev, "Registered carveout doesn't fit len request\n");
+		return -ENOMEM;
+	}
+
+	if (da != FW_RSC_ADDR_ANY && mem->da == FW_RSC_ADDR_ANY) {
+		/* Update existing carveout da */
+		mem->da = da;
+	} else if (da != FW_RSC_ADDR_ANY && mem->da != FW_RSC_ADDR_ANY) {
+		delta = da - mem->da;
+
+		/* Check requested resource belongs to registered carveout */
+		if (delta < 0) {
+			dev_err(dev,
+				"Registered carveout doesn't fit da request\n");
+			return -ENOMEM;
+		}
+
+		if (delta + len > mem->len) {
+			dev_err(dev,
+				"Registered carveout doesn't fit len request\n");
+			return -ENOMEM;
+		}
+	}
+
+	return 0;
+}
+
 int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
 {
 	struct rproc *rproc = rvdev->rproc;
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 11/17] remoteproc: modify rproc_handle_carveout to support pre-registered region
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (9 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 10/17] remoteproc: add helper function to check carveout device address Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-07-27 13:14 ` [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator Loic Pallardy
                   ` (7 subsequent siblings)
  18 siblings, 0 replies; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

In current version rproc_handle_carveout() function registers carveout
for allocation.
This patch extends rproc_handle_carveout() function to support
pre-registered region. Match is done on region name, then requested
device address and length are checked.
If match found, pre-registered region is associated with resource
table request.
If no name match found, new carveout is registered for allocation.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/remoteproc/remoteproc_core.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index 5dd5edf..c543d04 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -840,6 +840,29 @@ static int rproc_handle_carveout(struct rproc *rproc,
 	dev_dbg(dev, "carveout rsc: name: %s, da 0x%x, pa 0x%x, len 0x%x, flags 0x%x\n",
 		rsc->name, rsc->da, rsc->pa, rsc->len, rsc->flags);
 
+	/*
+	 * Check carveout rsc already part of a registered carveout,
+	 * Search by name, then check the da and length
+	 */
+	carveout = rproc_find_carveout_by_name(rproc, rsc->name);
+
+	if (carveout) {
+		if (carveout->rsc_offset != FW_RSC_ADDR_ANY) {
+			dev_err(dev,
+				"Carveout already associated to resource table\n");
+			return -ENOMEM;
+		}
+
+		if (rproc_check_carveout_da(rproc, carveout, rsc->da, rsc->len))
+			return -ENOMEM;
+
+		/* Update memory carveout with resource table info */
+		carveout->rsc_offset = offset;
+		carveout->flags = rsc->flags;
+
+		return 0;
+	}
+
 	/* Register carveout in in list */
 	carveout = rproc_mem_entry_init(dev, 0, 0, rsc->len, rsc->da,
 					rproc_alloc_carveout,
@@ -1120,8 +1143,15 @@ static int rproc_alloc_registered_carveouts(struct rproc *rproc)
 			 * In this case, the device address and the physical address
 			 * are the same.
 			 */
+
+			/* Use va if defined else dma to generate pa */
 			if (entry->va)
 				rsc->pa = (u32)rproc_va_to_pa(entry->va);
+			else
+				rsc->pa = (u32)entry->dma;
+
+			rsc->da = entry->da;
+			rsc->len = entry->len;
 		}
 	}
 
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (10 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 11/17] remoteproc: modify rproc_handle_carveout to support pre-registered region Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-10-10  5:32   ` Bjorn Andersson
  2018-10-24  0:14   ` Suman Anna
  2018-07-27 13:14 ` [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool Loic Pallardy
                   ` (6 subsequent siblings)
  18 siblings, 2 replies; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

Current version of rproc_alloc_vring function supports only dynamic vring
allocation.

This patch allows to allocate vrings based on memory region declatation.
Vrings are now manage like memory carveouts, to communize memory management
code in rproc_alloc_registered_carveouts().

Allocated buffer is retrieved in rp_find_vq() thanks to
rproc_find_carveout_by_name() functions for.

This patch sets vrings names to vdev"x"vring"y" with x vdev index in
resource table and y vring index in vdev. This will be updated when
name will be associated to vdev in firmware resource table.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/remoteproc/remoteproc_core.c     | 61 +++++++++++++++++---------------
 drivers/remoteproc/remoteproc_internal.h |  2 ++
 drivers/remoteproc/remoteproc_virtio.c   | 14 +++++++-
 include/linux/remoteproc.h               |  6 ++--
 4 files changed, 51 insertions(+), 32 deletions(-)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index c543d04..4edc6f0 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -53,6 +53,11 @@ typedef int (*rproc_handle_resources_t)(struct rproc *rproc,
 typedef int (*rproc_handle_resource_t)(struct rproc *rproc,
 				 void *, int offset, int avail);
 
+static int rproc_alloc_carveout(struct rproc *rproc,
+				struct rproc_mem_entry *mem);
+static int rproc_release_carveout(struct rproc *rproc,
+				  struct rproc_mem_entry *mem);
+
 /* Unique indices for remoteproc devices */
 static DEFINE_IDA(rproc_dev_index);
 
@@ -312,21 +317,33 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
 	struct device *dev = &rproc->dev;
 	struct rproc_vring *rvring = &rvdev->vring[i];
 	struct fw_rsc_vdev *rsc;
-	dma_addr_t dma;
-	void *va;
 	int ret, size, notifyid;
+	struct rproc_mem_entry *mem;
 
 	/* actual size of vring (in bytes) */
 	size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
 
-	/*
-	 * Allocate non-cacheable memory for the vring. In the future
-	 * this call will also configure the IOMMU for us
-	 */
-	va = dma_alloc_coherent(dev->parent, size, &dma, GFP_KERNEL);
-	if (!va) {
-		dev_err(dev->parent, "dma_alloc_coherent failed\n");
-		return -EINVAL;
+	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
+
+	/* Search for pre-registered carveout */
+	mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d", rvdev->index,
+					  i);
+	if (mem) {
+		if (rproc_check_carveout_da(rproc, mem, rsc->vring[i].da, size))
+			return -ENOMEM;
+	} else {
+		/* Register carveout in in list */
+		mem = rproc_mem_entry_init(dev, 0, 0, size, rsc->vring[i].da,
+					   rproc_alloc_carveout,
+					   rproc_release_carveout,
+					   "vdev%dvring%d",
+					   rvdev->index, i);
+		if (!mem) {
+			dev_err(dev, "Can't allocate memory entry structure\n");
+			return -ENOMEM;
+		}
+
+		rproc_add_carveout(rproc, mem);
 	}
 
 	/*
@@ -337,7 +354,6 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
 	ret = idr_alloc(&rproc->notifyids, rvring, 0, 0, GFP_KERNEL);
 	if (ret < 0) {
 		dev_err(dev, "idr_alloc failed: %d\n", ret);
-		dma_free_coherent(dev->parent, size, va, dma);
 		return ret;
 	}
 	notifyid = ret;
@@ -346,21 +362,9 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
 	if (notifyid > rproc->max_notifyid)
 		rproc->max_notifyid = notifyid;
 
-	dev_dbg(dev, "vring%d: va %pK dma %pad size 0x%x idr %d\n",
-		i, va, &dma, size, notifyid);
-
-	rvring->va = va;
-	rvring->dma = dma;
 	rvring->notifyid = notifyid;
 
-	/*
-	 * Let the rproc know the notifyid and da of this vring.
-	 * Not all platforms use dma_alloc_coherent to automatically
-	 * set up the iommu. In this case the device address (da) will
-	 * hold the physical address and not the device address.
-	 */
-	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
-	rsc->vring[i].da = dma;
+	/* Let the rproc know the notifyid of this vring.*/
 	rsc->vring[i].notifyid = notifyid;
 	return 0;
 }
@@ -392,12 +396,10 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
 
 void rproc_free_vring(struct rproc_vring *rvring)
 {
-	int size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
 	struct rproc *rproc = rvring->rvdev->rproc;
 	int idx = rvring->rvdev->vring - rvring;
 	struct fw_rsc_vdev *rsc;
 
-	dma_free_coherent(rproc->dev.parent, size, rvring->va, rvring->dma);
 	idr_remove(&rproc->notifyids, rvring->notifyid);
 
 	/* reset resource entry info */
@@ -484,6 +486,7 @@ static int rproc_handle_vdev(struct rproc *rproc, struct fw_rsc_vdev *rsc,
 
 	rvdev->id = rsc->id;
 	rvdev->rproc = rproc;
+	rvdev->index = rproc->nb_vdev++;
 
 	/* parse the vrings */
 	for (i = 0; i < rsc->num_of_vrings; i++) {
@@ -528,9 +531,6 @@ void rproc_vdev_release(struct kref *ref)
 
 	for (id = 0; id < ARRAY_SIZE(rvdev->vring); id++) {
 		rvring = &rvdev->vring[id];
-		if (!rvring->va)
-			continue;
-
 		rproc_free_vring(rvring);
 	}
 
@@ -1322,6 +1322,9 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
 	/* reset max_notifyid */
 	rproc->max_notifyid = -1;
 
+	/* reset handled vdev */
+	rproc->nb_vdev = 0;
+
 	/* handle fw resources which are required to boot rproc */
 	ret = rproc_handle_resources(rproc, rproc_loading_handlers);
 	if (ret) {
diff --git a/drivers/remoteproc/remoteproc_internal.h b/drivers/remoteproc/remoteproc_internal.h
index 7570beb..f6cad24 100644
--- a/drivers/remoteproc/remoteproc_internal.h
+++ b/drivers/remoteproc/remoteproc_internal.h
@@ -60,6 +60,8 @@ struct dentry *rproc_create_trace_file(const char *name, struct rproc *rproc,
 int rproc_elf_load_rsc_table(struct rproc *rproc, const struct firmware *fw);
 struct resource_table *rproc_elf_find_loaded_rsc_table(struct rproc *rproc,
 						       const struct firmware *fw);
+struct rproc_mem_entry *
+rproc_find_carveout_by_name(struct rproc *rproc, const char *name, ...);
 
 static inline
 int rproc_fw_sanity_check(struct rproc *rproc, const struct firmware *fw)
diff --git a/drivers/remoteproc/remoteproc_virtio.c b/drivers/remoteproc/remoteproc_virtio.c
index bbecd44..de21f62 100644
--- a/drivers/remoteproc/remoteproc_virtio.c
+++ b/drivers/remoteproc/remoteproc_virtio.c
@@ -76,7 +76,9 @@ static struct virtqueue *rp_find_vq(struct virtio_device *vdev,
 	struct rproc_vdev *rvdev = vdev_to_rvdev(vdev);
 	struct rproc *rproc = vdev_to_rproc(vdev);
 	struct device *dev = &rproc->dev;
+	struct rproc_mem_entry *mem;
 	struct rproc_vring *rvring;
+	struct fw_rsc_vdev *rsc;
 	struct virtqueue *vq;
 	void *addr;
 	int len, size;
@@ -88,8 +90,14 @@ static struct virtqueue *rp_find_vq(struct virtio_device *vdev,
 	if (!name)
 		return NULL;
 
+	/* Search allocated memory region by name */
+	mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d", rvdev->index,
+					  id);
+	if (!mem || !mem->va)
+		return ERR_PTR(-ENOMEM);
+
 	rvring = &rvdev->vring[id];
-	addr = rvring->va;
+	addr = mem->va;
 	len = rvring->len;
 
 	/* zero vring */
@@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct virtio_device *vdev,
 	rvring->vq = vq;
 	vq->priv = rvring;
 
+	/* Update vring in resource table */
+	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
+	rsc->vring[id].da = mem->da;
+
 	return vq;
 }
 
diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
index 4cdd0c6..6b3a234 100644
--- a/include/linux/remoteproc.h
+++ b/include/linux/remoteproc.h
@@ -453,6 +453,7 @@ struct rproc_dump_segment {
  * @table_sz: size of @cached_table
  * @has_iommu: flag to indicate if remote processor is behind an MMU
  * @dump_segments: list of segments in the firmware
+ * @nb_vdev: number of vdev currently handled by rproc
  */
 struct rproc {
 	struct list_head node;
@@ -485,6 +486,7 @@ struct rproc {
 	bool has_iommu;
 	bool auto_boot;
 	struct list_head dump_segments;
+	int nb_vdev;
 };
 
 /**
@@ -512,7 +514,6 @@ struct rproc_subdev {
 /**
  * struct rproc_vring - remoteproc vring state
  * @va:	virtual address
- * @dma: dma address
  * @len: length, in bytes
  * @da: device address
  * @align: vring alignment
@@ -522,7 +523,6 @@ struct rproc_subdev {
  */
 struct rproc_vring {
 	void *va;
-	dma_addr_t dma;
 	int len;
 	u32 da;
 	u32 align;
@@ -541,6 +541,7 @@ struct rproc_vring {
  * @vdev: the virio device
  * @vring: the vrings for this vdev
  * @rsc_offset: offset of the vdev's resource entry
+ * @index: vdev position versus other vdev declared in resource table
  */
 struct rproc_vdev {
 	struct kref refcount;
@@ -553,6 +554,7 @@ struct rproc_vdev {
 	struct virtio_device vdev;
 	struct rproc_vring vring[RVDEV_NUM_VRINGS];
 	u32 rsc_offset;
+	u32 index;
 };
 
 struct rproc *rproc_get_by_phandle(phandle phandle);
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (11 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-09-27 17:17   ` Wendy Liang
  2018-10-10  5:58   ` Bjorn Andersson
  2018-07-27 13:14 ` [PATCH v4 14/17] remoteproc: keystone: declare reserved memory region for vdev device Loic Pallardy
                   ` (5 subsequent siblings)
  18 siblings, 2 replies; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

This patch creates a dedicated vdev subdevice for each vdev declared
in firmware resource table and associates carveout named "vdev%dbuffer"
(with %d vdev index in resource table) if any as dma coherent memory pool.

Then vdev subdevice is used as parent for virtio device.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/remoteproc/remoteproc_core.c     | 35 +++++++++++++++++++++++---
 drivers/remoteproc/remoteproc_internal.h |  1 +
 drivers/remoteproc/remoteproc_virtio.c   | 42 +++++++++++++++++++++++++++++++-
 include/linux/remoteproc.h               |  1 +
 4 files changed, 75 insertions(+), 4 deletions(-)

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
index 4edc6f0..adcc66e 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -39,6 +39,7 @@
 #include <linux/idr.h>
 #include <linux/elf.h>
 #include <linux/crc32.h>
+#include <linux/of_reserved_mem.h>
 #include <linux/virtio_ids.h>
 #include <linux/virtio_ring.h>
 #include <asm/byteorder.h>
@@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc *rproc)
 	iommu_domain_free(domain);
 }
 
-static phys_addr_t rproc_va_to_pa(void *cpu_addr)
+phys_addr_t rproc_va_to_pa(void *cpu_addr)
 {
 	/*
 	 * Return physical address according to virtual address location
@@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void *cpu_addr)
 	WARN_ON(!virt_addr_valid(cpu_addr));
 	return virt_to_phys(cpu_addr);
 }
+EXPORT_SYMBOL(rproc_va_to_pa);
 
 /**
  * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc address
@@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct rproc_subdev *subdev, bool crashed)
 }
 
 /**
+ * rproc_rvdev_release() - release the existence of a rvdev
+ *
+ * @dev: the subdevice's dev
+ */
+static void rproc_rvdev_release(struct device *dev)
+{
+	struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev, dev);
+
+	of_reserved_mem_device_release(dev);
+
+	kfree(rvdev);
+}
+
+/**
  * rproc_handle_vdev() - handle a vdev fw resource
  * @rproc: the remote processor
  * @rsc: the vring resource descriptor
@@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc *rproc, struct fw_rsc_vdev *rsc,
 	struct device *dev = &rproc->dev;
 	struct rproc_vdev *rvdev;
 	int i, ret;
+	char name[16];
 
 	/* make sure resource isn't truncated */
 	if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct fw_rsc_vdev_vring)
@@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc *rproc, struct fw_rsc_vdev *rsc,
 	rvdev->rproc = rproc;
 	rvdev->index = rproc->nb_vdev++;
 
+	/* Initialise vdev subdevice */
+	snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
+	rvdev->dev.parent = rproc->dev.parent;
+	rvdev->dev.release = rproc_rvdev_release;
+	dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev->dev.parent), name);
+	dev_set_drvdata(&rvdev->dev, rvdev);
+	dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
+
+	ret = device_register(&rvdev->dev);
+	if (ret)
+		goto free_rvdev;
+
 	/* parse the vrings */
 	for (i = 0; i < rsc->num_of_vrings; i++) {
 		ret = rproc_parse_vring(rvdev, rsc, i);
@@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc *rproc, struct fw_rsc_vdev *rsc,
 	for (i--; i >= 0; i--)
 		rproc_free_vring(&rvdev->vring[i]);
 free_rvdev:
-	kfree(rvdev);
+	device_unregister(&rvdev->dev);
 	return ret;
 }
 
@@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
 
 	rproc_remove_subdev(rproc, &rvdev->subdev);
 	list_del(&rvdev->node);
-	kfree(rvdev);
+	device_unregister(&rvdev->dev);
 }
 
 /**
diff --git a/drivers/remoteproc/remoteproc_internal.h b/drivers/remoteproc/remoteproc_internal.h
index f6cad24..bfeacfd 100644
--- a/drivers/remoteproc/remoteproc_internal.h
+++ b/drivers/remoteproc/remoteproc_internal.h
@@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const char *name, struct rproc *rproc,
 int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
 
 void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
+phys_addr_t rproc_va_to_pa(void *cpu_addr);
 int rproc_trigger_recovery(struct rproc *rproc);
 
 int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware *fw);
diff --git a/drivers/remoteproc/remoteproc_virtio.c b/drivers/remoteproc/remoteproc_virtio.c
index de21f62..9ee63c0 100644
--- a/drivers/remoteproc/remoteproc_virtio.c
+++ b/drivers/remoteproc/remoteproc_virtio.c
@@ -17,7 +17,9 @@
  * GNU General Public License for more details.
  */
 
+#include <linux/dma-mapping.h>
 #include <linux/export.h>
+#include <linux/of_reserved_mem.h>
 #include <linux/remoteproc.h>
 #include <linux/virtio.h>
 #include <linux/virtio_config.h>
@@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct device *dev)
 int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
 {
 	struct rproc *rproc = rvdev->rproc;
-	struct device *dev = &rproc->dev;
+	struct device *dev = &rvdev->dev;
 	struct virtio_device *vdev = &rvdev->vdev;
+	struct rproc_mem_entry *mem;
 	int ret;
 
+	/* Try to find dedicated vdev buffer carveout */
+	mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer", rvdev->index);
+	if (mem) {
+		phys_addr_t pa;
+
+		if (mem->of_resm_idx != -1) {
+			struct device_node *np = rproc->dev.parent->of_node;
+
+			/* Associate reserved memory to vdev device */
+			ret = of_reserved_mem_device_init_by_idx(dev, np,
+								 mem->of_resm_idx);
+			if (ret) {
+				dev_err(dev, "Can't associate reserved memory\n");
+				goto out;
+			}
+		} else {
+			if (mem->va) {
+				dev_warn(dev, "vdev %d buffer already mapped\n",
+					 rvdev->index);
+				pa = rproc_va_to_pa(mem->va);
+			} else {
+				/* Use dma address as carveout no memmapped yet */
+				pa = (phys_addr_t)mem->dma;
+			}
+
+			/* Associate vdev buffer memory pool to vdev subdev */
+			ret = dmam_declare_coherent_memory(dev, pa,
+							   mem->da,
+							   mem->len,
+							   DMA_MEMORY_EXCLUSIVE);
+			if (ret < 0) {
+				dev_err(dev, "Failed to associate buffer\n");
+				goto out;
+			}
+		}
+	}
+
 	vdev->id.device	= id,
 	vdev->config = &rproc_virtio_config_ops,
 	vdev->dev.parent = dev;
diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
index 6b3a234..2921dd2 100644
--- a/include/linux/remoteproc.h
+++ b/include/linux/remoteproc.h
@@ -547,6 +547,7 @@ struct rproc_vdev {
 	struct kref refcount;
 
 	struct rproc_subdev subdev;
+	struct device dev;
 
 	unsigned int id;
 	struct list_head node;
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 14/17] remoteproc: keystone: declare reserved memory region for vdev device
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (12 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-07-27 13:14 ` [PATCH v4 15/17] remoteproc: da8xx: " Loic Pallardy
                   ` (4 subsequent siblings)
  18 siblings, 0 replies; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

This patch introduces keystone_rproc_parse_fw() to declare a
carveout region based on reserved memory for vdev buffer
allocation, if a memory region has been declared on rproc DT node.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/remoteproc/keystone_remoteproc.c | 53 +++++++++++++++++++++++++++++---
 1 file changed, 49 insertions(+), 4 deletions(-)

diff --git a/drivers/remoteproc/keystone_remoteproc.c b/drivers/remoteproc/keystone_remoteproc.c
index aaac311..ef542f2 100644
--- a/drivers/remoteproc/keystone_remoteproc.c
+++ b/drivers/remoteproc/keystone_remoteproc.c
@@ -293,11 +293,56 @@ static void *keystone_rproc_da_to_va(struct rproc *rproc, u64 da, int len)
 	return (__force void *)va;
 }
 
+/*
+ * Custom function to register platform specific memory region(s)
+ * before loading firmware resource table thanks to generic
+ * rproc_elf_load_rsc_table() function.
+ */
+static int keystone_rproc_parse_fw(struct rproc *rproc,
+				   const struct firmware *fw)
+{
+	struct device *dev = rproc->dev.parent;
+	struct rproc_mem_entry *mem;
+	struct device_node *node;
+	struct resource res;
+	int err;
+
+	node = of_parse_phandle(dev->of_node, "memory-region", 0);
+	if (!node) {
+		dev_warn(dev, "No memory-region specified\n");
+		goto out;
+	}
+
+	err = of_address_to_resource(node, 0, &res);
+	if (err) {
+		dev_err(dev, "Bad memory-region definition\n");
+		return err;
+	}
+
+	/* Register memory region for vdev buffer allocation */
+	mem = rproc_of_resm_mem_entry_init(dev, 0, resource_size(&res),
+					   res.start, "vdev0buffer");
+
+	if (!mem)
+		return -ENOMEM;
+
+	rproc_add_carveout(rproc, mem);
+
+out:
+	return rproc_elf_load_rsc_table(rproc, fw);
+}
+
 static const struct rproc_ops keystone_rproc_ops = {
-	.start		= keystone_rproc_start,
-	.stop		= keystone_rproc_stop,
-	.kick		= keystone_rproc_kick,
-	.da_to_va	= keystone_rproc_da_to_va,
+	.start			= keystone_rproc_start,
+	.stop			= keystone_rproc_stop,
+	.kick			= keystone_rproc_kick,
+	.da_to_va		= keystone_rproc_da_to_va,
+	.parse_fw		= keystone_rproc_parse_fw,
+	.load			= rproc_elf_load_segments,
+	.find_loaded_rsc_table	= rproc_elf_find_loaded_rsc_table,
+	.sanity_check		= rproc_elf_sanity_check,
+	.get_boot_addr		= rproc_elf_get_boot_addr,
+
 };
 
 static int keystone_rproc_of_get_memories(struct platform_device *pdev,
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 15/17] remoteproc: da8xx: declare reserved memory region for vdev device
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (13 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 14/17] remoteproc: keystone: declare reserved memory region for vdev device Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-10-24  2:57   ` Suman Anna
  2018-07-27 13:14 ` [PATCH v4 16/17] remoteproc: st: add reserved memory support Loic Pallardy
                   ` (3 subsequent siblings)
  18 siblings, 1 reply; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

This patch introduces da8xx_rproc_parse_fw() to declare a
carveout region based on reserved memory for vdev buffer
allocation.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/remoteproc/da8xx_remoteproc.c | 38 +++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/drivers/remoteproc/da8xx_remoteproc.c b/drivers/remoteproc/da8xx_remoteproc.c
index b668e32..679a076 100644
--- a/drivers/remoteproc/da8xx_remoteproc.c
+++ b/drivers/remoteproc/da8xx_remoteproc.c
@@ -16,6 +16,7 @@
 #include <linux/irq.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/of_address.h>
 #include <linux/of_reserved_mem.h>
 #include <linux/platform_device.h>
 #include <linux/remoteproc.h>
@@ -179,10 +180,47 @@ static void da8xx_rproc_kick(struct rproc *rproc, int vqid)
 	writel(SYSCFG_CHIPSIG2, drproc->chipsig);
 }
 
+static int da8xx_rproc_parse_fw(struct rproc *rproc, const struct firmware *fw)
+{
+	struct device *dev = rproc->dev.parent;
+	struct rproc_mem_entry *mem;
+	struct device_node *node;
+	struct resource res;
+	int err;
+
+	node = of_parse_phandle(dev->of_node, "memory-region", 0);
+	if (!node) {
+		dev_err(dev, "No memory-region specified\n");
+		return -EINVAL;
+	}
+
+	err = of_address_to_resource(node, 0, &res);
+	if (err) {
+		dev_err(dev, "Bad memory-region definition\n");
+		return err;
+	}
+
+	/* Register memory region for vdev buffer allocation */
+	mem = rproc_of_resm_mem_entry_init(dev, 0, resource_size(&res),
+					   res.start, "vdev0buffer");
+
+	if (!mem)
+		return -ENOMEM;
+
+	rproc_add_carveout(rproc, mem);
+
+	return rproc_elf_load_rsc_table(rproc, fw);
+}
+
 static const struct rproc_ops da8xx_rproc_ops = {
 	.start = da8xx_rproc_start,
 	.stop = da8xx_rproc_stop,
 	.kick = da8xx_rproc_kick,
+	.parse_fw = da8xx_rproc_parse_fw,
+	.load = rproc_elf_load_segments,
+	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
+	.sanity_check = rproc_elf_sanity_check,
+	.get_boot_addr = rproc_elf_get_boot_addr,
 };
 
 static int da8xx_rproc_get_internal_memories(struct platform_device *pdev,
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 16/17] remoteproc: st: add reserved memory support
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (14 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 15/17] remoteproc: da8xx: " Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-10-24  3:01   ` Suman Anna
  2018-07-27 13:14 ` [PATCH v4 17/17] rpmsg: virtio: allocate buffer from parent Loic Pallardy
                   ` (2 subsequent siblings)
  18 siblings, 1 reply; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

ST remote processor needs some specified memory regions for
firmware and IPC.
Memory regions are defined as reserved memory and should
be registered in remoteproc core thanks to rproc_add_carveout
function before rproc_start. For this, st rproc driver implements
prepare ops.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/remoteproc/st_remoteproc.c | 96 +++++++++++++++++++++++++++++++++-----
 1 file changed, 85 insertions(+), 11 deletions(-)

diff --git a/drivers/remoteproc/st_remoteproc.c b/drivers/remoteproc/st_remoteproc.c
index aacef0e..45958d5 100644
--- a/drivers/remoteproc/st_remoteproc.c
+++ b/drivers/remoteproc/st_remoteproc.c
@@ -19,6 +19,7 @@
 #include <linux/mfd/syscon.h>
 #include <linux/module.h>
 #include <linux/of.h>
+#include <linux/of_address.h>
 #include <linux/of_device.h>
 #include <linux/of_reserved_mem.h>
 #include <linux/platform_device.h>
@@ -91,6 +92,82 @@ static void st_rproc_kick(struct rproc *rproc, int vqid)
 		dev_err(dev, "failed to send message via mbox: %d\n", ret);
 }
 
+static int st_rproc_mem_alloc(struct rproc *rproc,
+			      struct rproc_mem_entry *mem)
+{
+	struct device *dev = rproc->dev.parent;
+	void *va;
+
+	va = ioremap_wc(mem->dma, mem->len);
+	if (!va) {
+		dev_err(dev, "Unable to map memory region: %pa+%zx\n",
+			&mem->dma, mem->len);
+		return -ENOMEM;
+	}
+
+	/* Update memory entry va */
+	mem->va = va;
+
+	return 0;
+}
+
+static int st_rproc_mem_release(struct rproc *rproc,
+				struct rproc_mem_entry *mem)
+{
+	iounmap(mem->va);
+
+	return 0;
+}
+
+static int st_rproc_parse_fw(struct rproc *rproc, const struct firmware *fw)
+{
+	struct device *dev = rproc->dev.parent;
+	struct device_node *np = dev->of_node;
+	struct rproc_mem_entry *mem;
+	void *va;
+	struct reserved_mem *rmem;
+	struct of_phandle_iterator it;
+	int err, index = 0;
+
+	of_phandle_iterator_init(&it, np, "memory-region", NULL, 0);
+	while ((err = of_phandle_iterator_next(&it)) == 0) {
+		va = NULL;
+		rmem = of_reserved_mem_lookup(it.node);
+
+		/*  No need to map vdev buffer */
+		if (strcmp(it.node->name, "vdev0buffer")) {
+			va = devm_ioremap_wc(dev, rmem->base, rmem->size);
+			if (!va) {
+				dev_err(dev, "Unable to map memory region: %pa+%zx\n",
+					&rmem->base, rmem->size);
+				return -ENOMEM;
+			}
+
+			/* Register memory region */
+			mem = rproc_mem_entry_init(dev, va,
+						   (dma_addr_t)rmem->base,
+						   rmem->size, rmem->base,
+						   st_rproc_mem_alloc,
+						   st_rproc_mem_release,
+						   it.node->name);
+		} else {
+			/* Register reserved memory for vdev buffer allocation */
+			mem = rproc_of_resm_mem_entry_init(dev, index,
+							   rmem->size,
+							   rmem->base,
+							   it.node->name);
+		}
+
+		if (!mem)
+			return -ENOMEM;
+
+		rproc_add_carveout(rproc, mem);
+		index++;
+	}
+
+	return rproc_elf_load_rsc_table(rproc, fw);
+}
+
 static int st_rproc_start(struct rproc *rproc)
 {
 	struct st_rproc *ddata = rproc->priv;
@@ -158,9 +235,14 @@ static int st_rproc_stop(struct rproc *rproc)
 }
 
 static const struct rproc_ops st_rproc_ops = {
-	.kick		= st_rproc_kick,
-	.start		= st_rproc_start,
-	.stop		= st_rproc_stop,
+	.kick			= st_rproc_kick,
+	.start			= st_rproc_start,
+	.stop			= st_rproc_stop,
+	.parse_fw		= st_rproc_parse_fw,
+	.load			= rproc_elf_load_segments,
+	.find_loaded_rsc_table	= rproc_elf_find_loaded_rsc_table,
+	.sanity_check		= rproc_elf_sanity_check,
+	.get_boot_addr		= rproc_elf_get_boot_addr,
 };
 
 /*
@@ -254,12 +336,6 @@ static int st_rproc_parse_dt(struct platform_device *pdev)
 		return -EINVAL;
 	}
 
-	err = of_reserved_mem_device_init(dev);
-	if (err) {
-		dev_err(dev, "Failed to obtain shared memory\n");
-		return err;
-	}
-
 	err = clk_prepare(ddata->clk);
 	if (err)
 		dev_err(dev, "failed to get clock\n");
@@ -387,8 +463,6 @@ static int st_rproc_remove(struct platform_device *pdev)
 
 	clk_disable_unprepare(ddata->clk);
 
-	of_reserved_mem_device_release(&pdev->dev);
-
 	for (i = 0; i < ST_RPROC_MAX_VRING * MBOX_MAX; i++)
 		mbox_free_channel(ddata->mbox_chan[i]);
 
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* [PATCH v4 17/17] rpmsg: virtio: allocate buffer from parent
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (15 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 16/17] remoteproc: st: add reserved memory support Loic Pallardy
@ 2018-07-27 13:14 ` Loic Pallardy
  2018-09-28  7:56   ` Anup Patel
  2018-09-21  6:04 ` [PATCH v4 00/17] remoteproc: add fixed memory region support Anup Patel
  2018-10-23 16:42 ` Suman Anna
  18 siblings, 1 reply; 75+ messages in thread
From: Loic Pallardy @ 2018-07-27 13:14 UTC (permalink / raw)
  To: bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna, Loic Pallardy

Remoteproc is now capable to create one specific sub-device per
virtio link to associate a dedicated memory pool.
This implies to change device used by virtio_rpmsg for
buffer allocation from grand-parent to parent.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
---
 drivers/rpmsg/virtio_rpmsg_bus.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/rpmsg/virtio_rpmsg_bus.c b/drivers/rpmsg/virtio_rpmsg_bus.c
index 664f957..5c89201 100644
--- a/drivers/rpmsg/virtio_rpmsg_bus.c
+++ b/drivers/rpmsg/virtio_rpmsg_bus.c
@@ -912,7 +912,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	total_buf_space = vrp->num_bufs * vrp->buf_size;
 
 	/* allocate coherent memory for the buffers */
-	bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
+	bufs_va = dma_alloc_coherent(vdev->dev.parent,
 				     total_buf_space, &vrp->bufs_dma,
 				     GFP_KERNEL);
 	if (!bufs_va) {
@@ -980,7 +980,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	return 0;
 
 free_coherent:
-	dma_free_coherent(vdev->dev.parent->parent, total_buf_space,
+	dma_free_coherent(vdev->dev.parent, total_buf_space,
 			  bufs_va, vrp->bufs_dma);
 vqs_del:
 	vdev->config->del_vqs(vrp->vdev);
@@ -1015,7 +1015,7 @@ static void rpmsg_remove(struct virtio_device *vdev)
 
 	vdev->config->del_vqs(vrp->vdev);
 
-	dma_free_coherent(vdev->dev.parent->parent, total_buf_space,
+	dma_free_coherent(vdev->dev.parent, total_buf_space,
 			  vrp->rbufs, vrp->bufs_dma);
 
 	kfree(vrp);
-- 
1.9.1


^ permalink raw reply related	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 00/17] remoteproc: add fixed memory region support
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (16 preceding siblings ...)
  2018-07-27 13:14 ` [PATCH v4 17/17] rpmsg: virtio: allocate buffer from parent Loic Pallardy
@ 2018-09-21  6:04 ` Anup Patel
  2018-09-26 16:00   ` Loic PALLARDY
  2018-10-23 16:42 ` Suman Anna
  18 siblings, 1 reply; 75+ messages in thread
From: Anup Patel @ 2018-09-21  6:04 UTC (permalink / raw)
  To: Loic Pallardy
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	linux-kernel@vger.kernel.org List, arnaud pouliquen,
	Benjamin Gaignard, s-anna

Hi Loic/Bjorn,

Can we consider this series for v4.20?

I am interested in for patch "rpmsg: virtio: allocate buffer
from parent". This patch allows me to used VirtIO
RPMSG driver inside Guest/VM.

Regards,
Anup

^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 00/17] remoteproc: add fixed memory region support
  2018-09-21  6:04 ` [PATCH v4 00/17] remoteproc: add fixed memory region support Anup Patel
@ 2018-09-26 16:00   ` Loic PALLARDY
  2018-09-28  7:54     ` Anup Patel
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-09-26 16:00 UTC (permalink / raw)
  To: Anup Patel, Bjorn Andersson
  Cc: Ohad Ben-Cohen, linux-remoteproc,
	linux-kernel@vger.kernel.org List, Arnaud POULIQUEN,
	Benjamin Gaignard, s-anna

Hi Anup,

> -----Original Message-----
> From: Anup Patel <anup@brainfault.org>
> Sent: Friday, September 21, 2018 8:05 AM
> To: Loic PALLARDY <loic.pallardy@st.com>
> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
> <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; linux-
> kernel@vger.kernel.org List <linux-kernel@vger.kernel.org>; Arnaud
> POULIQUEN <arnaud.pouliquen@st.com>; Benjamin Gaignard
> <benjamin.gaignard@linaro.org>; s-anna@ti.com
> Subject: Re: [PATCH v4 00/17] remoteproc: add fixed memory region support
> 
> Hi Loic/Bjorn,
> 
> Can we consider this series for v4.20?

I discussed with Bjorn last week at Linaro connect.
He should review the series hopefully soon.
> 
> I am interested in for patch "rpmsg: virtio: allocate buffer
> from parent". This patch allows me to used VirtIO
> RPMSG driver inside Guest/VM.
Please send your acked-by or reviewed-by or tested-by 
It will help Bjorn to see interest for this series

Regards,
Loic

> 
> Regards,
> Anup

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-07-27 13:14 ` [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool Loic Pallardy
@ 2018-09-27 17:17   ` Wendy Liang
  2018-09-27 19:22     ` Loic PALLARDY
  2018-10-10  5:58   ` Bjorn Andersson
  1 sibling, 1 reply; 75+ messages in thread
From: Wendy Liang @ 2018-09-27 17:17 UTC (permalink / raw)
  To: Loic Pallardy
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, arnaud.pouliquen, benjamin.gaignard,
	Suman Anna

On Fri, Jul 27, 2018 at 6:16 AM Loic Pallardy <loic.pallardy@st.com> wrote:
>
> This patch creates a dedicated vdev subdevice for each vdev declared
> in firmware resource table and associates carveout named "vdev%dbuffer"
> (with %d vdev index in resource table) if any as dma coherent memory pool.
>
> Then vdev subdevice is used as parent for virtio device.
>
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> ---
>  drivers/remoteproc/remoteproc_core.c     | 35 +++++++++++++++++++++++---
>  drivers/remoteproc/remoteproc_internal.h |  1 +
>  drivers/remoteproc/remoteproc_virtio.c   | 42 +++++++++++++++++++++++++++++++-
>  include/linux/remoteproc.h               |  1 +
>  4 files changed, 75 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index 4edc6f0..adcc66e 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -39,6 +39,7 @@
>  #include <linux/idr.h>
>  #include <linux/elf.h>
>  #include <linux/crc32.h>
> +#include <linux/of_reserved_mem.h>
>  #include <linux/virtio_ids.h>
>  #include <linux/virtio_ring.h>
>  #include <asm/byteorder.h>
> @@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc *rproc)
>         iommu_domain_free(domain);
>  }
>
> -static phys_addr_t rproc_va_to_pa(void *cpu_addr)
> +phys_addr_t rproc_va_to_pa(void *cpu_addr)
>  {
>         /*
>          * Return physical address according to virtual address location
> @@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void *cpu_addr)
>         WARN_ON(!virt_addr_valid(cpu_addr));
>         return virt_to_phys(cpu_addr);
>  }
> +EXPORT_SYMBOL(rproc_va_to_pa);
>
>  /**
>   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc address
> @@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct rproc_subdev *subdev, bool crashed)
>  }
>
>  /**
> + * rproc_rvdev_release() - release the existence of a rvdev
> + *
> + * @dev: the subdevice's dev
> + */
> +static void rproc_rvdev_release(struct device *dev)
> +{
> +       struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev, dev);
> +
> +       of_reserved_mem_device_release(dev);
> +
> +       kfree(rvdev);
> +}
> +
> +/**
>   * rproc_handle_vdev() - handle a vdev fw resource
>   * @rproc: the remote processor
>   * @rsc: the vring resource descriptor
> @@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc *rproc, struct fw_rsc_vdev *rsc,
>         struct device *dev = &rproc->dev;
>         struct rproc_vdev *rvdev;
>         int i, ret;
> +       char name[16];
>
>         /* make sure resource isn't truncated */
>         if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct fw_rsc_vdev_vring)
> @@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc *rproc, struct fw_rsc_vdev *rsc,
>         rvdev->rproc = rproc;
>         rvdev->index = rproc->nb_vdev++;
>
> +       /* Initialise vdev subdevice */
> +       snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
> +       rvdev->dev.parent = rproc->dev.parent;
> +       rvdev->dev.release = rproc_rvdev_release;
> +       dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev->dev.parent), name);
> +       dev_set_drvdata(&rvdev->dev, rvdev);
> +       dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
I tried the latest kernel, this function will not set the DMA coherent mask as
dma_supported() of the &rvdev->dev will return false.
As this is a device created at run time, should it be force to support DMA?
should it directly set the dma_coherent_mask?

> +
> +       ret = device_register(&rvdev->dev);
> +       if (ret)
> +               goto free_rvdev;
> +
>         /* parse the vrings */
>         for (i = 0; i < rsc->num_of_vrings; i++) {
>                 ret = rproc_parse_vring(rvdev, rsc, i);
> @@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc *rproc, struct fw_rsc_vdev *rsc,
>         for (i--; i >= 0; i--)
>                 rproc_free_vring(&rvdev->vring[i]);
>  free_rvdev:
> -       kfree(rvdev);
> +       device_unregister(&rvdev->dev);
>         return ret;
>  }
>
> @@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
>
>         rproc_remove_subdev(rproc, &rvdev->subdev);
>         list_del(&rvdev->node);
> -       kfree(rvdev);
> +       device_unregister(&rvdev->dev);
>  }
>
>  /**
> diff --git a/drivers/remoteproc/remoteproc_internal.h b/drivers/remoteproc/remoteproc_internal.h
> index f6cad24..bfeacfd 100644
> --- a/drivers/remoteproc/remoteproc_internal.h
> +++ b/drivers/remoteproc/remoteproc_internal.h
> @@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const char *name, struct rproc *rproc,
>  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
>
>  void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
> +phys_addr_t rproc_va_to_pa(void *cpu_addr);
>  int rproc_trigger_recovery(struct rproc *rproc);
>
>  int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware *fw);
> diff --git a/drivers/remoteproc/remoteproc_virtio.c b/drivers/remoteproc/remoteproc_virtio.c
> index de21f62..9ee63c0 100644
> --- a/drivers/remoteproc/remoteproc_virtio.c
> +++ b/drivers/remoteproc/remoteproc_virtio.c
> @@ -17,7 +17,9 @@
>   * GNU General Public License for more details.
>   */
>
> +#include <linux/dma-mapping.h>
>  #include <linux/export.h>
> +#include <linux/of_reserved_mem.h>
>  #include <linux/remoteproc.h>
>  #include <linux/virtio.h>
>  #include <linux/virtio_config.h>
> @@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct device *dev)
>  int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
>  {
>         struct rproc *rproc = rvdev->rproc;
> -       struct device *dev = &rproc->dev;
> +       struct device *dev = &rvdev->dev;
>         struct virtio_device *vdev = &rvdev->vdev;
> +       struct rproc_mem_entry *mem;
>         int ret;
>
> +       /* Try to find dedicated vdev buffer carveout */
> +       mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer", rvdev->index);
> +       if (mem) {
> +               phys_addr_t pa;
> +
> +               if (mem->of_resm_idx != -1) {
> +                       struct device_node *np = rproc->dev.parent->of_node;
> +
> +                       /* Associate reserved memory to vdev device */
> +                       ret = of_reserved_mem_device_init_by_idx(dev, np,
> +                                                                mem->of_resm_idx);
> +                       if (ret) {
> +                               dev_err(dev, "Can't associate reserved memory\n");
> +                               goto out;
> +                       }
> +               } else {
> +                       if (mem->va) {
> +                               dev_warn(dev, "vdev %d buffer already mapped\n",
> +                                        rvdev->index);
> +                               pa = rproc_va_to_pa(mem->va);
> +                       } else {
> +                               /* Use dma address as carveout no memmapped yet */
> +                               pa = (phys_addr_t)mem->dma;
> +                       }
> +
> +                       /* Associate vdev buffer memory pool to vdev subdev */
> +                       ret = dmam_declare_coherent_memory(dev, pa,
> +                                                          mem->da,
> +                                                          mem->len,
> +                                                          DMA_MEMORY_EXCLUSIVE);
> +                       if (ret < 0) {
> +                               dev_err(dev, "Failed to associate buffer\n");
> +                               goto out;
> +                       }
> +               }
> +       }
> +
>         vdev->id.device = id,
>         vdev->config = &rproc_virtio_config_ops,
>         vdev->dev.parent = dev;
> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> index 6b3a234..2921dd2 100644
> --- a/include/linux/remoteproc.h
> +++ b/include/linux/remoteproc.h
> @@ -547,6 +547,7 @@ struct rproc_vdev {
>         struct kref refcount;
>
>         struct rproc_subdev subdev;
> +       struct device dev;
>
>         unsigned int id;
>         struct list_head node;
> --
> 1.9.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-remoteproc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-09-27 17:17   ` Wendy Liang
@ 2018-09-27 19:22     ` Loic PALLARDY
  2018-09-27 20:18       ` Wendy Liang
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-09-27 19:22 UTC (permalink / raw)
  To: Wendy Liang
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, Arnaud POULIQUEN, benjamin.gaignard,
	Suman Anna

Hi Wendy

> -----Original Message-----
> From: Wendy Liang <sunnyliangjy@gmail.com>
> Sent: Thursday, September 27, 2018 7:17 PM
> To: Loic PALLARDY <loic.pallardy@st.com>
> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
> <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; Linux Kernel
> Mailing List <linux-kernel@vger.kernel.org>; Arnaud POULIQUEN
> <arnaud.pouliquen@st.com>; benjamin.gaignard@linaro.org; Suman Anna
> <s-anna@ti.com>
> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
> specific dma memory pool
> 
> On Fri, Jul 27, 2018 at 6:16 AM Loic Pallardy <loic.pallardy@st.com> wrote:
> >
> > This patch creates a dedicated vdev subdevice for each vdev declared
> > in firmware resource table and associates carveout named "vdev%dbuffer"
> > (with %d vdev index in resource table) if any as dma coherent memory
> pool.
> >
> > Then vdev subdevice is used as parent for virtio device.
> >
> > Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > ---
> >  drivers/remoteproc/remoteproc_core.c     | 35
> +++++++++++++++++++++++---
> >  drivers/remoteproc/remoteproc_internal.h |  1 +
> >  drivers/remoteproc/remoteproc_virtio.c   | 42
> +++++++++++++++++++++++++++++++-
> >  include/linux/remoteproc.h               |  1 +
> >  4 files changed, 75 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/remoteproc/remoteproc_core.c
> b/drivers/remoteproc/remoteproc_core.c
> > index 4edc6f0..adcc66e 100644
> > --- a/drivers/remoteproc/remoteproc_core.c
> > +++ b/drivers/remoteproc/remoteproc_core.c
> > @@ -39,6 +39,7 @@
> >  #include <linux/idr.h>
> >  #include <linux/elf.h>
> >  #include <linux/crc32.h>
> > +#include <linux/of_reserved_mem.h>
> >  #include <linux/virtio_ids.h>
> >  #include <linux/virtio_ring.h>
> >  #include <asm/byteorder.h>
> > @@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc
> *rproc)
> >         iommu_domain_free(domain);
> >  }
> >
> > -static phys_addr_t rproc_va_to_pa(void *cpu_addr)
> > +phys_addr_t rproc_va_to_pa(void *cpu_addr)
> >  {
> >         /*
> >          * Return physical address according to virtual address location
> > @@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void
> *cpu_addr)
> >         WARN_ON(!virt_addr_valid(cpu_addr));
> >         return virt_to_phys(cpu_addr);
> >  }
> > +EXPORT_SYMBOL(rproc_va_to_pa);
> >
> >  /**
> >   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc
> address
> > @@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct
> rproc_subdev *subdev, bool crashed)
> >  }
> >
> >  /**
> > + * rproc_rvdev_release() - release the existence of a rvdev
> > + *
> > + * @dev: the subdevice's dev
> > + */
> > +static void rproc_rvdev_release(struct device *dev)
> > +{
> > +       struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev, dev);
> > +
> > +       of_reserved_mem_device_release(dev);
> > +
> > +       kfree(rvdev);
> > +}
> > +
> > +/**
> >   * rproc_handle_vdev() - handle a vdev fw resource
> >   * @rproc: the remote processor
> >   * @rsc: the vring resource descriptor
> > @@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
> struct fw_rsc_vdev *rsc,
> >         struct device *dev = &rproc->dev;
> >         struct rproc_vdev *rvdev;
> >         int i, ret;
> > +       char name[16];
> >
> >         /* make sure resource isn't truncated */
> >         if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct
> fw_rsc_vdev_vring)
> > @@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc *rproc,
> struct fw_rsc_vdev *rsc,
> >         rvdev->rproc = rproc;
> >         rvdev->index = rproc->nb_vdev++;
> >
> > +       /* Initialise vdev subdevice */
> > +       snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
> > +       rvdev->dev.parent = rproc->dev.parent;
> > +       rvdev->dev.release = rproc_rvdev_release;
> > +       dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev-
> >dev.parent), name);
> > +       dev_set_drvdata(&rvdev->dev, rvdev);
> > +       dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
> I tried the latest kernel, this function will not set the DMA coherent mask as
> dma_supported() of the &rvdev->dev will return false.
> As this is a device created at run time, should it be force to support DMA?
> should it directly set the dma_coherent_mask?

Thanks for pointing me this issue. I tested on top of 4.18-rc1 few months ago...
Could you please give me kernel version on which you are testing the series?
Is you platform 32bit or 64bit ?
I'll rebase and check on my side.

Regards,
Loic

> 
> > +
> > +       ret = device_register(&rvdev->dev);
> > +       if (ret)
> > +               goto free_rvdev;
> > +
> >         /* parse the vrings */
> >         for (i = 0; i < rsc->num_of_vrings; i++) {
> >                 ret = rproc_parse_vring(rvdev, rsc, i);
> > @@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
> struct fw_rsc_vdev *rsc,
> >         for (i--; i >= 0; i--)
> >                 rproc_free_vring(&rvdev->vring[i]);
> >  free_rvdev:
> > -       kfree(rvdev);
> > +       device_unregister(&rvdev->dev);
> >         return ret;
> >  }
> >
> > @@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
> >
> >         rproc_remove_subdev(rproc, &rvdev->subdev);
> >         list_del(&rvdev->node);
> > -       kfree(rvdev);
> > +       device_unregister(&rvdev->dev);
> >  }
> >
> >  /**
> > diff --git a/drivers/remoteproc/remoteproc_internal.h
> b/drivers/remoteproc/remoteproc_internal.h
> > index f6cad24..bfeacfd 100644
> > --- a/drivers/remoteproc/remoteproc_internal.h
> > +++ b/drivers/remoteproc/remoteproc_internal.h
> > @@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const char
> *name, struct rproc *rproc,
> >  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
> >
> >  void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
> > +phys_addr_t rproc_va_to_pa(void *cpu_addr);
> >  int rproc_trigger_recovery(struct rproc *rproc);
> >
> >  int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware *fw);
> > diff --git a/drivers/remoteproc/remoteproc_virtio.c
> b/drivers/remoteproc/remoteproc_virtio.c
> > index de21f62..9ee63c0 100644
> > --- a/drivers/remoteproc/remoteproc_virtio.c
> > +++ b/drivers/remoteproc/remoteproc_virtio.c
> > @@ -17,7 +17,9 @@
> >   * GNU General Public License for more details.
> >   */
> >
> > +#include <linux/dma-mapping.h>
> >  #include <linux/export.h>
> > +#include <linux/of_reserved_mem.h>
> >  #include <linux/remoteproc.h>
> >  #include <linux/virtio.h>
> >  #include <linux/virtio_config.h>
> > @@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct device
> *dev)
> >  int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
> >  {
> >         struct rproc *rproc = rvdev->rproc;
> > -       struct device *dev = &rproc->dev;
> > +       struct device *dev = &rvdev->dev;
> >         struct virtio_device *vdev = &rvdev->vdev;
> > +       struct rproc_mem_entry *mem;
> >         int ret;
> >
> > +       /* Try to find dedicated vdev buffer carveout */
> > +       mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer", rvdev-
> >index);
> > +       if (mem) {
> > +               phys_addr_t pa;
> > +
> > +               if (mem->of_resm_idx != -1) {
> > +                       struct device_node *np = rproc->dev.parent->of_node;
> > +
> > +                       /* Associate reserved memory to vdev device */
> > +                       ret = of_reserved_mem_device_init_by_idx(dev, np,
> > +                                                                mem->of_resm_idx);
> > +                       if (ret) {
> > +                               dev_err(dev, "Can't associate reserved memory\n");
> > +                               goto out;
> > +                       }
> > +               } else {
> > +                       if (mem->va) {
> > +                               dev_warn(dev, "vdev %d buffer already mapped\n",
> > +                                        rvdev->index);
> > +                               pa = rproc_va_to_pa(mem->va);
> > +                       } else {
> > +                               /* Use dma address as carveout no memmapped yet */
> > +                               pa = (phys_addr_t)mem->dma;
> > +                       }
> > +
> > +                       /* Associate vdev buffer memory pool to vdev subdev */
> > +                       ret = dmam_declare_coherent_memory(dev, pa,
> > +                                                          mem->da,
> > +                                                          mem->len,
> > +                                                          DMA_MEMORY_EXCLUSIVE);
> > +                       if (ret < 0) {
> > +                               dev_err(dev, "Failed to associate buffer\n");
> > +                               goto out;
> > +                       }
> > +               }
> > +       }
> > +
> >         vdev->id.device = id,
> >         vdev->config = &rproc_virtio_config_ops,
> >         vdev->dev.parent = dev;
> > diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> > index 6b3a234..2921dd2 100644
> > --- a/include/linux/remoteproc.h
> > +++ b/include/linux/remoteproc.h
> > @@ -547,6 +547,7 @@ struct rproc_vdev {
> >         struct kref refcount;
> >
> >         struct rproc_subdev subdev;
> > +       struct device dev;
> >
> >         unsigned int id;
> >         struct list_head node;
> > --
> > 1.9.1
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-remoteproc"
> in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-09-27 19:22     ` Loic PALLARDY
@ 2018-09-27 20:18       ` Wendy Liang
  2018-10-24  1:22         ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Wendy Liang @ 2018-09-27 20:18 UTC (permalink / raw)
  To: Loic Pallardy
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, arnaud.pouliquen, benjamin.gaignard,
	Suman Anna

Hi Loic,


On Thu, Sep 27, 2018 at 12:22 PM Loic PALLARDY <loic.pallardy@st.com> wrote:
>
> Hi Wendy
>
> > -----Original Message-----
> > From: Wendy Liang <sunnyliangjy@gmail.com>
> > Sent: Thursday, September 27, 2018 7:17 PM
> > To: Loic PALLARDY <loic.pallardy@st.com>
> > Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
> > <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; Linux Kernel
> > Mailing List <linux-kernel@vger.kernel.org>; Arnaud POULIQUEN
> > <arnaud.pouliquen@st.com>; benjamin.gaignard@linaro.org; Suman Anna
> > <s-anna@ti.com>
> > Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
> > specific dma memory pool
> >
> > On Fri, Jul 27, 2018 at 6:16 AM Loic Pallardy <loic.pallardy@st.com> wrote:
> > >
> > > This patch creates a dedicated vdev subdevice for each vdev declared
> > > in firmware resource table and associates carveout named "vdev%dbuffer"
> > > (with %d vdev index in resource table) if any as dma coherent memory
> > pool.
> > >
> > > Then vdev subdevice is used as parent for virtio device.
> > >
> > > Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > > ---
> > >  drivers/remoteproc/remoteproc_core.c     | 35
> > +++++++++++++++++++++++---
> > >  drivers/remoteproc/remoteproc_internal.h |  1 +
> > >  drivers/remoteproc/remoteproc_virtio.c   | 42
> > +++++++++++++++++++++++++++++++-
> > >  include/linux/remoteproc.h               |  1 +
> > >  4 files changed, 75 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/drivers/remoteproc/remoteproc_core.c
> > b/drivers/remoteproc/remoteproc_core.c
> > > index 4edc6f0..adcc66e 100644
> > > --- a/drivers/remoteproc/remoteproc_core.c
> > > +++ b/drivers/remoteproc/remoteproc_core.c
> > > @@ -39,6 +39,7 @@
> > >  #include <linux/idr.h>
> > >  #include <linux/elf.h>
> > >  #include <linux/crc32.h>
> > > +#include <linux/of_reserved_mem.h>
> > >  #include <linux/virtio_ids.h>
> > >  #include <linux/virtio_ring.h>
> > >  #include <asm/byteorder.h>
> > > @@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc
> > *rproc)
> > >         iommu_domain_free(domain);
> > >  }
> > >
> > > -static phys_addr_t rproc_va_to_pa(void *cpu_addr)
> > > +phys_addr_t rproc_va_to_pa(void *cpu_addr)
> > >  {
> > >         /*
> > >          * Return physical address according to virtual address location
> > > @@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void
> > *cpu_addr)
> > >         WARN_ON(!virt_addr_valid(cpu_addr));
> > >         return virt_to_phys(cpu_addr);
> > >  }
> > > +EXPORT_SYMBOL(rproc_va_to_pa);
> > >
> > >  /**
> > >   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc
> > address
> > > @@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct
> > rproc_subdev *subdev, bool crashed)
> > >  }
> > >
> > >  /**
> > > + * rproc_rvdev_release() - release the existence of a rvdev
> > > + *
> > > + * @dev: the subdevice's dev
> > > + */
> > > +static void rproc_rvdev_release(struct device *dev)
> > > +{
> > > +       struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev, dev);
> > > +
> > > +       of_reserved_mem_device_release(dev);
> > > +
> > > +       kfree(rvdev);
> > > +}
> > > +
> > > +/**
> > >   * rproc_handle_vdev() - handle a vdev fw resource
> > >   * @rproc: the remote processor
> > >   * @rsc: the vring resource descriptor
> > > @@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
> > struct fw_rsc_vdev *rsc,
> > >         struct device *dev = &rproc->dev;
> > >         struct rproc_vdev *rvdev;
> > >         int i, ret;
> > > +       char name[16];
> > >
> > >         /* make sure resource isn't truncated */
> > >         if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct
> > fw_rsc_vdev_vring)
> > > @@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc *rproc,
> > struct fw_rsc_vdev *rsc,
> > >         rvdev->rproc = rproc;
> > >         rvdev->index = rproc->nb_vdev++;
> > >
> > > +       /* Initialise vdev subdevice */
> > > +       snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
> > > +       rvdev->dev.parent = rproc->dev.parent;
> > > +       rvdev->dev.release = rproc_rvdev_release;
> > > +       dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev-
> > >dev.parent), name);
> > > +       dev_set_drvdata(&rvdev->dev, rvdev);
> > > +       dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
> > I tried the latest kernel, this function will not set the DMA coherent mask as
> > dma_supported() of the &rvdev->dev will return false.
> > As this is a device created at run time, should it be force to support DMA?
> > should it directly set the dma_coherent_mask?
>
> Thanks for pointing me this issue. I tested on top of 4.18-rc1 few months ago...
> Could you please give me kernel version on which you are testing the series?
> Is you platform 32bit or 64bit ?
> I'll rebase and check on my side.

I am testing with 4.19-rc4 on aarch64 platform.

Best Regards,
Wendy
>
> Regards,
> Loic
>
> >
> > > +
> > > +       ret = device_register(&rvdev->dev);
> > > +       if (ret)
> > > +               goto free_rvdev;
> > > +
> > >         /* parse the vrings */
> > >         for (i = 0; i < rsc->num_of_vrings; i++) {
> > >                 ret = rproc_parse_vring(rvdev, rsc, i);
> > > @@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
> > struct fw_rsc_vdev *rsc,
> > >         for (i--; i >= 0; i--)
> > >                 rproc_free_vring(&rvdev->vring[i]);
> > >  free_rvdev:
> > > -       kfree(rvdev);
> > > +       device_unregister(&rvdev->dev);
> > >         return ret;
> > >  }
> > >
> > > @@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
> > >
> > >         rproc_remove_subdev(rproc, &rvdev->subdev);
> > >         list_del(&rvdev->node);
> > > -       kfree(rvdev);
> > > +       device_unregister(&rvdev->dev);
> > >  }
> > >
> > >  /**
> > > diff --git a/drivers/remoteproc/remoteproc_internal.h
> > b/drivers/remoteproc/remoteproc_internal.h
> > > index f6cad24..bfeacfd 100644
> > > --- a/drivers/remoteproc/remoteproc_internal.h
> > > +++ b/drivers/remoteproc/remoteproc_internal.h
> > > @@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const char
> > *name, struct rproc *rproc,
> > >  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
> > >
> > >  void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
> > > +phys_addr_t rproc_va_to_pa(void *cpu_addr);
> > >  int rproc_trigger_recovery(struct rproc *rproc);
> > >
> > >  int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware *fw);
> > > diff --git a/drivers/remoteproc/remoteproc_virtio.c
> > b/drivers/remoteproc/remoteproc_virtio.c
> > > index de21f62..9ee63c0 100644
> > > --- a/drivers/remoteproc/remoteproc_virtio.c
> > > +++ b/drivers/remoteproc/remoteproc_virtio.c
> > > @@ -17,7 +17,9 @@
> > >   * GNU General Public License for more details.
> > >   */
> > >
> > > +#include <linux/dma-mapping.h>
> > >  #include <linux/export.h>
> > > +#include <linux/of_reserved_mem.h>
> > >  #include <linux/remoteproc.h>
> > >  #include <linux/virtio.h>
> > >  #include <linux/virtio_config.h>
> > > @@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct device
> > *dev)
> > >  int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
> > >  {
> > >         struct rproc *rproc = rvdev->rproc;
> > > -       struct device *dev = &rproc->dev;
> > > +       struct device *dev = &rvdev->dev;
> > >         struct virtio_device *vdev = &rvdev->vdev;
> > > +       struct rproc_mem_entry *mem;
> > >         int ret;
> > >
> > > +       /* Try to find dedicated vdev buffer carveout */
> > > +       mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer", rvdev-
> > >index);
> > > +       if (mem) {
> > > +               phys_addr_t pa;
> > > +
> > > +               if (mem->of_resm_idx != -1) {
> > > +                       struct device_node *np = rproc->dev.parent->of_node;
> > > +
> > > +                       /* Associate reserved memory to vdev device */
> > > +                       ret = of_reserved_mem_device_init_by_idx(dev, np,
> > > +                                                                mem->of_resm_idx);
> > > +                       if (ret) {
> > > +                               dev_err(dev, "Can't associate reserved memory\n");
> > > +                               goto out;
> > > +                       }
> > > +               } else {
> > > +                       if (mem->va) {
> > > +                               dev_warn(dev, "vdev %d buffer already mapped\n",
> > > +                                        rvdev->index);
> > > +                               pa = rproc_va_to_pa(mem->va);
> > > +                       } else {
> > > +                               /* Use dma address as carveout no memmapped yet */
> > > +                               pa = (phys_addr_t)mem->dma;
> > > +                       }
> > > +
> > > +                       /* Associate vdev buffer memory pool to vdev subdev */
> > > +                       ret = dmam_declare_coherent_memory(dev, pa,
> > > +                                                          mem->da,
> > > +                                                          mem->len,
> > > +                                                          DMA_MEMORY_EXCLUSIVE);
> > > +                       if (ret < 0) {
> > > +                               dev_err(dev, "Failed to associate buffer\n");
> > > +                               goto out;
> > > +                       }
> > > +               }
> > > +       }
> > > +
> > >         vdev->id.device = id,
> > >         vdev->config = &rproc_virtio_config_ops,
> > >         vdev->dev.parent = dev;
> > > diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> > > index 6b3a234..2921dd2 100644
> > > --- a/include/linux/remoteproc.h
> > > +++ b/include/linux/remoteproc.h
> > > @@ -547,6 +547,7 @@ struct rproc_vdev {
> > >         struct kref refcount;
> > >
> > >         struct rproc_subdev subdev;
> > > +       struct device dev;
> > >
> > >         unsigned int id;
> > >         struct list_head node;
> > > --
> > > 1.9.1
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-remoteproc"
> > in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 00/17] remoteproc: add fixed memory region support
  2018-09-26 16:00   ` Loic PALLARDY
@ 2018-09-28  7:54     ` Anup Patel
  0 siblings, 0 replies; 75+ messages in thread
From: Anup Patel @ 2018-09-28  7:54 UTC (permalink / raw)
  To: Loic Pallardy
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	linux-kernel@vger.kernel.org List, arnaud pouliquen,
	Benjamin Gaignard, s-anna

On Wed, Sep 26, 2018 at 9:31 PM Loic PALLARDY <loic.pallardy@st.com> wrote:
>
> Hi Anup,
>
> > -----Original Message-----
> > From: Anup Patel <anup@brainfault.org>
> > Sent: Friday, September 21, 2018 8:05 AM
> > To: Loic PALLARDY <loic.pallardy@st.com>
> > Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
> > <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; linux-
> > kernel@vger.kernel.org List <linux-kernel@vger.kernel.org>; Arnaud
> > POULIQUEN <arnaud.pouliquen@st.com>; Benjamin Gaignard
> > <benjamin.gaignard@linaro.org>; s-anna@ti.com
> > Subject: Re: [PATCH v4 00/17] remoteproc: add fixed memory region support
> >
> > Hi Loic/Bjorn,
> >
> > Can we consider this series for v4.20?
>
> I discussed with Bjorn last week at Linaro connect.
> He should review the series hopefully soon.
> >
> > I am interested in for patch "rpmsg: virtio: allocate buffer
> > from parent". This patch allows me to used VirtIO
> > RPMSG driver inside Guest/VM.
> Please send your acked-by or reviewed-by or tested-by
> It will help Bjorn to see interest for this series

Sure, I will provided Reviewed-by and Tested-by for
patch "rpmsg: virtio: allocate buffer from parent".

Thanks,
Anup

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 17/17] rpmsg: virtio: allocate buffer from parent
  2018-07-27 13:14 ` [PATCH v4 17/17] rpmsg: virtio: allocate buffer from parent Loic Pallardy
@ 2018-09-28  7:56   ` Anup Patel
  0 siblings, 0 replies; 75+ messages in thread
From: Anup Patel @ 2018-09-28  7:56 UTC (permalink / raw)
  To: Loic Pallardy
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	linux-kernel@vger.kernel.org List, arnaud pouliquen,
	Benjamin Gaignard, s-anna

On Fri, Jul 27, 2018 at 6:45 PM Loic Pallardy <loic.pallardy@st.com> wrote:
>
> Remoteproc is now capable to create one specific sub-device per
> virtio link to associate a dedicated memory pool.
> This implies to change device used by virtio_rpmsg for
> buffer allocation from grand-parent to parent.
>
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> ---
>  drivers/rpmsg/virtio_rpmsg_bus.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/rpmsg/virtio_rpmsg_bus.c b/drivers/rpmsg/virtio_rpmsg_bus.c
> index 664f957..5c89201 100644
> --- a/drivers/rpmsg/virtio_rpmsg_bus.c
> +++ b/drivers/rpmsg/virtio_rpmsg_bus.c
> @@ -912,7 +912,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
>         total_buf_space = vrp->num_bufs * vrp->buf_size;
>
>         /* allocate coherent memory for the buffers */
> -       bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
> +       bufs_va = dma_alloc_coherent(vdev->dev.parent,
>                                      total_buf_space, &vrp->bufs_dma,
>                                      GFP_KERNEL);
>         if (!bufs_va) {
> @@ -980,7 +980,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
>         return 0;
>
>  free_coherent:
> -       dma_free_coherent(vdev->dev.parent->parent, total_buf_space,
> +       dma_free_coherent(vdev->dev.parent, total_buf_space,
>                           bufs_va, vrp->bufs_dma);
>  vqs_del:
>         vdev->config->del_vqs(vrp->vdev);
> @@ -1015,7 +1015,7 @@ static void rpmsg_remove(struct virtio_device *vdev)
>
>         vdev->config->del_vqs(vrp->vdev);
>
> -       dma_free_coherent(vdev->dev.parent->parent, total_buf_space,
> +       dma_free_coherent(vdev->dev.parent, total_buf_space,
>                           vrp->rbufs, vrp->bufs_dma);
>
>         kfree(vrp);
> --
> 1.9.1
>

Reviewed-by: Anup Patel <anup@brainfault.org>

Also, tried this patch on Linux running inside Guest/VM on
Xvisor ARM/ARM64.
Tested-by: Anup Patel <anup@brainfault.org>

Regards,
Anup

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-07-27 13:14 ` [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator Loic Pallardy
@ 2018-10-10  5:32   ` Bjorn Andersson
  2018-10-10 18:58     ` Loic PALLARDY
  2018-10-24  0:14   ` Suman Anna
  1 sibling, 1 reply; 75+ messages in thread
From: Bjorn Andersson @ 2018-10-10  5:32 UTC (permalink / raw)
  To: Loic Pallardy
  Cc: ohad, linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna

On Fri 27 Jul 06:14 PDT 2018, Loic Pallardy wrote:
>  int rproc_fw_sanity_check(struct rproc *rproc, const struct firmware *fw)
> diff --git a/drivers/remoteproc/remoteproc_virtio.c b/drivers/remoteproc/remoteproc_virtio.c
[..]
> @@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct virtio_device *vdev,
>  	rvring->vq = vq;
>  	vq->priv = rvring;
>  
> +	/* Update vring in resource table */
> +	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> +	rsc->vring[id].da = mem->da;
> +

This would now happen after we've started the remoteproc. Don't we need
to do this in-between allocating the carveouts and booting the
remoteproc?

Regards,
Bjorn

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-07-27 13:14 ` [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool Loic Pallardy
  2018-09-27 17:17   ` Wendy Liang
@ 2018-10-10  5:58   ` Bjorn Andersson
  2018-10-10 19:17     ` Loic PALLARDY
  1 sibling, 1 reply; 75+ messages in thread
From: Bjorn Andersson @ 2018-10-10  5:58 UTC (permalink / raw)
  To: Loic Pallardy
  Cc: ohad, linux-remoteproc, linux-kernel, arnaud.pouliquen,
	benjamin.gaignard, s-anna

On Fri 27 Jul 06:14 PDT 2018, Loic Pallardy wrote:

> This patch creates a dedicated vdev subdevice for each vdev declared
> in firmware resource table and associates carveout named "vdev%dbuffer"
> (with %d vdev index in resource table) if any as dma coherent memory pool.
> 
> Then vdev subdevice is used as parent for virtio device.
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> ---
>  drivers/remoteproc/remoteproc_core.c     | 35 +++++++++++++++++++++++---
>  drivers/remoteproc/remoteproc_internal.h |  1 +
>  drivers/remoteproc/remoteproc_virtio.c   | 42 +++++++++++++++++++++++++++++++-
>  include/linux/remoteproc.h               |  1 +
>  4 files changed, 75 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index 4edc6f0..adcc66e 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -39,6 +39,7 @@
>  #include <linux/idr.h>
>  #include <linux/elf.h>
>  #include <linux/crc32.h>
> +#include <linux/of_reserved_mem.h>
>  #include <linux/virtio_ids.h>
>  #include <linux/virtio_ring.h>
>  #include <asm/byteorder.h>
> @@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc *rproc)
>  	iommu_domain_free(domain);
>  }
>  
> -static phys_addr_t rproc_va_to_pa(void *cpu_addr)
> +phys_addr_t rproc_va_to_pa(void *cpu_addr)
>  {
>  	/*
>  	 * Return physical address according to virtual address location
> @@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void *cpu_addr)
>  	WARN_ON(!virt_addr_valid(cpu_addr));
>  	return virt_to_phys(cpu_addr);
>  }
> +EXPORT_SYMBOL(rproc_va_to_pa);
>  
>  /**
>   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc address
> @@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct rproc_subdev *subdev, bool crashed)
>  }
>  
>  /**
> + * rproc_rvdev_release() - release the existence of a rvdev
> + *
> + * @dev: the subdevice's dev
> + */
> +static void rproc_rvdev_release(struct device *dev)
> +{
> +	struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev, dev);
> +
> +	of_reserved_mem_device_release(dev);
> +
> +	kfree(rvdev);
> +}
> +
> +/**
>   * rproc_handle_vdev() - handle a vdev fw resource
>   * @rproc: the remote processor
>   * @rsc: the vring resource descriptor
> @@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc *rproc, struct fw_rsc_vdev *rsc,
>  	struct device *dev = &rproc->dev;
>  	struct rproc_vdev *rvdev;
>  	int i, ret;
> +	char name[16];
>  
>  	/* make sure resource isn't truncated */
>  	if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct fw_rsc_vdev_vring)
> @@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc *rproc, struct fw_rsc_vdev *rsc,
>  	rvdev->rproc = rproc;
>  	rvdev->index = rproc->nb_vdev++;
>  
> +	/* Initialise vdev subdevice */
> +	snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
> +	rvdev->dev.parent = rproc->dev.parent;
> +	rvdev->dev.release = rproc_rvdev_release;
> +	dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev->dev.parent), name);
> +	dev_set_drvdata(&rvdev->dev, rvdev);
> +	dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
> +
> +	ret = device_register(&rvdev->dev);
> +	if (ret)
> +		goto free_rvdev;
> +
>  	/* parse the vrings */
>  	for (i = 0; i < rsc->num_of_vrings; i++) {
>  		ret = rproc_parse_vring(rvdev, rsc, i);
> @@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc *rproc, struct fw_rsc_vdev *rsc,
>  	for (i--; i >= 0; i--)
>  		rproc_free_vring(&rvdev->vring[i]);
>  free_rvdev:
> -	kfree(rvdev);
> +	device_unregister(&rvdev->dev);
>  	return ret;
>  }
>  
> @@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
>  
>  	rproc_remove_subdev(rproc, &rvdev->subdev);
>  	list_del(&rvdev->node);
> -	kfree(rvdev);
> +	device_unregister(&rvdev->dev);
>  }
>  
>  /**
> diff --git a/drivers/remoteproc/remoteproc_internal.h b/drivers/remoteproc/remoteproc_internal.h
> index f6cad24..bfeacfd 100644
> --- a/drivers/remoteproc/remoteproc_internal.h
> +++ b/drivers/remoteproc/remoteproc_internal.h
> @@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const char *name, struct rproc *rproc,
>  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
>  
>  void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
> +phys_addr_t rproc_va_to_pa(void *cpu_addr);
>  int rproc_trigger_recovery(struct rproc *rproc);
>  
>  int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware *fw);
> diff --git a/drivers/remoteproc/remoteproc_virtio.c b/drivers/remoteproc/remoteproc_virtio.c
> index de21f62..9ee63c0 100644
> --- a/drivers/remoteproc/remoteproc_virtio.c
> +++ b/drivers/remoteproc/remoteproc_virtio.c
> @@ -17,7 +17,9 @@
>   * GNU General Public License for more details.
>   */
>  
> +#include <linux/dma-mapping.h>
>  #include <linux/export.h>
> +#include <linux/of_reserved_mem.h>
>  #include <linux/remoteproc.h>
>  #include <linux/virtio.h>
>  #include <linux/virtio_config.h>
> @@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct device *dev)
>  int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
>  {
>  	struct rproc *rproc = rvdev->rproc;
> -	struct device *dev = &rproc->dev;
> +	struct device *dev = &rvdev->dev;
>  	struct virtio_device *vdev = &rvdev->vdev;
> +	struct rproc_mem_entry *mem;
>  	int ret;
>  
> +	/* Try to find dedicated vdev buffer carveout */
> +	mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer", rvdev->index);
> +	if (mem) {
> +		phys_addr_t pa;
> +
> +		if (mem->of_resm_idx != -1) {
> +			struct device_node *np = rproc->dev.parent->of_node;
> +
> +			/* Associate reserved memory to vdev device */
> +			ret = of_reserved_mem_device_init_by_idx(dev, np,
> +								 mem->of_resm_idx);
> +			if (ret) {
> +				dev_err(dev, "Can't associate reserved memory\n");
> +				goto out;
> +			}
> +		} else {
> +			if (mem->va) {
> +				dev_warn(dev, "vdev %d buffer already mapped\n",
> +					 rvdev->index);
> +				pa = rproc_va_to_pa(mem->va);
> +			} else {
> +				/* Use dma address as carveout no memmapped yet */
> +				pa = (phys_addr_t)mem->dma;
> +			}
> +
> +			/* Associate vdev buffer memory pool to vdev subdev */
> +			ret = dmam_declare_coherent_memory(dev, pa,
> +							   mem->da,
> +							   mem->len,
> +							   DMA_MEMORY_EXCLUSIVE);

Is it not possible to associate the dma_mem with vdev->dev directly,
instead of injecting yet another device in-between the remoteproc
platform device and the virtio device?

That way the resulting allocation in e.g. virtio_rpmsg would be on the
virtio device itself, not its parent.

Regards,
Bjorn

> +			if (ret < 0) {
> +				dev_err(dev, "Failed to associate buffer\n");
> +				goto out;
> +			}
> +		}
> +	}
> +
>  	vdev->id.device	= id,
>  	vdev->config = &rproc_virtio_config_ops,
>  	vdev->dev.parent = dev;
> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> index 6b3a234..2921dd2 100644
> --- a/include/linux/remoteproc.h
> +++ b/include/linux/remoteproc.h
> @@ -547,6 +547,7 @@ struct rproc_vdev {
>  	struct kref refcount;
>  
>  	struct rproc_subdev subdev;
> +	struct device dev;
>  
>  	unsigned int id;
>  	struct list_head node;
> -- 
> 1.9.1
> 

^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-10-10  5:32   ` Bjorn Andersson
@ 2018-10-10 18:58     ` Loic PALLARDY
  2018-10-15  6:40       ` Bjorn Andersson
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-10 18:58 UTC (permalink / raw)
  To: Bjorn Andersson
  Cc: ohad, linux-remoteproc, linux-kernel, Arnaud POULIQUEN,
	benjamin.gaignard, s-anna



> -----Original Message-----
> From: Bjorn Andersson [mailto:bjorn.andersson@linaro.org]
> Sent: mercredi 10 octobre 2018 07:32
> To: Loic PALLARDY <loic.pallardy@st.com>
> Cc: ohad@wizery.com; linux-remoteproc@vger.kernel.org; linux-
> kernel@vger.kernel.org; Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> benjamin.gaignard@linaro.org; s-anna@ti.com
> Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on
> centralized carveout allocator
> 
> On Fri 27 Jul 06:14 PDT 2018, Loic Pallardy wrote:
> >  int rproc_fw_sanity_check(struct rproc *rproc, const struct firmware *fw)
> > diff --git a/drivers/remoteproc/remoteproc_virtio.c
> b/drivers/remoteproc/remoteproc_virtio.c
> [..]
> > @@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct
> virtio_device *vdev,
> >  	rvring->vq = vq;
> >  	vq->priv = rvring;
> >
> > +	/* Update vring in resource table */
> > +	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > +	rsc->vring[id].da = mem->da;
> > +
> 
> This would now happen after we've started the remoteproc. Don't we need
> to do this in-between allocating the carveouts and booting the
> remoteproc?

Yes da is updated after coprocessor boot, but before vdev status in resource table is set to DRIVER_OK and kick.
Coprocessor should not parse this resource before as vrings not initialized yet.
If coprocessor needs to get some information about vring carveout at boot time, carveout resources named vdev"x"vring"y" should be added to firmware resource table.
In that case information will be filled before coprocessor boot.

Regards,
Loic
> 
> Regards,
> Bjorn

^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-10-10  5:58   ` Bjorn Andersson
@ 2018-10-10 19:17     ` Loic PALLARDY
  2018-10-24  1:27       ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-10 19:17 UTC (permalink / raw)
  To: Bjorn Andersson
  Cc: ohad, linux-remoteproc, linux-kernel, Arnaud POULIQUEN,
	benjamin.gaignard, s-anna



> -----Original Message-----
> From: Bjorn Andersson [mailto:bjorn.andersson@linaro.org]
> Sent: mercredi 10 octobre 2018 07:58
> To: Loic PALLARDY <loic.pallardy@st.com>
> Cc: ohad@wizery.com; linux-remoteproc@vger.kernel.org; linux-
> kernel@vger.kernel.org; Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> benjamin.gaignard@linaro.org; s-anna@ti.com
> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
> specific dma memory pool
> 
> On Fri 27 Jul 06:14 PDT 2018, Loic Pallardy wrote:
> 
> > This patch creates a dedicated vdev subdevice for each vdev declared
> > in firmware resource table and associates carveout named "vdev%dbuffer"
> > (with %d vdev index in resource table) if any as dma coherent memory
> pool.
> >
> > Then vdev subdevice is used as parent for virtio device.
> >
> > Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > ---
> >  drivers/remoteproc/remoteproc_core.c     | 35
> +++++++++++++++++++++++---
> >  drivers/remoteproc/remoteproc_internal.h |  1 +
> >  drivers/remoteproc/remoteproc_virtio.c   | 42
> +++++++++++++++++++++++++++++++-
> >  include/linux/remoteproc.h               |  1 +
> >  4 files changed, 75 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/remoteproc/remoteproc_core.c
> b/drivers/remoteproc/remoteproc_core.c
> > index 4edc6f0..adcc66e 100644
> > --- a/drivers/remoteproc/remoteproc_core.c
> > +++ b/drivers/remoteproc/remoteproc_core.c
> > @@ -39,6 +39,7 @@
> >  #include <linux/idr.h>
> >  #include <linux/elf.h>
> >  #include <linux/crc32.h>
> > +#include <linux/of_reserved_mem.h>
> >  #include <linux/virtio_ids.h>
> >  #include <linux/virtio_ring.h>
> >  #include <asm/byteorder.h>
> > @@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc
> *rproc)
> >  	iommu_domain_free(domain);
> >  }
> >
> > -static phys_addr_t rproc_va_to_pa(void *cpu_addr)
> > +phys_addr_t rproc_va_to_pa(void *cpu_addr)
> >  {
> >  	/*
> >  	 * Return physical address according to virtual address location
> > @@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void
> *cpu_addr)
> >  	WARN_ON(!virt_addr_valid(cpu_addr));
> >  	return virt_to_phys(cpu_addr);
> >  }
> > +EXPORT_SYMBOL(rproc_va_to_pa);
> >
> >  /**
> >   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc
> address
> > @@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct
> rproc_subdev *subdev, bool crashed)
> >  }
> >
> >  /**
> > + * rproc_rvdev_release() - release the existence of a rvdev
> > + *
> > + * @dev: the subdevice's dev
> > + */
> > +static void rproc_rvdev_release(struct device *dev)
> > +{
> > +	struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev,
> dev);
> > +
> > +	of_reserved_mem_device_release(dev);
> > +
> > +	kfree(rvdev);
> > +}
> > +
> > +/**
> >   * rproc_handle_vdev() - handle a vdev fw resource
> >   * @rproc: the remote processor
> >   * @rsc: the vring resource descriptor
> > @@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
> struct fw_rsc_vdev *rsc,
> >  	struct device *dev = &rproc->dev;
> >  	struct rproc_vdev *rvdev;
> >  	int i, ret;
> > +	char name[16];
> >
> >  	/* make sure resource isn't truncated */
> >  	if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct
> fw_rsc_vdev_vring)
> > @@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc *rproc,
> struct fw_rsc_vdev *rsc,
> >  	rvdev->rproc = rproc;
> >  	rvdev->index = rproc->nb_vdev++;
> >
> > +	/* Initialise vdev subdevice */
> > +	snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
> > +	rvdev->dev.parent = rproc->dev.parent;
> > +	rvdev->dev.release = rproc_rvdev_release;
> > +	dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev-
> >dev.parent), name);
> > +	dev_set_drvdata(&rvdev->dev, rvdev);
> > +	dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
> > +
> > +	ret = device_register(&rvdev->dev);
> > +	if (ret)
> > +		goto free_rvdev;
> > +
> >  	/* parse the vrings */
> >  	for (i = 0; i < rsc->num_of_vrings; i++) {
> >  		ret = rproc_parse_vring(rvdev, rsc, i);
> > @@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
> struct fw_rsc_vdev *rsc,
> >  	for (i--; i >= 0; i--)
> >  		rproc_free_vring(&rvdev->vring[i]);
> >  free_rvdev:
> > -	kfree(rvdev);
> > +	device_unregister(&rvdev->dev);
> >  	return ret;
> >  }
> >
> > @@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
> >
> >  	rproc_remove_subdev(rproc, &rvdev->subdev);
> >  	list_del(&rvdev->node);
> > -	kfree(rvdev);
> > +	device_unregister(&rvdev->dev);
> >  }
> >
> >  /**
> > diff --git a/drivers/remoteproc/remoteproc_internal.h
> b/drivers/remoteproc/remoteproc_internal.h
> > index f6cad24..bfeacfd 100644
> > --- a/drivers/remoteproc/remoteproc_internal.h
> > +++ b/drivers/remoteproc/remoteproc_internal.h
> > @@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const char
> *name, struct rproc *rproc,
> >  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
> >
> >  void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
> > +phys_addr_t rproc_va_to_pa(void *cpu_addr);
> >  int rproc_trigger_recovery(struct rproc *rproc);
> >
> >  int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware *fw);
> > diff --git a/drivers/remoteproc/remoteproc_virtio.c
> b/drivers/remoteproc/remoteproc_virtio.c
> > index de21f62..9ee63c0 100644
> > --- a/drivers/remoteproc/remoteproc_virtio.c
> > +++ b/drivers/remoteproc/remoteproc_virtio.c
> > @@ -17,7 +17,9 @@
> >   * GNU General Public License for more details.
> >   */
> >
> > +#include <linux/dma-mapping.h>
> >  #include <linux/export.h>
> > +#include <linux/of_reserved_mem.h>
> >  #include <linux/remoteproc.h>
> >  #include <linux/virtio.h>
> >  #include <linux/virtio_config.h>
> > @@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct device
> *dev)
> >  int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
> >  {
> >  	struct rproc *rproc = rvdev->rproc;
> > -	struct device *dev = &rproc->dev;
> > +	struct device *dev = &rvdev->dev;
> >  	struct virtio_device *vdev = &rvdev->vdev;
> > +	struct rproc_mem_entry *mem;
> >  	int ret;
> >
> > +	/* Try to find dedicated vdev buffer carveout */
> > +	mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer",
> rvdev->index);
> > +	if (mem) {
> > +		phys_addr_t pa;
> > +
> > +		if (mem->of_resm_idx != -1) {
> > +			struct device_node *np = rproc->dev.parent-
> >of_node;
> > +
> > +			/* Associate reserved memory to vdev device */
> > +			ret = of_reserved_mem_device_init_by_idx(dev, np,
> > +								 mem-
> >of_resm_idx);
> > +			if (ret) {
> > +				dev_err(dev, "Can't associate reserved
> memory\n");
> > +				goto out;
> > +			}
> > +		} else {
> > +			if (mem->va) {
> > +				dev_warn(dev, "vdev %d buffer already
> mapped\n",
> > +					 rvdev->index);
> > +				pa = rproc_va_to_pa(mem->va);
> > +			} else {
> > +				/* Use dma address as carveout no
> memmapped yet */
> > +				pa = (phys_addr_t)mem->dma;
> > +			}
> > +
> > +			/* Associate vdev buffer memory pool to vdev
> subdev */
> > +			ret = dmam_declare_coherent_memory(dev, pa,
> > +							   mem->da,
> > +							   mem->len,
> > +
> DMA_MEMORY_EXCLUSIVE);
> 
> Is it not possible to associate the dma_mem with vdev->dev directly,
> instead of injecting yet another device in-between the remoteproc
> platform device and the virtio device?

Thank you for the review.
I'll check how to use virtio device. An additional ops for memory registration may be needed at virtio level. 

> 
> That way the resulting allocation in e.g. virtio_rpmsg would be on the
> virtio device itself, not its parent.
Yes agree, it will simplify allocation/inheritance...

Regards,
Loic
> 
> Regards,
> Bjorn
> 
> > +			if (ret < 0) {
> > +				dev_err(dev, "Failed to associate buffer\n");
> > +				goto out;
> > +			}
> > +		}
> > +	}
> > +
> >  	vdev->id.device	= id,
> >  	vdev->config = &rproc_virtio_config_ops,
> >  	vdev->dev.parent = dev;
> > diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> > index 6b3a234..2921dd2 100644
> > --- a/include/linux/remoteproc.h
> > +++ b/include/linux/remoteproc.h
> > @@ -547,6 +547,7 @@ struct rproc_vdev {
> >  	struct kref refcount;
> >
> >  	struct rproc_subdev subdev;
> > +	struct device dev;
> >
> >  	unsigned int id;
> >  	struct list_head node;
> > --
> > 1.9.1
> >

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-10-10 18:58     ` Loic PALLARDY
@ 2018-10-15  6:40       ` Bjorn Andersson
  2018-10-23 23:24         ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Bjorn Andersson @ 2018-10-15  6:40 UTC (permalink / raw)
  To: Loic PALLARDY
  Cc: ohad, linux-remoteproc, linux-kernel, Arnaud POULIQUEN,
	benjamin.gaignard, s-anna

On Wed 10 Oct 11:58 PDT 2018, Loic PALLARDY wrote:

> 
> 
> > -----Original Message-----
> > From: Bjorn Andersson [mailto:bjorn.andersson@linaro.org]
> > Sent: mercredi 10 octobre 2018 07:32
> > To: Loic PALLARDY <loic.pallardy@st.com>
> > Cc: ohad@wizery.com; linux-remoteproc@vger.kernel.org; linux-
> > kernel@vger.kernel.org; Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> > benjamin.gaignard@linaro.org; s-anna@ti.com
> > Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on
> > centralized carveout allocator
> > 
> > On Fri 27 Jul 06:14 PDT 2018, Loic Pallardy wrote:
> > >  int rproc_fw_sanity_check(struct rproc *rproc, const struct firmware *fw)
> > > diff --git a/drivers/remoteproc/remoteproc_virtio.c
> > b/drivers/remoteproc/remoteproc_virtio.c
> > [..]
> > > @@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct
> > virtio_device *vdev,
> > >  	rvring->vq = vq;
> > >  	vq->priv = rvring;
> > >
> > > +	/* Update vring in resource table */
> > > +	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > > +	rsc->vring[id].da = mem->da;
> > > +
> > 
> > This would now happen after we've started the remoteproc. Don't we need
> > to do this in-between allocating the carveouts and booting the
> > remoteproc?
> 
> Yes da is updated after coprocessor boot, but before vdev status in resource table is set to DRIVER_OK and kick.
> Coprocessor should not parse this resource before as vrings not initialized yet.
> If coprocessor needs to get some information about vring carveout at boot time, carveout resources named vdev"x"vring"y" should be added to firmware resource table.
> In that case information will be filled before coprocessor boot.
> 

Makes sense, thanks for clarifying. Applied.

Regards,
Bjorn

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 00/17] remoteproc: add fixed memory region support
  2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
                   ` (17 preceding siblings ...)
  2018-09-21  6:04 ` [PATCH v4 00/17] remoteproc: add fixed memory region support Anup Patel
@ 2018-10-23 16:42 ` Suman Anna
  18 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-23 16:42 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

Hi Bjorn,

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> The aim of the series is to implement carveout memory management as
> discussed during OpenAMP weekly call and defined in proposed document [1]
> 
> This first series focus only on adding support of the different types of
> carveout memories (dynamic, fixed, platform driver depend...).
> 64bit resource table will be addressed in a next series.
> Cent
> [1]: http://openamp.github.io/docs/mca/coprocessor-memory-definition-v6.pdf

Some of the patches in this series are breaking TI platforms, and I have
found couple of generic bugs as well. You may want to reconsider some of
these patches for 4.20. Apologies for the late review, but please look
through the comments I will be posting on various patches through the
rest of the day.

regards
Suman

> 
> ---
> Changes since V3:
> - Rebase on top of Bjorn remoteproc next git (3d065621c921)
> - Apply on top of "remoteproc: replace "%p" with "%pK"" [2]
> - Lot of changes to centralize carveout allocation before rproc_start()
> - Remove prepare/unprepare ops and rely on parse_fw instead of
> - Add possibility to assign reserved memory region to vdev device
> - Adapt TI da8xx and keystone remoteproc driver
> 
> As V3 series, V4 series takes some assumptions for carveout names
> associated to vdev:
> - For vring: "vdev%xvring%x" with vdev index from resource table and vring index
>   in vdev.
> - For vdev buffer: "vdev%xbuffer" with vdev index from resource table
> 
> [2] https://patchwork.kernel.org/patch/10511599
> 
> Changes since V2:
> Reshuffle the series to: 
> - Take into account Bjorn's comments.
> - Add patch to check consistency between carveout resource request and IOMMU
>   support.
> - Introduce platform specific prepare and unprepare ops to enable HW like
>   clock, bus, regulator, memory region... before loading co-processor firmware.
> - Rely on memory carveout management for all remoteproc memory allocations.
> - Lookup pre-registered carveout by name first.
> - Create a subdevice for each vdev declared in firmware resource table that
>   will be used by virtio based driver to retrieve specific memory pool.
> 
> This series takes some assumptions for carveout names associated to vdev:
> - For vring: "vdev%xvring%x" with vdev index from resource table and vring index
>   in vdev.
> - For vdev buffer: "vdev%xbuffer" with vdev index from resource table
> 
> This will be changed in the future, adding names field in vdev resource in
> next resource table version.
> 
> 
> Changes since V1:
> - Minor corrections on first 7 patches (error management)
> - Add "memory device" support on the top of first 7 patches.
>   Goal is to answer use case reported during OpenAMP weekly discussion:
>   - "Be able to specify memory region for vring and buffer allocation, even
>     if no specific request defined in firmware resource table."
>   Patches offer the capability to create a "memory device" associated to a
>   carveout with a dedicated DMA memory pool. Different resource handlers are
>   modified to look-up for specific carveout by name. If match found and associated
>   "memory device" present, device is used instead of rproc platform device for
>   allocation.
> 
> 
> Loic Pallardy (17):
>   remoteproc: configure IOMMU only if device address requested
>   remoteproc: add rproc_va_to_pa function
>   remoteproc: add release ops in rproc_mem_entry struct
>   remoteproc: add name in rproc_mem_entry struct
>   remoteproc: add helper function to allocate and init rproc_mem_entry
>     struct
>   remoteproc: introduce rproc_add_carveout function
>   remoteproc: introduce rproc_find_carveout_by_name function
>   remoteproc: add alloc ops in rproc_mem_entry struct
>   remoteproc: add helper function to allocate rproc_mem_entry from
>     reserved memory
>   remoteproc: add helper function to check carveout device address
>   remoteproc: modify rproc_handle_carveout to support pre-registered
>     region
>   remoteproc: modify vring allocation to rely on centralized carveout
>     allocator
>   remoteproc: create vdev subdevice with specific dma memory pool
>   remoteproc: keystone: declare reserved memory region for vdev device
>   remoteproc: da8xx: declare reserved memory region for vdev device
>   remoteproc: st: add reserved memory support
>   rpmsg: virtio: allocate buffer from parent
> 
>  drivers/remoteproc/da8xx_remoteproc.c    |  38 ++
>  drivers/remoteproc/keystone_remoteproc.c |  53 ++-
>  drivers/remoteproc/remoteproc_core.c     | 572 +++++++++++++++++++++++++------
>  drivers/remoteproc/remoteproc_debugfs.c  |   1 +
>  drivers/remoteproc/remoteproc_internal.h |   3 +
>  drivers/remoteproc/remoteproc_virtio.c   |  56 ++-
>  drivers/remoteproc/st_remoteproc.c       |  96 +++++-
>  drivers/rpmsg/virtio_rpmsg_bus.c         |   6 +-
>  include/linux/remoteproc.h               |  35 +-
>  9 files changed, 728 insertions(+), 132 deletions(-)
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function
  2018-07-27 13:14 ` [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function Loic Pallardy
@ 2018-10-23 16:50   ` Suman Anna
  2018-10-23 19:51     ` Loic PALLARDY
  0 siblings, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-23 16:50 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

Hi Loic, Bjorn,

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> This new function translates CPU virtual address in
> CPU physical one according to virtual address location.
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>
> ---
>  drivers/remoteproc/remoteproc_core.c | 18 +++++++++++++++++-
>  1 file changed, 17 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index 437fabf..8e5fe1e 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -140,6 +140,22 @@ static void rproc_disable_iommu(struct rproc *rproc)
>  	iommu_domain_free(domain);
>  }
>  
> +static phys_addr_t rproc_va_to_pa(void *cpu_addr)
> +{
> +	/*
> +	 * Return physical address according to virtual address location
> +	 * - in vmalloc: if region ioremapped or defined as dma_alloc_coherent
> +	 * - in kernel: if region allocated in generic dma memory pool
> +	 */
> +	if (is_vmalloc_addr(cpu_addr)) {
> +		return page_to_phys(vmalloc_to_page(cpu_addr)) +
> +				    offset_in_page(cpu_addr);
> +	}
> +
> +	WARN_ON(!virt_addr_valid(cpu_addr));
> +	return virt_to_phys(cpu_addr);
> +}
> +
>  /**
>   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc address
>   * @rproc: handle of a remote processor
> @@ -711,7 +727,7 @@ static int rproc_handle_carveout(struct rproc *rproc,
>  	 * In this case, the device address and the physical address
>  	 * are the same.
>  	 */
> -	rsc->pa = dma;
> +	rsc->pa = (u32)rproc_va_to_pa(va);

While I agree with the direction here, we ought to add a check here
warning users if some address bits are getting lost as a result of the
typecast. Granted the issue may have been present previously with
dma_addr_t as well, but most platforms were using 32-bit dma addresses,
so this was kinda masked. There are ARMv7 platforms with LPAE enabled
allowing physical addresses > 32-bits.

In anycase, we definitely have a need for a v2 for the fw_rsc_carveout
structure to deal with 64-bit addresses.

regards
Suman

>  
>  	carveout->va = va;
>  	carveout->len = rsc->len;
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 03/17] remoteproc: add release ops in rproc_mem_entry struct
  2018-07-27 13:14 ` [PATCH v4 03/17] remoteproc: add release ops in rproc_mem_entry struct Loic Pallardy
@ 2018-10-23 16:53   ` Suman Anna
  2018-10-23 20:48     ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-23 16:53 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> Memory entry could be allocated in different ways (ioremap,
> dma_alloc_coherent, internal RAM allocator...).
> This patch introduces a release ops in rproc_mem_entry structure
> to associate dedicated release mechanism to each memory entry descriptor
> in order to keep remoteproc core generic.
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>

Acked-by: Suman Anna <s-anna@ti.com>

> ---
>  drivers/remoteproc/remoteproc_core.c | 23 +++++++++++++++++++++--
>  include/linux/remoteproc.h           |  5 ++++-
>  2 files changed, 25 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index 8e5fe1e..24fe11f 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -600,6 +600,24 @@ static int rproc_handle_devmem(struct rproc *rproc, struct fw_rsc_devmem *rsc,
>  }
>  
>  /**
> + * rproc_release_carveout() - release acquired carveout
> + * @rproc: rproc handle
> + * @mem: the memory entry to release
> + *
> + * This function releases specified memory entry @mem allocated via
> + * dma_alloc_coherent() function by @rproc.
> + */
> +static int rproc_release_carveout(struct rproc *rproc,
> +				  struct rproc_mem_entry *mem)
> +{
> +	struct device *dev = &rproc->dev;
> +
> +	/* clean up carveout allocations */
> +	dma_free_coherent(dev->parent, mem->len, mem->va, mem->dma);
> +	return 0;
> +}
> +
> +/**
>   * rproc_handle_carveout() - handle phys contig memory allocation requests
>   * @rproc: rproc handle
>   * @rsc: the resource entry
> @@ -733,6 +751,7 @@ static int rproc_handle_carveout(struct rproc *rproc,
>  	carveout->len = rsc->len;
>  	carveout->dma = dma;
>  	carveout->da = rsc->da;
> +	carveout->release = rproc_release_carveout;
>  
>  	list_add_tail(&carveout->node, &rproc->carveouts);
>  
> @@ -920,8 +939,8 @@ static void rproc_resource_cleanup(struct rproc *rproc)
>  
>  	/* clean up carveout allocations */
>  	list_for_each_entry_safe(entry, tmp, &rproc->carveouts, node) {
> -		dma_free_coherent(dev->parent, entry->len, entry->va,
> -				  entry->dma);
> +		if (entry->release)
> +			entry->release(rproc, entry);
>  		list_del(&entry->node);
>  		kfree(entry);
>  	}
> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> index e3c5d85..bef2e83 100644
> --- a/include/linux/remoteproc.h
> +++ b/include/linux/remoteproc.h
> @@ -305,12 +305,15 @@ struct fw_rsc_vdev {
>  	struct fw_rsc_vdev_vring vring[0];
>  } __packed;
>  
> +struct rproc;
> +
>  /**
>   * struct rproc_mem_entry - memory entry descriptor
>   * @va:	virtual address
>   * @dma: dma address
>   * @len: length, in bytes
>   * @da: device address
> + * @release: release associated memory
>   * @priv: associated data
>   * @node: list node
>   */
> @@ -321,9 +324,9 @@ struct rproc_mem_entry {
>  	u32 da;
>  	void *priv;
>  	struct list_head node;
> +	int (*release)(struct rproc *rproc, struct rproc_mem_entry *mem);
>  };
>  
> -struct rproc;
>  struct firmware;
>  
>  /**
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 06/17] remoteproc: introduce rproc_add_carveout function
  2018-07-27 13:14 ` [PATCH v4 06/17] remoteproc: introduce rproc_add_carveout function Loic Pallardy
@ 2018-10-23 17:05   ` Suman Anna
  2018-10-23 19:48     ` Loic PALLARDY
  0 siblings, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-23 17:05 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

Hi Bjorn, Loic,

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> This patch introduces a new API to allow platform driver to register
> platform specific carveout regions.
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>

Hmm, I do not prefer that this function be exported. It adds no value,
and this adds an asymmetric API for remoteproc drivers to add a mem
entry to the internal remoteproc lists, while relying on the remoteproc
core to remove them. The carveout list maintenance is a remoteproc
internal scheme, and so should be maintained that way. Further more, on
the current series, all the existing calls are immediately after a
rproc_of_resm_mem_entry_init() or a rproc_mem_entry_init() function,
which means this could very well be folded into those individual functions.

regards
Suman

> ---
>  drivers/remoteproc/remoteproc_core.c | 16 +++++++++++++++-
>  include/linux/remoteproc.h           |  2 ++
>  2 files changed, 17 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index b76760e..fe6c4e4 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -747,7 +747,7 @@ static int rproc_handle_carveout(struct rproc *rproc,
>  	if (!carveout)
>  		goto free_carv;
>  
> -	list_add_tail(&carveout->node, &rproc->carveouts);
> +	rproc_add_carveout(rproc, carveout);
>  
>  	return 0;
>  
> @@ -761,6 +761,20 @@ static int rproc_handle_carveout(struct rproc *rproc,
>  }
>  
>  /**
> + * rproc_add_carveout() - register an allocated carveout region
> + * @rproc: rproc handle
> + * @mem: memory entry to register
> + *
> + * This function registers specified memory entry in @rproc carveouts list.
> + * Specified carveout should have been allocated before registering.
> + */
> +void rproc_add_carveout(struct rproc *rproc, struct rproc_mem_entry *mem)
> +{
> +	list_add_tail(&mem->node, &rproc->carveouts);
> +}
> +EXPORT_SYMBOL(rproc_add_carveout);
> +
> +/**
>   * rproc_mem_entry_init() - allocate and initialize rproc_mem_entry struct
>   * @dev: pointer on device struct
>   * @va: virtual address
> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> index 4bc961f..55f30fc 100644
> --- a/include/linux/remoteproc.h
> +++ b/include/linux/remoteproc.h
> @@ -558,6 +558,8 @@ struct rproc *rproc_alloc(struct device *dev, const char *name,
>  int rproc_del(struct rproc *rproc);
>  void rproc_free(struct rproc *rproc);
>  
> +void rproc_add_carveout(struct rproc *rproc, struct rproc_mem_entry *mem);
> +
>  struct rproc_mem_entry *
>  rproc_mem_entry_init(struct device *dev,
>  		     void *va, dma_addr_t dma, int len, u32 da,
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 04/17] remoteproc: add name in rproc_mem_entry struct
  2018-07-27 13:14 ` [PATCH v4 04/17] remoteproc: add name " Loic Pallardy
@ 2018-10-23 17:06   ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-23 17:06 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> Add name field in struct rproc_mem_entry.
> This new field will be used to match memory area
> requested in resource table with pre-registered carveout.
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>

Acked-by: Suman Anna <s-anna@ti.com>

> ---
>  drivers/remoteproc/remoteproc_core.c    | 1 +
>  drivers/remoteproc/remoteproc_debugfs.c | 1 +
>  include/linux/remoteproc.h              | 2 ++
>  3 files changed, 4 insertions(+)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index 24fe11f..d7e3138 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -752,6 +752,7 @@ static int rproc_handle_carveout(struct rproc *rproc,
>  	carveout->dma = dma;
>  	carveout->da = rsc->da;
>  	carveout->release = rproc_release_carveout;
> +	strlcpy(carveout->name, rsc->name, sizeof(carveout->name));
>  
>  	list_add_tail(&carveout->node, &rproc->carveouts);
>  
> diff --git a/drivers/remoteproc/remoteproc_debugfs.c b/drivers/remoteproc/remoteproc_debugfs.c
> index a5c29f2..e90135c 100644
> --- a/drivers/remoteproc/remoteproc_debugfs.c
> +++ b/drivers/remoteproc/remoteproc_debugfs.c
> @@ -260,6 +260,7 @@ static int rproc_carveouts_show(struct seq_file *seq, void *p)
>  
>  	list_for_each_entry(carveout, &rproc->carveouts, node) {
>  		seq_puts(seq, "Carveout memory entry:\n");
> +		seq_printf(seq, "\tName: %s\n", carveout->name);
>  		seq_printf(seq, "\tVirtual address: %pK\n", carveout->va);
>  		seq_printf(seq, "\tDMA address: %pad\n", &carveout->dma);
>  		seq_printf(seq, "\tDevice address: 0x%x\n", carveout->da);
> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> index bef2e83..0e21098 100644
> --- a/include/linux/remoteproc.h
> +++ b/include/linux/remoteproc.h
> @@ -315,6 +315,7 @@ struct fw_rsc_vdev {
>   * @da: device address
>   * @release: release associated memory
>   * @priv: associated data
> + * @name: associated memory region name (optional)
>   * @node: list node
>   */
>  struct rproc_mem_entry {
> @@ -323,6 +324,7 @@ struct rproc_mem_entry {
>  	int len;
>  	u32 da;
>  	void *priv;
> +	char name[32];
>  	struct list_head node;
>  	int (*release)(struct rproc *rproc, struct rproc_mem_entry *mem);
>  };
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 01/17] remoteproc: configure IOMMU only if device address requested
  2018-07-27 13:14 ` [PATCH v4 01/17] remoteproc: configure IOMMU only if device address requested Loic Pallardy
@ 2018-10-23 17:25   ` Suman Anna
  2018-10-23 19:40     ` Loic PALLARDY
  0 siblings, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-23 17:25 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

Hi Loic,

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> If there is no IOMMU associate to remote processor device,
> remoteproc_core won't be able to satisfy device address requested
> in firmware resource table.
> Return an error as configuration won't be coherent.
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>

This patch is breaking my Davinci platforms. It is not really required
that you _should_ have IOMMUs when a valid DA is mentioned. Please see
the existing description (paras 4 and 5) on the fw_rsc_carveout
kerneldoc in remoteproc.h file.

We do have platforms where we have some internal sub-modules within the
remote processor sub-system that provides some linear
address-translation (most common case with 32-bit processors supporting
64-bit addresses). Also, we have some upcoming SoCs where we have an MMU
but is not programmable by Linux.

There is one comment there, but I don't think this is actually handled
in the current remoteproc core.
"If @da is set to
 * FW_RSC_ADDR_ANY, then the host will dynamically allocate it, and then
 * overwrite @da with the dynamically allocated address."

regards
Suman

> ---
>  drivers/remoteproc/remoteproc_core.c | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index 4cd1a8e..437fabf 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -657,7 +657,15 @@ static int rproc_handle_carveout(struct rproc *rproc,
>  	 * to use the iommu-based DMA API: we expect 'dma' to contain the
>  	 * physical address in this case.
>  	 */
> -	if (rproc->domain) {
> +
> +	if (rsc->da != FW_RSC_ADDR_ANY && !rproc->domain) {
> +		dev_err(dev->parent,
> +			"Bad carveout rsc configuration\n");
> +		ret = -ENOMEM;
> +		goto dma_free;
> +	}
> +
> +	if (rsc->da != FW_RSC_ADDR_ANY && rproc->domain) {
>  		mapping = kzalloc(sizeof(*mapping), GFP_KERNEL);
>  		if (!mapping) {
>  			ret = -ENOMEM;
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 05/17] remoteproc: add helper function to allocate and init rproc_mem_entry struct
  2018-07-27 13:14 ` [PATCH v4 05/17] remoteproc: add helper function to allocate and init " Loic Pallardy
@ 2018-10-23 19:24   ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-23 19:24 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> This patch introduces rproc_mem_entry_init helper function to
> simplify rproc_mem_entry structure allocation and filling by
> client.
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>

Reviewed-by: Suman Anna <s-anna@ti.com>

> ---
>  drivers/remoteproc/remoteproc_core.c | 65 +++++++++++++++++++++++++++---------
>  include/linux/remoteproc.h           |  6 ++++
>  2 files changed, 55 insertions(+), 16 deletions(-)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index d7e3138..b76760e 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -639,7 +639,7 @@ static int rproc_handle_carveout(struct rproc *rproc,
>  				 struct fw_rsc_carveout *rsc,
>  				 int offset, int avail)
>  {
> -	struct rproc_mem_entry *carveout, *mapping;
> +	struct rproc_mem_entry *carveout, *mapping = NULL;
>  	struct device *dev = &rproc->dev;
>  	dma_addr_t dma;
>  	void *va;
> @@ -659,16 +659,11 @@ static int rproc_handle_carveout(struct rproc *rproc,
>  	dev_dbg(dev, "carveout rsc: name: %s, da 0x%x, pa 0x%x, len 0x%x, flags 0x%x\n",
>  		rsc->name, rsc->da, rsc->pa, rsc->len, rsc->flags);
>  
> -	carveout = kzalloc(sizeof(*carveout), GFP_KERNEL);
> -	if (!carveout)
> -		return -ENOMEM;
> -
>  	va = dma_alloc_coherent(dev->parent, rsc->len, &dma, GFP_KERNEL);
>  	if (!va) {
>  		dev_err(dev->parent,
>  			"failed to allocate dma memory: len 0x%x\n", rsc->len);
> -		ret = -ENOMEM;
> -		goto free_carv;
> +		return -ENOMEM;
>  	}
>  
>  	dev_dbg(dev, "carveout va %pK, dma %pad, len 0x%x\n",
> @@ -747,27 +742,65 @@ static int rproc_handle_carveout(struct rproc *rproc,
>  	 */
>  	rsc->pa = (u32)rproc_va_to_pa(va);
>  
> -	carveout->va = va;
> -	carveout->len = rsc->len;
> -	carveout->dma = dma;
> -	carveout->da = rsc->da;
> -	carveout->release = rproc_release_carveout;
> -	strlcpy(carveout->name, rsc->name, sizeof(carveout->name));
> +	carveout = rproc_mem_entry_init(dev, va, dma, rsc->len, rsc->da,
> +					rproc_release_carveout, rsc->name);
> +	if (!carveout)
> +		goto free_carv;
>  
>  	list_add_tail(&carveout->node, &rproc->carveouts);
>  
>  	return 0;
>  
> +free_carv:
> +	kfree(carveout);
>  free_mapping:
>  	kfree(mapping);
>  dma_free:
>  	dma_free_coherent(dev->parent, rsc->len, va, dma);
> -free_carv:
> -	kfree(carveout);
>  	return ret;
>  }
>  
> -/*
> +/**
> + * rproc_mem_entry_init() - allocate and initialize rproc_mem_entry struct
> + * @dev: pointer on device struct
> + * @va: virtual address
> + * @dma: dma address
> + * @len: memory carveout length
> + * @da: device address
> + * @release: memory carveout function
> + * @name: carveout name
> + *
> + * This function allocates a rproc_mem_entry struct and fill it with parameters
> + * provided by client.
> + */
> +struct rproc_mem_entry *
> +rproc_mem_entry_init(struct device *dev,
> +		     void *va, dma_addr_t dma, int len, u32 da,
> +		     int (*release)(struct rproc *, struct rproc_mem_entry *),
> +		     const char *name, ...)
> +{
> +	struct rproc_mem_entry *mem;
> +	va_list args;
> +
> +	mem = kzalloc(sizeof(*mem), GFP_KERNEL);
> +	if (!mem)
> +		return mem;
> +
> +	mem->va = va;
> +	mem->dma = dma;
> +	mem->da = da;
> +	mem->len = len;
> +	mem->release = release;
> +
> +	va_start(args, name);
> +	vsnprintf(mem->name, sizeof(mem->name), name, args);
> +	va_end(args);
> +
> +	return mem;
> +}
> +EXPORT_SYMBOL(rproc_mem_entry_init);
> +
> +/**
>   * A lookup table for resource handlers. The indices are defined in
>   * enum fw_resource_type.
>   */
> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> index 0e21098..4bc961f 100644
> --- a/include/linux/remoteproc.h
> +++ b/include/linux/remoteproc.h
> @@ -558,6 +558,12 @@ struct rproc *rproc_alloc(struct device *dev, const char *name,
>  int rproc_del(struct rproc *rproc);
>  void rproc_free(struct rproc *rproc);
>  
> +struct rproc_mem_entry *
> +rproc_mem_entry_init(struct device *dev,
> +		     void *va, dma_addr_t dma, int len, u32 da,
> +		     int (*release)(struct rproc *, struct rproc_mem_entry *),
> +		     const char *name, ...);
> +
>  int rproc_boot(struct rproc *rproc);
>  void rproc_shutdown(struct rproc *rproc);
>  void rproc_report_crash(struct rproc *rproc, enum rproc_crash_type type);
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 07/17] remoteproc: introduce rproc_find_carveout_by_name function
  2018-07-27 13:14 ` [PATCH v4 07/17] remoteproc: introduce rproc_find_carveout_by_name function Loic Pallardy
@ 2018-10-23 19:28   ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-23 19:28 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

Hi Loic,

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> This patch provides a new function to find a carveout according
> to a name.
> If match found, this function returns a pointer on the corresponding
> carveout (rproc_mem_entry structure).
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> ---
>  drivers/remoteproc/remoteproc_core.c | 42 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 42 insertions(+)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index fe6c4e4..77b39ba 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -217,6 +217,48 @@ void *rproc_da_to_va(struct rproc *rproc, u64 da, int len)
>  }
>  EXPORT_SYMBOL(rproc_da_to_va);
>  
> +/**
> + * rproc_find_carveout_by_name() - lookup the carveout region by a name
> + * @rproc: handle of a remote processor
> + * @name,..: carveout name to find (standard printf format)
> + *
> + * Platform driver has the capability to register some pre-allacoted carveout

%s/pre-allacoted/pre-allocated/

> + * (physically contiguous memory regions) before rproc firmware loading and
> + * associated resource table analysis. These regions may be dedicated memory
> + * regions internal to the coprocessor or specified DDR region with specific

I guess this is the part that perhaps was being thought of for
representing internal memories as well for removing the da_to_va ops
right. Question is how to handle the different alias views for the same
address space. Anyway, that is perhaps a discussion for later given that
the current series does not remove the ops.

> + * attributes
> + *
> + * This function is a helper function with which we can go over the
> + * allocated carveouts and return associated region characteristics like
> + * coprocessor address, length or processor virtual address.
> + *
> + * Return: a valid pointer on carveout entry on success or NULL on failure.
> + */
> +struct rproc_mem_entry *
> +rproc_find_carveout_by_name(struct rproc *rproc, const char *name, ...)
> +{
> +	va_list args;
> +	char _name[32];
> +	struct rproc_mem_entry *carveout, *mem = NULL;
> +
> +	if (!name)
> +		return NULL;
> +
> +	va_start(args, name);
> +	vsnprintf(_name, sizeof(_name), name, args);
> +	va_end(args);
> +
> +	list_for_each_entry(carveout, &rproc->carveouts, node) {
> +		/* Compare carveout and requested names */
> +		if (!strcmp(carveout->name, _name)) {
> +			mem = carveout;

Bjorn had some comments on v3 about returning carveout directly here
instead of break, avoids the mem variable. I am ok either way.

regards
Suman

> +			break;
> +		}
> +	}
> +
> +	return mem;
> +}
> +
>  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
>  {
>  	struct rproc *rproc = rvdev->rproc;
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 01/17] remoteproc: configure IOMMU only if device address requested
  2018-10-23 17:25   ` Suman Anna
@ 2018-10-23 19:40     ` Loic PALLARDY
  2018-10-24  3:46       ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-23 19:40 UTC (permalink / raw)
  To: Suman Anna, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

Hi Suman,

> -----Original Message-----
> From: Suman Anna <s-anna@ti.com>
> Sent: mardi 23 octobre 2018 19:26
> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> ohad@wizery.com
> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> benjamin.gaignard@linaro.org
> Subject: Re: [PATCH v4 01/17] remoteproc: configure IOMMU only if device
> address requested
> 
> Hi Loic,
> 
> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> > If there is no IOMMU associate to remote processor device,
> > remoteproc_core won't be able to satisfy device address requested
> > in firmware resource table.
> > Return an error as configuration won't be coherent.
> >
> > Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> 
> This patch is breaking my Davinci platforms. It is not really required
> that you _should_ have IOMMUs when a valid DA is mentioned. Please see
> the existing description (paras 4 and 5) on the fw_rsc_carveout
> kerneldoc in remoteproc.h file.

Thanks for pointing this comment. Indeed sMMU is not mandatory, and at first sight I agree we should remove the restriction introduced by the patch.
Driver porting on the series should be done before adding this.
> 
> We do have platforms where we have some internal sub-modules within the
> remote processor sub-system that provides some linear
> address-translation (most common case with 32-bit processors supporting
> 64-bit addresses). Also, we have some upcoming SoCs where we have an
> MMU
> but is not programmable by Linux.
> 
> There is one comment there, but I don't think this is actually handled
> in the current remoteproc core.
> "If @da is set to
>  * FW_RSC_ADDR_ANY, then the host will dynamically allocate it, and then
>  * overwrite @da with the dynamically allocated address."
> 
I don't remember it was implemented like described.

I have remarks about the comment:
"* We will always use @da to negotiate the device addresses, even if it
 * isn't using an iommu. In that case, though, it will obviously contain
 * physical addresses."

When there is no sMMU, we can't consider that da contains a physical address because coprocessor can have its own memory map just because it is a 32bit processor accessing only a part of the memory and the main is 64bit one. The 2 processors won't see the internal memory at the same base address for example.

So what should we do when carveout allocated by host is not fitting with resource table request?
- put a warning and overwrite da address in the resource table?
- stop rproc probe as no match detected?

Later in the series, carveout allocation is changed. Resource table carveout are either linked with an existing carveout registered by driver or added to carveout list for allocations.
In the case you described, TI driver should first register the specific carveout regions thank to the helper.

Regards,
Loic

> regards
> Suman
> 
> > ---
> >  drivers/remoteproc/remoteproc_core.c | 10 +++++++++-
> >  1 file changed, 9 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/remoteproc/remoteproc_core.c
> b/drivers/remoteproc/remoteproc_core.c
> > index 4cd1a8e..437fabf 100644
> > --- a/drivers/remoteproc/remoteproc_core.c
> > +++ b/drivers/remoteproc/remoteproc_core.c
> > @@ -657,7 +657,15 @@ static int rproc_handle_carveout(struct rproc
> *rproc,
> >  	 * to use the iommu-based DMA API: we expect 'dma' to contain the
> >  	 * physical address in this case.
> >  	 */
> > -	if (rproc->domain) {
> > +
> > +	if (rsc->da != FW_RSC_ADDR_ANY && !rproc->domain) {
> > +		dev_err(dev->parent,
> > +			"Bad carveout rsc configuration\n");
> > +		ret = -ENOMEM;
> > +		goto dma_free;
> > +	}
> > +
> > +	if (rsc->da != FW_RSC_ADDR_ANY && rproc->domain) {
> >  		mapping = kzalloc(sizeof(*mapping), GFP_KERNEL);
> >  		if (!mapping) {
> >  			ret = -ENOMEM;
> >


^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 06/17] remoteproc: introduce rproc_add_carveout function
  2018-10-23 17:05   ` Suman Anna
@ 2018-10-23 19:48     ` Loic PALLARDY
  0 siblings, 0 replies; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-23 19:48 UTC (permalink / raw)
  To: Suman Anna, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard



> -----Original Message-----
> From: Suman Anna <s-anna@ti.com>
> Sent: mardi 23 octobre 2018 19:05
> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> ohad@wizery.com
> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> benjamin.gaignard@linaro.org
> Subject: Re: [PATCH v4 06/17] remoteproc: introduce rproc_add_carveout
> function
> 
> Hi Bjorn, Loic,
> 
> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> > This patch introduces a new API to allow platform driver to register
> > platform specific carveout regions.
> >
> > Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>
> 
> Hmm, I do not prefer that this function be exported. It adds no value,
> and this adds an asymmetric API for remoteproc drivers to add a mem
> entry to the internal remoteproc lists, while relying on the remoteproc
> core to remove them. The carveout list maintenance is a remoteproc
> internal scheme, and so should be maintained that way. Further more, on
> the current series, all the existing calls are immediately after a
> rproc_of_resm_mem_entry_init() or a rproc_mem_entry_init() function,
> which means this could very well be folded into those individual functions.
> 
I think it has been introduced in first series. Look that rproc_add_carveout() call could be integrated in helper functions now.
I can propose a patch to correct this.

Regards,
Loic
> regards
> Suman
> 
> > ---
> >  drivers/remoteproc/remoteproc_core.c | 16 +++++++++++++++-
> >  include/linux/remoteproc.h           |  2 ++
> >  2 files changed, 17 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/remoteproc/remoteproc_core.c
> b/drivers/remoteproc/remoteproc_core.c
> > index b76760e..fe6c4e4 100644
> > --- a/drivers/remoteproc/remoteproc_core.c
> > +++ b/drivers/remoteproc/remoteproc_core.c
> > @@ -747,7 +747,7 @@ static int rproc_handle_carveout(struct rproc
> *rproc,
> >  	if (!carveout)
> >  		goto free_carv;
> >
> > -	list_add_tail(&carveout->node, &rproc->carveouts);
> > +	rproc_add_carveout(rproc, carveout);
> >
> >  	return 0;
> >
> > @@ -761,6 +761,20 @@ static int rproc_handle_carveout(struct rproc
> *rproc,
> >  }
> >
> >  /**
> > + * rproc_add_carveout() - register an allocated carveout region
> > + * @rproc: rproc handle
> > + * @mem: memory entry to register
> > + *
> > + * This function registers specified memory entry in @rproc carveouts list.
> > + * Specified carveout should have been allocated before registering.
> > + */
> > +void rproc_add_carveout(struct rproc *rproc, struct rproc_mem_entry
> *mem)
> > +{
> > +	list_add_tail(&mem->node, &rproc->carveouts);
> > +}
> > +EXPORT_SYMBOL(rproc_add_carveout);
> > +
> > +/**
> >   * rproc_mem_entry_init() - allocate and initialize rproc_mem_entry struct
> >   * @dev: pointer on device struct
> >   * @va: virtual address
> > diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> > index 4bc961f..55f30fc 100644
> > --- a/include/linux/remoteproc.h
> > +++ b/include/linux/remoteproc.h
> > @@ -558,6 +558,8 @@ struct rproc *rproc_alloc(struct device *dev, const
> char *name,
> >  int rproc_del(struct rproc *rproc);
> >  void rproc_free(struct rproc *rproc);
> >
> > +void rproc_add_carveout(struct rproc *rproc, struct rproc_mem_entry
> *mem);
> > +
> >  struct rproc_mem_entry *
> >  rproc_mem_entry_init(struct device *dev,
> >  		     void *va, dma_addr_t dma, int len, u32 da,
> >


^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function
  2018-10-23 16:50   ` Suman Anna
@ 2018-10-23 19:51     ` Loic PALLARDY
  2018-10-24  3:19       ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-23 19:51 UTC (permalink / raw)
  To: Suman Anna, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

Hi Suman,

> -----Original Message-----
> From: Suman Anna <s-anna@ti.com>
> Sent: mardi 23 octobre 2018 18:51
> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> ohad@wizery.com
> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> benjamin.gaignard@linaro.org
> Subject: Re: [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function
> 
> Hi Loic, Bjorn,
> 
> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> > This new function translates CPU virtual address in
> > CPU physical one according to virtual address location.
> >
> > Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>
> > ---
> >  drivers/remoteproc/remoteproc_core.c | 18 +++++++++++++++++-
> >  1 file changed, 17 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/remoteproc/remoteproc_core.c
> b/drivers/remoteproc/remoteproc_core.c
> > index 437fabf..8e5fe1e 100644
> > --- a/drivers/remoteproc/remoteproc_core.c
> > +++ b/drivers/remoteproc/remoteproc_core.c
> > @@ -140,6 +140,22 @@ static void rproc_disable_iommu(struct rproc
> *rproc)
> >  	iommu_domain_free(domain);
> >  }
> >
> > +static phys_addr_t rproc_va_to_pa(void *cpu_addr)
> > +{
> > +	/*
> > +	 * Return physical address according to virtual address location
> > +	 * - in vmalloc: if region ioremapped or defined as
> dma_alloc_coherent
> > +	 * - in kernel: if region allocated in generic dma memory pool
> > +	 */
> > +	if (is_vmalloc_addr(cpu_addr)) {
> > +		return page_to_phys(vmalloc_to_page(cpu_addr)) +
> > +				    offset_in_page(cpu_addr);
> > +	}
> > +
> > +	WARN_ON(!virt_addr_valid(cpu_addr));
> > +	return virt_to_phys(cpu_addr);
> > +}
> > +
> >  /**
> >   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc
> address
> >   * @rproc: handle of a remote processor
> > @@ -711,7 +727,7 @@ static int rproc_handle_carveout(struct rproc
> *rproc,
> >  	 * In this case, the device address and the physical address
> >  	 * are the same.
> >  	 */
> > -	rsc->pa = dma;
> > +	rsc->pa = (u32)rproc_va_to_pa(va);
> 
> While I agree with the direction here, we ought to add a check here
> warning users if some address bits are getting lost as a result of the
> typecast. Granted the issue may have been present previously with
> dma_addr_t as well, but most platforms were using 32-bit dma addresses,
> so this was kinda masked. There are ARMv7 platforms with LPAE enabled
> allowing physical addresses > 32-bits.
> 
> In anycase, we definitely have a need for a v2 for the fw_rsc_carveout
> structure to deal with 64-bit addresses.
> 

Agree with you.
Assumption for this series was to keep resource table as it is. Resource table improvement is planned in a second step.
Regards,
Loic

> regards
> Suman
> 
> >
> >  	carveout->va = va;
> >  	carveout->len = rsc->len;
> >


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 03/17] remoteproc: add release ops in rproc_mem_entry struct
  2018-10-23 16:53   ` Suman Anna
@ 2018-10-23 20:48     ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-23 20:48 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

On 10/23/18 11:53 AM, Suman Anna wrote:
> On 7/27/18 8:14 AM, Loic Pallardy wrote:
>> Memory entry could be allocated in different ways (ioremap,
>> dma_alloc_coherent, internal RAM allocator...).
>> This patch introduces a release ops in rproc_mem_entry structure
>> to associate dedicated release mechanism to each memory entry descriptor
>> in order to keep remoteproc core generic.
>>
>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>> Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>
> 
> Acked-by: Suman Anna <s-anna@ti.com>
> 
>> ---
>>  drivers/remoteproc/remoteproc_core.c | 23 +++++++++++++++++++++--
>>  include/linux/remoteproc.h           |  5 ++++-
>>  2 files changed, 25 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
>> index 8e5fe1e..24fe11f 100644
>> --- a/drivers/remoteproc/remoteproc_core.c
>> +++ b/drivers/remoteproc/remoteproc_core.c
>> @@ -600,6 +600,24 @@ static int rproc_handle_devmem(struct rproc *rproc, struct fw_rsc_devmem *rsc,
>>  }
>>  
>>  /**
>> + * rproc_release_carveout() - release acquired carveout
>> + * @rproc: rproc handle
>> + * @mem: the memory entry to release
>> + *
>> + * This function releases specified memory entry @mem allocated via
>> + * dma_alloc_coherent() function by @rproc.
>> + */
>> +static int rproc_release_carveout(struct rproc *rproc,
>> +				  struct rproc_mem_entry *mem)
>> +{
>> +	struct device *dev = &rproc->dev;
>> +
>> +	/* clean up carveout allocations */
>> +	dma_free_coherent(dev->parent, mem->len, mem->va, mem->dma);
>> +	return 0;
>> +}
>> +
>> +/**
>>   * rproc_handle_carveout() - handle phys contig memory allocation requests
>>   * @rproc: rproc handle
>>   * @rsc: the resource entry
>> @@ -733,6 +751,7 @@ static int rproc_handle_carveout(struct rproc *rproc,
>>  	carveout->len = rsc->len;
>>  	carveout->dma = dma;
>>  	carveout->da = rsc->da;
>> +	carveout->release = rproc_release_carveout;
>>  
>>  	list_add_tail(&carveout->node, &rproc->carveouts);
>>  
>> @@ -920,8 +939,8 @@ static void rproc_resource_cleanup(struct rproc *rproc)
>>  
>>  	/* clean up carveout allocations */
>>  	list_for_each_entry_safe(entry, tmp, &rproc->carveouts, node) {
>> -		dma_free_coherent(dev->parent, entry->len, entry->va,
>> -				  entry->dma);
>> +		if (entry->release)
>> +			entry->release(rproc, entry);
>>  		list_del(&entry->node);
>>  		kfree(entry);
>>  	}
>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
>> index e3c5d85..bef2e83 100644
>> --- a/include/linux/remoteproc.h
>> +++ b/include/linux/remoteproc.h
>> @@ -305,12 +305,15 @@ struct fw_rsc_vdev {
>>  	struct fw_rsc_vdev_vring vring[0];
>>  } __packed;
>>  
>> +struct rproc;
>> +
>>  /**
>>   * struct rproc_mem_entry - memory entry descriptor
>>   * @va:	virtual address
>>   * @dma: dma address
>>   * @len: length, in bytes
>>   * @da: device address
>> + * @release: release associated memory

Missed this earlier, move this line after the @node.

regards
Suman

>>   * @priv: associated data
>>   * @node: list node
>>   */
>> @@ -321,9 +324,9 @@ struct rproc_mem_entry {
>>  	u32 da;
>>  	void *priv;
>>  	struct list_head node;
>> +	int (*release)(struct rproc *rproc, struct rproc_mem_entry *mem);
>>  };
>>  
>> -struct rproc;
>>  struct firmware;
>>  
>>  /**
>>
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 08/17] remoteproc: add alloc ops in rproc_mem_entry struct
  2018-07-27 13:14 ` [PATCH v4 08/17] remoteproc: add alloc ops in rproc_mem_entry struct Loic Pallardy
@ 2018-10-23 21:20   ` Suman Anna
  2018-10-24 16:00     ` Loic PALLARDY
  0 siblings, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-23 21:20 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

Hi Loic,

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> Memory entry could be allocated in different ways (ioremap,
> dma_alloc_coherent, internal RAM allocator...).
> This patch introduces an alloc ops in rproc_mem_entry structure
> to associate dedicated allocation mechanism to each memory entry
> descriptor in order to do remote core agnostic from memory allocators.
> 
> The introduction of this ops allows to perform allocation of all registered
> carveout at the same time, just before calling rproc_start().
> It simplifies and makes uniform carveout management whatever origin.

This patch is causing a kernel crash with trace entries. Please see
further below for the cause.

> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> ---
>  drivers/remoteproc/remoteproc_core.c | 261 ++++++++++++++++++++++-------------
>  include/linux/remoteproc.h           |   7 +
>  2 files changed, 175 insertions(+), 93 deletions(-)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index 77b39ba..2c51549 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -642,74 +642,31 @@ static int rproc_handle_devmem(struct rproc *rproc, struct fw_rsc_devmem *rsc,
>  }
>  
>  /**
> - * rproc_release_carveout() - release acquired carveout
> + * rproc_alloc_carveout() - allocated specified carveout
>   * @rproc: rproc handle
> - * @mem: the memory entry to release
> - *
> - * This function releases specified memory entry @mem allocated via
> - * dma_alloc_coherent() function by @rproc.
> - */
> -static int rproc_release_carveout(struct rproc *rproc,
> -				  struct rproc_mem_entry *mem)
> -{
> -	struct device *dev = &rproc->dev;
> -
> -	/* clean up carveout allocations */
> -	dma_free_coherent(dev->parent, mem->len, mem->va, mem->dma);
> -	return 0;
> -}
> -
> -/**
> - * rproc_handle_carveout() - handle phys contig memory allocation requests
> - * @rproc: rproc handle
> - * @rsc: the resource entry
> - * @avail: size of available data (for image validation)
> - *
> - * This function will handle firmware requests for allocation of physically
> - * contiguous memory regions.
> - *
> - * These request entries should come first in the firmware's resource table,
> - * as other firmware entries might request placing other data objects inside
> - * these memory regions (e.g. data/code segments, trace resource entries, ...).
> + * @mem: the memory entry to allocate
>   *
> - * Allocating memory this way helps utilizing the reserved physical memory
> - * (e.g. CMA) more efficiently, and also minimizes the number of TLB entries
> - * needed to map it (in case @rproc is using an IOMMU). Reducing the TLB
> - * pressure is important; it may have a substantial impact on performance.
> + * This function allocate specified memory entry @mem using
> + * dma_alloc_coherent() as default allocator
>   */
> -static int rproc_handle_carveout(struct rproc *rproc,
> -				 struct fw_rsc_carveout *rsc,
> -				 int offset, int avail)
> +static int rproc_alloc_carveout(struct rproc *rproc,
> +				struct rproc_mem_entry *mem)
>  {
> -	struct rproc_mem_entry *carveout, *mapping = NULL;
> +	struct rproc_mem_entry *mapping = NULL;
>  	struct device *dev = &rproc->dev;
>  	dma_addr_t dma;
>  	void *va;
>  	int ret;
>  
> -	if (sizeof(*rsc) > avail) {
> -		dev_err(dev, "carveout rsc is truncated\n");
> -		return -EINVAL;
> -	}
> -
> -	/* make sure reserved bytes are zeroes */
> -	if (rsc->reserved) {
> -		dev_err(dev, "carveout rsc has non zero reserved bytes\n");
> -		return -EINVAL;
> -	}
> -
> -	dev_dbg(dev, "carveout rsc: name: %s, da 0x%x, pa 0x%x, len 0x%x, flags 0x%x\n",
> -		rsc->name, rsc->da, rsc->pa, rsc->len, rsc->flags);
> -
> -	va = dma_alloc_coherent(dev->parent, rsc->len, &dma, GFP_KERNEL);
> +	va = dma_alloc_coherent(dev->parent, mem->len, &dma, GFP_KERNEL);
>  	if (!va) {
>  		dev_err(dev->parent,
> -			"failed to allocate dma memory: len 0x%x\n", rsc->len);
> +			"failed to allocate dma memory: len 0x%x\n", mem->len);
>  		return -ENOMEM;
>  	}
>  
>  	dev_dbg(dev, "carveout va %pK, dma %pad, len 0x%x\n",
> -		va, &dma, rsc->len);
> +		va, &dma, mem->len);
>  
>  	/*
>  	 * Ok, this is non-standard.
> @@ -729,22 +686,22 @@ static int rproc_handle_carveout(struct rproc *rproc,
>  	 * physical address in this case.
>  	 */
>  
> -	if (rsc->da != FW_RSC_ADDR_ANY && !rproc->domain) {
> -		dev_err(dev->parent,
> -			"Bad carveout rsc configuration\n");
> -		ret = -ENOMEM;
> -		goto dma_free;
> -	}
> +	if (mem->da != FW_RSC_ADDR_ANY) {
> +		if (!rproc->domain) {
> +			dev_err(dev->parent,
> +				"Bad carveout rsc configuration\n");
> +			ret = -ENOMEM;
> +			goto dma_free;
> +		}

Same comment from Patch 1.

>  
> -	if (rsc->da != FW_RSC_ADDR_ANY && rproc->domain) {
>  		mapping = kzalloc(sizeof(*mapping), GFP_KERNEL);
>  		if (!mapping) {
>  			ret = -ENOMEM;
>  			goto dma_free;
>  		}
>  
> -		ret = iommu_map(rproc->domain, rsc->da, dma, rsc->len,
> -				rsc->flags);
> +		ret = iommu_map(rproc->domain, mem->da, dma, mem->len,
> +				mem->flags);
>  		if (ret) {
>  			dev_err(dev, "iommu_map failed: %d\n", ret);
>  			goto free_mapping;
> @@ -757,52 +714,102 @@ static int rproc_handle_carveout(struct rproc *rproc,
>  		 * We can't trust the remote processor not to change the
>  		 * resource table, so we must maintain this info independently.
>  		 */
> -		mapping->da = rsc->da;
> -		mapping->len = rsc->len;
> +		mapping->da = mem->da;
> +		mapping->len = mem->len;
>  		list_add_tail(&mapping->node, &rproc->mappings);
>  
>  		dev_dbg(dev, "carveout mapped 0x%x to %pad\n",
> -			rsc->da, &dma);
> +			mem->da, &dma);
> +	} else {
> +		mem->da = (u32)dma;

Hmm, what was the purpose of this? So, this appears to be handling the
missing implementation for the comment in the fw_rsc_carveout about
FW_RSC_ADDR_ANY.

>  	}
>  
> -	/*
> -	 * Some remote processors might need to know the pa
> -	 * even though they are behind an IOMMU. E.g., OMAP4's
> -	 * remote M3 processor needs this so it can control
> -	 * on-chip hardware accelerators that are not behind
> -	 * the IOMMU, and therefor must know the pa.
> -	 *
> -	 * Generally we don't want to expose physical addresses
> -	 * if we don't have to (remote processors are generally
> -	 * _not_ trusted), so we might want to do this only for
> -	 * remote processor that _must_ have this (e.g. OMAP4's
> -	 * dual M3 subsystem).
> -	 *
> -	 * Non-IOMMU processors might also want to have this info.
> -	 * In this case, the device address and the physical address
> -	 * are the same.
> -	 */
> -	rsc->pa = (u32)rproc_va_to_pa(va);
> -
> -	carveout = rproc_mem_entry_init(dev, va, dma, rsc->len, rsc->da,
> -					rproc_release_carveout, rsc->name);
> -	if (!carveout)
> -		goto free_carv;
> -
> -	rproc_add_carveout(rproc, carveout);
> +	mem->dma = (u32)dma;

We don't need the typecast, mem->dma is already of type dma_addr_t. Same
comment above on the else part as well.

> +	mem->va = va;
>  
>  	return 0;
>  
> -free_carv:
> -	kfree(carveout);
>  free_mapping:
>  	kfree(mapping);
>  dma_free:
> -	dma_free_coherent(dev->parent, rsc->len, va, dma);
> +	dma_free_coherent(dev->parent, mem->len, va, dma);
>  	return ret;
>  }
>  
>  /**
> + * rproc_release_carveout() - release acquired carveout
> + * @rproc: rproc handle
> + * @mem: the memory entry to release
> + *
> + * This function releases specified memory entry @mem allocated via
> + * rproc_alloc_carveout() function by @rproc.
> + */
> +static int rproc_release_carveout(struct rproc *rproc,
> +				  struct rproc_mem_entry *mem)
> +{
> +	struct device *dev = &rproc->dev;
> +
> +	/* clean up carveout allocations */
> +	dma_free_coherent(dev->parent, mem->len, mem->va, mem->dma);
> +	return 0;
> +}
> +
> +/**
> + * rproc_handle_carveout() - handle phys contig memory allocation requests
> + * @rproc: rproc handle
> + * @rsc: the resource entry
> + * @avail: size of available data (for image validation)
> + *
> + * This function will handle firmware requests for allocation of physically
> + * contiguous memory regions.
> + *
> + * These request entries should come first in the firmware's resource table,
> + * as other firmware entries might request placing other data objects inside
> + * these memory regions (e.g. data/code segments, trace resource entries, ...).
> + *
> + * Allocating memory this way helps utilizing the reserved physical memory
> + * (e.g. CMA) more efficiently, and also minimizes the number of TLB entries
> + * needed to map it (in case @rproc is using an IOMMU). Reducing the TLB
> + * pressure is important; it may have a substantial impact on performance.
> + */
> +static int rproc_handle_carveout(struct rproc *rproc,
> +				 struct fw_rsc_carveout *rsc,
> +				 int offset, int avail)
> +{
> +	struct rproc_mem_entry *carveout;
> +	struct device *dev = &rproc->dev;
> +
> +	if (sizeof(*rsc) > avail) {
> +		dev_err(dev, "carveout rsc is truncated\n");
> +		return -EINVAL;
> +	}
> +
> +	/* make sure reserved bytes are zeroes */
> +	if (rsc->reserved) {
> +		dev_err(dev, "carveout rsc has non zero reserved bytes\n");
> +		return -EINVAL;
> +	}
> +
> +	dev_dbg(dev, "carveout rsc: name: %s, da 0x%x, pa 0x%x, len 0x%x, flags 0x%x\n",
> +		rsc->name, rsc->da, rsc->pa, rsc->len, rsc->flags);
> +
> +	/* Register carveout in in list */
> +	carveout = rproc_mem_entry_init(dev, 0, 0, rsc->len, rsc->da,
> +					rproc_alloc_carveout,
> +					rproc_release_carveout, rsc->name);
> +	if (!carveout) {
> +		dev_err(dev, "Can't allocate memory entry structure\n");
> +		return -ENOMEM;
> +	}
> +
> +	carveout->flags = rsc->flags;
> +	carveout->rsc_offset = offset;
> +	rproc_add_carveout(rproc, carveout);

Once we get rid of rproc_add_carveout, the list addition will mostly be
handled in rproc_mem_entry_init itself.

> +
> +	return 0;
> +}
> +
> +/**
>   * rproc_add_carveout() - register an allocated carveout region
>   * @rproc: rproc handle
>   * @mem: memory entry to register
> @@ -832,6 +839,7 @@ void rproc_add_carveout(struct rproc *rproc, struct rproc_mem_entry *mem)
>  struct rproc_mem_entry *
>  rproc_mem_entry_init(struct device *dev,
>  		     void *va, dma_addr_t dma, int len, u32 da,
> +		     int (*alloc)(struct rproc *, struct rproc_mem_entry *),
>  		     int (*release)(struct rproc *, struct rproc_mem_entry *),
>  		     const char *name, ...)
>  {
> @@ -846,7 +854,9 @@ struct rproc_mem_entry *
>  	mem->dma = dma;
>  	mem->da = da;
>  	mem->len = len;
> +	mem->alloc = alloc;
>  	mem->release = release;
> +	mem->rsc_offset = FW_RSC_ADDR_ANY;
>  
>  	va_start(args, name);
>  	vsnprintf(mem->name, sizeof(mem->name), name, args);
> @@ -978,6 +988,63 @@ static void rproc_unprepare_subdevices(struct rproc *rproc)
>  }
>  
>  /**
> + * rproc_alloc_registered_carveouts() - allocate all carveouts registered
> + * in the list
> + * @rproc: the remote processor handle
> + *
> + * This function parses registered carveout list, performs allocation
> + * if alloc() ops registered and updates resource table information
> + * if rsc_offset set.
> + *
> + * Return: 0 on success
> + */
> +static int rproc_alloc_registered_carveouts(struct rproc *rproc)
> +{
> +	struct rproc_mem_entry *entry, *tmp;
> +	struct fw_rsc_carveout *rsc;
> +	struct device *dev = &rproc->dev;
> +	int ret;
> +
> +	list_for_each_entry_safe(entry, tmp, &rproc->carveouts, node) {
> +		if (entry->alloc) {
> +			ret = entry->alloc(rproc, entry);
> +			if (ret) {
> +				dev_err(dev, "Unable to allocate carveout %s: %d\n",
> +					entry->name, ret);
> +				return -ENOMEM;
> +			}
> +		}
> +
> +		if (entry->rsc_offset != FW_RSC_ADDR_ANY) {
> +			/* update resource table */
> +			rsc = (void *)rproc->table_ptr + entry->rsc_offset;
> +
> +			/*
> +			 * Some remote processors might need to know the pa
> +			 * even though they are behind an IOMMU. E.g., OMAP4's
> +			 * remote M3 processor needs this so it can control
> +			 * on-chip hardware accelerators that are not behind
> +			 * the IOMMU, and therefor must know the pa.
> +			 *
> +			 * Generally we don't want to expose physical addresses
> +			 * if we don't have to (remote processors are generally
> +			 * _not_ trusted), so we might want to do this only for
> +			 * remote processor that _must_ have this (e.g. OMAP4's
> +			 * dual M3 subsystem).
> +			 *
> +			 * Non-IOMMU processors might also want to have this info.
> +			 * In this case, the device address and the physical address
> +			 * are the same.
> +			 */
> +			if (entry->va)
> +				rsc->pa = (u32)rproc_va_to_pa(entry->va);
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +/**
>   * rproc_coredump_cleanup() - clean up dump_segments list
>   * @rproc: the remote processor handle
>   */
> @@ -1148,6 +1215,14 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
>  		goto clean_up_resources;
>  	}
>  
> +	/* Allocate carveout resources associated to rproc */
> +	ret = rproc_alloc_registered_carveouts(rproc);
> +	if (ret) {
> +		dev_err(dev, "Failed to allocate associated carveouts: %d\n",
> +			ret);
> +		goto clean_up_resources;
> +	}

This is causing an issue with RSC_TRACE on where the trace region on the
remote processor is actually backed by a DDR carveout address. The
allocations are now being done after processing the resources from the
rproc_loading_handlers, which causes the RSC_TRACE to be configured with
an incorrect kernel va, and accessing it through debugfs then results in
a kernel crash.

regards
Suman

> +
>  	ret = rproc_start(rproc, fw);
>  	if (ret)
>  		goto clean_up_resources;
> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> index 55f30fc..ea95b04 100644
> --- a/include/linux/remoteproc.h
> +++ b/include/linux/remoteproc.h
> @@ -317,6 +317,9 @@ struct fw_rsc_vdev {
>   * @priv: associated data
>   * @name: associated memory region name (optional)
>   * @node: list node
> + * @rsc_offset: offset in resource table
> + * @flags: iommu protection flags
> + * @alloc: specific memory allocator function
>   */
>  struct rproc_mem_entry {
>  	void *va;
> @@ -326,6 +329,9 @@ struct rproc_mem_entry {
>  	void *priv;
>  	char name[32];
>  	struct list_head node;
> +	u32 rsc_offset;
> +	u32 flags;
> +	int (*alloc)(struct rproc *rproc, struct rproc_mem_entry *mem);
>  	int (*release)(struct rproc *rproc, struct rproc_mem_entry *mem);
>  };
>  
> @@ -563,6 +569,7 @@ struct rproc *rproc_alloc(struct device *dev, const char *name,
>  struct rproc_mem_entry *
>  rproc_mem_entry_init(struct device *dev,
>  		     void *va, dma_addr_t dma, int len, u32 da,
> +		     int (*alloc)(struct rproc *, struct rproc_mem_entry *),
>  		     int (*release)(struct rproc *, struct rproc_mem_entry *),
>  		     const char *name, ...);
>  
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 10/17] remoteproc: add helper function to check carveout device address
  2018-07-27 13:14 ` [PATCH v4 10/17] remoteproc: add helper function to check carveout device address Loic Pallardy
@ 2018-10-23 22:14   ` Suman Anna
  2018-10-24 15:24     ` Loic PALLARDY
  0 siblings, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-23 22:14 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

Hi Loic,

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> This patch introduces a function to verify that a specified carveout
> is fitting request device address and associated length
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> ---
>  drivers/remoteproc/remoteproc_core.c | 47 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 47 insertions(+)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index 1e0fe3e..5dd5edf 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -259,6 +259,53 @@ struct rproc_mem_entry *
>  	return mem;
>  }
>  
> +/**
> + * rproc_check_carveout_da() - Check specified carveout da configuration
> + * @rproc: handle of a remote processor
> + * @mem: pointer on carveout to check
> + * @da: area device address
> + * @len: associated area size
> + *
> + * This function is a helper function to verify requested device area (couple
> + * da, len) is part of specified carevout.

%s/carevout/carveout/

> + *
> + * Return: 0 if carveout matchs request else -ENOMEM

%s/matchs/matches/

> + */
> +int rproc_check_carveout_da(struct rproc *rproc, struct rproc_mem_entry *mem,

static int since this seems to be only a local function.

> +			    u32 da, u32 len)
> +{
> +	struct device *dev = &rproc->dev;
> +	int delta = 0;
> +
> +	/* Check requested resource length */
> +	if (len > mem->len) {
> +		dev_err(dev, "Registered carveout doesn't fit len request\n");
> +		return -ENOMEM;

ENOMEM not typically used for these kind of errors, you were probably
inclined to used this since it is dealing with memory.

> +	}
> +

Both the below codepaths are exercised only when da is not
FW_RSC_ADDR_ANY, and you are returning 0 otherwise (which is the case of
matches as per your description above). Is that what you really want -
should it be an error

> +	if (da != FW_RSC_ADDR_ANY && mem->da == FW_RSC_ADDR_ANY) {
> +		/* Update existing carveout da */
> +		mem->da = da;

Where would you need to update this?

regards
Suman

> +	} else if (da != FW_RSC_ADDR_ANY && mem->da != FW_RSC_ADDR_ANY) {
> +		delta = da - mem->da;
> +
> +		/* Check requested resource belongs to registered carveout */
> +		if (delta < 0) {
> +			dev_err(dev,
> +				"Registered carveout doesn't fit da request\n");
> +			return -ENOMEM;
> +		}
> +
> +		if (delta + len > mem->len) {
> +			dev_err(dev,
> +				"Registered carveout doesn't fit len request\n");
> +			return -ENOMEM;
> +		}
> +	}
> +
> +	return 0;


> +}
> +
>  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
>  {
>  	struct rproc *rproc = rvdev->rproc;
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-10-15  6:40       ` Bjorn Andersson
@ 2018-10-23 23:24         ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-23 23:24 UTC (permalink / raw)
  To: Bjorn Andersson, Loic PALLARDY
  Cc: ohad, linux-remoteproc, linux-kernel, Arnaud POULIQUEN,
	benjamin.gaignard

Hi Bjorn, Loic,

On 10/15/18 1:40 AM, Bjorn Andersson wrote:

> On Wed 10 Oct 11:58 PDT 2018, Loic PALLARDY wrote:
> 
>>
>>
>>> -----Original Message-----
>>> From: Bjorn Andersson [mailto:bjorn.andersson@linaro.org]
>>> Sent: mercredi 10 octobre 2018 07:32
>>> To: Loic PALLARDY <loic.pallardy@st.com>
>>> Cc: ohad@wizery.com; linux-remoteproc@vger.kernel.org; linux-
>>> kernel@vger.kernel.org; Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
>>> benjamin.gaignard@linaro.org; s-anna@ti.com
>>> Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on
>>> centralized carveout allocator
>>>
>>> On Fri 27 Jul 06:14 PDT 2018, Loic Pallardy wrote:
>>>>  int rproc_fw_sanity_check(struct rproc *rproc, const struct firmware *fw)
>>>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
>>> b/drivers/remoteproc/remoteproc_virtio.c
>>> [..]
>>>> @@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct
>>> virtio_device *vdev,
>>>>  	rvring->vq = vq;
>>>>  	vq->priv = rvring;
>>>>
>>>> +	/* Update vring in resource table */
>>>> +	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
>>>> +	rsc->vring[id].da = mem->da;
>>>> +
>>>
>>> This would now happen after we've started the remoteproc. Don't we need
>>> to do this in-between allocating the carveouts and booting the
>>> remoteproc?
>>
>> Yes da is updated after coprocessor boot, but before vdev status in resource table is set to DRIVER_OK and kick.
>> Coprocessor should not parse this resource before as vrings not initialized yet.
>> If coprocessor needs to get some information about vring carveout at boot time, carveout resources named vdev"x"vring"y" should be added to firmware resource table.
>> In that case information will be filled before coprocessor boot.
>>
> 
> Makes sense, thanks for clarifying. Applied.

Unfortunately, our current firmwares are not doing this, and so this
patch is breaking the IPC functionality in our platform drivers. We do
wait for the vdev status before actually attempting any Tx from the
remote side.

Our newer platforms will definitely be leveraging this before the vring
inits to account for early-boot of rprocs in bootloaders and
late-attach/ipc-only modes in kernel.

regards
Suman

> 
> Regards,
> Bjorn
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-07-27 13:14 ` [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator Loic Pallardy
  2018-10-10  5:32   ` Bjorn Andersson
@ 2018-10-24  0:14   ` Suman Anna
  2018-10-24 15:14     ` Loic PALLARDY
  1 sibling, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-24  0:14 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> Current version of rproc_alloc_vring function supports only dynamic vring
> allocation.
> 
> This patch allows to allocate vrings based on memory region declatation.
> Vrings are now manage like memory carveouts, to communize memory management
> code in rproc_alloc_registered_carveouts().
> 
> Allocated buffer is retrieved in rp_find_vq() thanks to
> rproc_find_carveout_by_name() functions for.
> 
> This patch sets vrings names to vdev"x"vring"y" with x vdev index in
> resource table and y vring index in vdev. This will be updated when
> name will be associated to vdev in firmware resource table.
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> ---
>  drivers/remoteproc/remoteproc_core.c     | 61 +++++++++++++++++---------------
>  drivers/remoteproc/remoteproc_internal.h |  2 ++
>  drivers/remoteproc/remoteproc_virtio.c   | 14 +++++++-
>  include/linux/remoteproc.h               |  6 ++--
>  4 files changed, 51 insertions(+), 32 deletions(-)
> 
> diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c
> index c543d04..4edc6f0 100644
> --- a/drivers/remoteproc/remoteproc_core.c
> +++ b/drivers/remoteproc/remoteproc_core.c
> @@ -53,6 +53,11 @@ typedef int (*rproc_handle_resources_t)(struct rproc *rproc,
>  typedef int (*rproc_handle_resource_t)(struct rproc *rproc,
>  				 void *, int offset, int avail);
>  
> +static int rproc_alloc_carveout(struct rproc *rproc,
> +				struct rproc_mem_entry *mem);
> +static int rproc_release_carveout(struct rproc *rproc,
> +				  struct rproc_mem_entry *mem);
> +
>  /* Unique indices for remoteproc devices */
>  static DEFINE_IDA(rproc_dev_index);
>  
> @@ -312,21 +317,33 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
>  	struct device *dev = &rproc->dev;
>  	struct rproc_vring *rvring = &rvdev->vring[i];
>  	struct fw_rsc_vdev *rsc;
> -	dma_addr_t dma;
> -	void *va;
>  	int ret, size, notifyid;
> +	struct rproc_mem_entry *mem;
>  
>  	/* actual size of vring (in bytes) */
>  	size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
>  
> -	/*
> -	 * Allocate non-cacheable memory for the vring. In the future
> -	 * this call will also configure the IOMMU for us
> -	 */
> -	va = dma_alloc_coherent(dev->parent, size, &dma, GFP_KERNEL);
> -	if (!va) {
> -		dev_err(dev->parent, "dma_alloc_coherent failed\n");
> -		return -EINVAL;
> +	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> +
> +	/* Search for pre-registered carveout */
> +	mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d", rvdev->index,
> +					  i);
> +	if (mem) {
> +		if (rproc_check_carveout_da(rproc, mem, rsc->vring[i].da, size))
> +			return -ENOMEM;
> +	} else {
> +		/* Register carveout in in list */
> +		mem = rproc_mem_entry_init(dev, 0, 0, size, rsc->vring[i].da,
> +					   rproc_alloc_carveout,
> +					   rproc_release_carveout,
> +					   "vdev%dvring%d",
> +					   rvdev->index, i);
> +		if (!mem) {
> +			dev_err(dev, "Can't allocate memory entry structure\n");
> +			return -ENOMEM;
> +		}
> +
> +		rproc_add_carveout(rproc, mem);
>  	}
>  
>  	/*
> @@ -337,7 +354,6 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
>  	ret = idr_alloc(&rproc->notifyids, rvring, 0, 0, GFP_KERNEL);
>  	if (ret < 0) {
>  		dev_err(dev, "idr_alloc failed: %d\n", ret);
> -		dma_free_coherent(dev->parent, size, va, dma);
>  		return ret;
>  	}
>  	notifyid = ret;
> @@ -346,21 +362,9 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
>  	if (notifyid > rproc->max_notifyid)
>  		rproc->max_notifyid = notifyid;
>  
> -	dev_dbg(dev, "vring%d: va %pK dma %pad size 0x%x idr %d\n",
> -		i, va, &dma, size, notifyid);
> -
> -	rvring->va = va;
> -	rvring->dma = dma;
>  	rvring->notifyid = notifyid;
>  
> -	/*
> -	 * Let the rproc know the notifyid and da of this vring.
> -	 * Not all platforms use dma_alloc_coherent to automatically
> -	 * set up the iommu. In this case the device address (da) will
> -	 * hold the physical address and not the device address.
> -	 */
> -	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> -	rsc->vring[i].da = dma;
> +	/* Let the rproc know the notifyid of this vring.*/
>  	rsc->vring[i].notifyid = notifyid;
>  	return 0;
>  }
> @@ -392,12 +396,10 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
>  
>  void rproc_free_vring(struct rproc_vring *rvring)
>  {
> -	int size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
>  	struct rproc *rproc = rvring->rvdev->rproc;
>  	int idx = rvring->rvdev->vring - rvring;
>  	struct fw_rsc_vdev *rsc;
>  
> -	dma_free_coherent(rproc->dev.parent, size, rvring->va, rvring->dma);
>  	idr_remove(&rproc->notifyids, rvring->notifyid);

Also, I am not sure if FW_RSC_ADDR_ANY semantics were enforced
previously on the vring da. It was simply overwritten irrespective of
the value. Now, I am running again into the "bad carveout rsc
configuration value" due to the iommu_domain_check if !FW_RSC_ADDR_ANY.

FWIW, the rproc_free_vring was actually using the value 0 when resetting.

regards
Suman

>  
>  	/* reset resource entry info */
> @@ -484,6 +486,7 @@ static int rproc_handle_vdev(struct rproc *rproc, struct fw_rsc_vdev *rsc,
>  
>  	rvdev->id = rsc->id;
>  	rvdev->rproc = rproc;
> +	rvdev->index = rproc->nb_vdev++;
>  
>  	/* parse the vrings */
>  	for (i = 0; i < rsc->num_of_vrings; i++) {
> @@ -528,9 +531,6 @@ void rproc_vdev_release(struct kref *ref)
>  
>  	for (id = 0; id < ARRAY_SIZE(rvdev->vring); id++) {
>  		rvring = &rvdev->vring[id];
> -		if (!rvring->va)
> -			continue;
> -
>  		rproc_free_vring(rvring);
>  	}
>  
> @@ -1322,6 +1322,9 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
>  	/* reset max_notifyid */
>  	rproc->max_notifyid = -1;
>  
> +	/* reset handled vdev */
> +	rproc->nb_vdev = 0;
> +
>  	/* handle fw resources which are required to boot rproc */
>  	ret = rproc_handle_resources(rproc, rproc_loading_handlers);
>  	if (ret) {
> diff --git a/drivers/remoteproc/remoteproc_internal.h b/drivers/remoteproc/remoteproc_internal.h
> index 7570beb..f6cad24 100644
> --- a/drivers/remoteproc/remoteproc_internal.h
> +++ b/drivers/remoteproc/remoteproc_internal.h
> @@ -60,6 +60,8 @@ struct dentry *rproc_create_trace_file(const char *name, struct rproc *rproc,
>  int rproc_elf_load_rsc_table(struct rproc *rproc, const struct firmware *fw);
>  struct resource_table *rproc_elf_find_loaded_rsc_table(struct rproc *rproc,
>  						       const struct firmware *fw);
> +struct rproc_mem_entry *
> +rproc_find_carveout_by_name(struct rproc *rproc, const char *name, ...);
>  
>  static inline
>  int rproc_fw_sanity_check(struct rproc *rproc, const struct firmware *fw)
> diff --git a/drivers/remoteproc/remoteproc_virtio.c b/drivers/remoteproc/remoteproc_virtio.c
> index bbecd44..de21f62 100644
> --- a/drivers/remoteproc/remoteproc_virtio.c
> +++ b/drivers/remoteproc/remoteproc_virtio.c
> @@ -76,7 +76,9 @@ static struct virtqueue *rp_find_vq(struct virtio_device *vdev,
>  	struct rproc_vdev *rvdev = vdev_to_rvdev(vdev);
>  	struct rproc *rproc = vdev_to_rproc(vdev);
>  	struct device *dev = &rproc->dev;
> +	struct rproc_mem_entry *mem;
>  	struct rproc_vring *rvring;
> +	struct fw_rsc_vdev *rsc;
>  	struct virtqueue *vq;
>  	void *addr;
>  	int len, size;
> @@ -88,8 +90,14 @@ static struct virtqueue *rp_find_vq(struct virtio_device *vdev,
>  	if (!name)
>  		return NULL;
>  
> +	/* Search allocated memory region by name */
> +	mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d", rvdev->index,
> +					  id);
> +	if (!mem || !mem->va)
> +		return ERR_PTR(-ENOMEM);
> +
>  	rvring = &rvdev->vring[id];
> -	addr = rvring->va;
> +	addr = mem->va;
>  	len = rvring->len;
>  
>  	/* zero vring */
> @@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct virtio_device *vdev,
>  	rvring->vq = vq;
>  	vq->priv = rvring;
>  
> +	/* Update vring in resource table */
> +	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> +	rsc->vring[id].da = mem->da;
> +
>  	return vq;
>  }
>  
> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> index 4cdd0c6..6b3a234 100644
> --- a/include/linux/remoteproc.h
> +++ b/include/linux/remoteproc.h
> @@ -453,6 +453,7 @@ struct rproc_dump_segment {
>   * @table_sz: size of @cached_table
>   * @has_iommu: flag to indicate if remote processor is behind an MMU
>   * @dump_segments: list of segments in the firmware
> + * @nb_vdev: number of vdev currently handled by rproc
>   */
>  struct rproc {
>  	struct list_head node;
> @@ -485,6 +486,7 @@ struct rproc {
>  	bool has_iommu;
>  	bool auto_boot;
>  	struct list_head dump_segments;
> +	int nb_vdev;
>  };
>  
>  /**
> @@ -512,7 +514,6 @@ struct rproc_subdev {
>  /**
>   * struct rproc_vring - remoteproc vring state
>   * @va:	virtual address
> - * @dma: dma address
>   * @len: length, in bytes
>   * @da: device address
>   * @align: vring alignment
> @@ -522,7 +523,6 @@ struct rproc_subdev {
>   */
>  struct rproc_vring {
>  	void *va;
> -	dma_addr_t dma;
>  	int len;
>  	u32 da;
>  	u32 align;
> @@ -541,6 +541,7 @@ struct rproc_vring {
>   * @vdev: the virio device
>   * @vring: the vrings for this vdev
>   * @rsc_offset: offset of the vdev's resource entry
> + * @index: vdev position versus other vdev declared in resource table
>   */
>  struct rproc_vdev {
>  	struct kref refcount;
> @@ -553,6 +554,7 @@ struct rproc_vdev {
>  	struct virtio_device vdev;
>  	struct rproc_vring vring[RVDEV_NUM_VRINGS];
>  	u32 rsc_offset;
> +	u32 index;
>  };
>  
>  struct rproc *rproc_get_by_phandle(phandle phandle);
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-09-27 20:18       ` Wendy Liang
@ 2018-10-24  1:22         ` Suman Anna
  2018-10-24  1:48           ` Suman Anna
  2018-10-24 12:40           ` Loic PALLARDY
  0 siblings, 2 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-24  1:22 UTC (permalink / raw)
  To: Wendy Liang, Loic Pallardy
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, arnaud.pouliquen, benjamin.gaignard

On 9/27/18 3:18 PM, Wendy Liang wrote:
> Hi Loic,
> 
> 
> On Thu, Sep 27, 2018 at 12:22 PM Loic PALLARDY <loic.pallardy@st.com> wrote:
>>
>> Hi Wendy
>>
>>> -----Original Message-----
>>> From: Wendy Liang <sunnyliangjy@gmail.com>
>>> Sent: Thursday, September 27, 2018 7:17 PM
>>> To: Loic PALLARDY <loic.pallardy@st.com>
>>> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
>>> <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; Linux Kernel
>>> Mailing List <linux-kernel@vger.kernel.org>; Arnaud POULIQUEN
>>> <arnaud.pouliquen@st.com>; benjamin.gaignard@linaro.org; Suman Anna
>>> <s-anna@ti.com>
>>> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
>>> specific dma memory pool
>>>
>>> On Fri, Jul 27, 2018 at 6:16 AM Loic Pallardy <loic.pallardy@st.com> wrote:
>>>>
>>>> This patch creates a dedicated vdev subdevice for each vdev declared
>>>> in firmware resource table and associates carveout named "vdev%dbuffer"
>>>> (with %d vdev index in resource table) if any as dma coherent memory
>>> pool.
>>>>
>>>> Then vdev subdevice is used as parent for virtio device.
>>>>
>>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>>> ---
>>>>  drivers/remoteproc/remoteproc_core.c     | 35
>>> +++++++++++++++++++++++---
>>>>  drivers/remoteproc/remoteproc_internal.h |  1 +
>>>>  drivers/remoteproc/remoteproc_virtio.c   | 42
>>> +++++++++++++++++++++++++++++++-
>>>>  include/linux/remoteproc.h               |  1 +
>>>>  4 files changed, 75 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>>> b/drivers/remoteproc/remoteproc_core.c
>>>> index 4edc6f0..adcc66e 100644
>>>> --- a/drivers/remoteproc/remoteproc_core.c
>>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>>> @@ -39,6 +39,7 @@
>>>>  #include <linux/idr.h>
>>>>  #include <linux/elf.h>
>>>>  #include <linux/crc32.h>
>>>> +#include <linux/of_reserved_mem.h>
>>>>  #include <linux/virtio_ids.h>
>>>>  #include <linux/virtio_ring.h>
>>>>  #include <asm/byteorder.h>
>>>> @@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc
>>> *rproc)
>>>>         iommu_domain_free(domain);
>>>>  }
>>>>
>>>> -static phys_addr_t rproc_va_to_pa(void *cpu_addr)
>>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr)
>>>>  {
>>>>         /*
>>>>          * Return physical address according to virtual address location
>>>> @@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void
>>> *cpu_addr)
>>>>         WARN_ON(!virt_addr_valid(cpu_addr));
>>>>         return virt_to_phys(cpu_addr);
>>>>  }
>>>> +EXPORT_SYMBOL(rproc_va_to_pa);
>>>>
>>>>  /**
>>>>   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc
>>> address
>>>> @@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct
>>> rproc_subdev *subdev, bool crashed)
>>>>  }
>>>>
>>>>  /**
>>>> + * rproc_rvdev_release() - release the existence of a rvdev
>>>> + *
>>>> + * @dev: the subdevice's dev
>>>> + */
>>>> +static void rproc_rvdev_release(struct device *dev)
>>>> +{
>>>> +       struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev, dev);
>>>> +
>>>> +       of_reserved_mem_device_release(dev);
>>>> +
>>>> +       kfree(rvdev);
>>>> +}
>>>> +
>>>> +/**
>>>>   * rproc_handle_vdev() - handle a vdev fw resource
>>>>   * @rproc: the remote processor
>>>>   * @rsc: the vring resource descriptor
>>>> @@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
>>> struct fw_rsc_vdev *rsc,
>>>>         struct device *dev = &rproc->dev;
>>>>         struct rproc_vdev *rvdev;
>>>>         int i, ret;
>>>> +       char name[16];
>>>>
>>>>         /* make sure resource isn't truncated */
>>>>         if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct
>>> fw_rsc_vdev_vring)
>>>> @@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc *rproc,
>>> struct fw_rsc_vdev *rsc,
>>>>         rvdev->rproc = rproc;
>>>>         rvdev->index = rproc->nb_vdev++;
>>>>
>>>> +       /* Initialise vdev subdevice */
>>>> +       snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
>>>> +       rvdev->dev.parent = rproc->dev.parent;
>>>> +       rvdev->dev.release = rproc_rvdev_release;
>>>> +       dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev-
>>>> dev.parent), name);
>>>> +       dev_set_drvdata(&rvdev->dev, rvdev);
>>>> +       dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
>>> I tried the latest kernel, this function will not set the DMA coherent mask as
>>> dma_supported() of the &rvdev->dev will return false.
>>> As this is a device created at run time, should it be force to support DMA?
>>> should it directly set the dma_coherent_mask?
>>
>> Thanks for pointing me this issue. I tested on top of 4.18-rc1 few months ago...
>> Could you please give me kernel version on which you are testing the series?
>> Is you platform 32bit or 64bit ?
>> I'll rebase and check on my side.
> 
> I am testing with 4.19-rc4 on aarch64 platform.

Btw, I ran into this on my v7 platform as well (4.19-rc6). The
dma_set_coherent_mask fails with error EIO. I did get my allocations
through though.

regards
Suman

> 
> Best Regards,
> Wendy
>>
>> Regards,
>> Loic
>>
>>>
>>>> +
>>>> +       ret = device_register(&rvdev->dev);
>>>> +       if (ret)
>>>> +               goto free_rvdev;
>>>> +
>>>>         /* parse the vrings */
>>>>         for (i = 0; i < rsc->num_of_vrings; i++) {
>>>>                 ret = rproc_parse_vring(rvdev, rsc, i);
>>>> @@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
>>> struct fw_rsc_vdev *rsc,
>>>>         for (i--; i >= 0; i--)
>>>>                 rproc_free_vring(&rvdev->vring[i]);
>>>>  free_rvdev:
>>>> -       kfree(rvdev);
>>>> +       device_unregister(&rvdev->dev);
>>>>         return ret;
>>>>  }
>>>>
>>>> @@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
>>>>
>>>>         rproc_remove_subdev(rproc, &rvdev->subdev);
>>>>         list_del(&rvdev->node);
>>>> -       kfree(rvdev);
>>>> +       device_unregister(&rvdev->dev);
>>>>  }
>>>>
>>>>  /**
>>>> diff --git a/drivers/remoteproc/remoteproc_internal.h
>>> b/drivers/remoteproc/remoteproc_internal.h
>>>> index f6cad24..bfeacfd 100644
>>>> --- a/drivers/remoteproc/remoteproc_internal.h
>>>> +++ b/drivers/remoteproc/remoteproc_internal.h
>>>> @@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const char
>>> *name, struct rproc *rproc,
>>>>  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
>>>>
>>>>  void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
>>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr);
>>>>  int rproc_trigger_recovery(struct rproc *rproc);
>>>>
>>>>  int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware *fw);
>>>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
>>> b/drivers/remoteproc/remoteproc_virtio.c
>>>> index de21f62..9ee63c0 100644
>>>> --- a/drivers/remoteproc/remoteproc_virtio.c
>>>> +++ b/drivers/remoteproc/remoteproc_virtio.c
>>>> @@ -17,7 +17,9 @@
>>>>   * GNU General Public License for more details.
>>>>   */
>>>>
>>>> +#include <linux/dma-mapping.h>
>>>>  #include <linux/export.h>
>>>> +#include <linux/of_reserved_mem.h>
>>>>  #include <linux/remoteproc.h>
>>>>  #include <linux/virtio.h>
>>>>  #include <linux/virtio_config.h>
>>>> @@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct device
>>> *dev)
>>>>  int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
>>>>  {
>>>>         struct rproc *rproc = rvdev->rproc;
>>>> -       struct device *dev = &rproc->dev;
>>>> +       struct device *dev = &rvdev->dev;
>>>>         struct virtio_device *vdev = &rvdev->vdev;
>>>> +       struct rproc_mem_entry *mem;
>>>>         int ret;
>>>>
>>>> +       /* Try to find dedicated vdev buffer carveout */
>>>> +       mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer", rvdev-
>>>> index);
>>>> +       if (mem) {
>>>> +               phys_addr_t pa;
>>>> +
>>>> +               if (mem->of_resm_idx != -1) {
>>>> +                       struct device_node *np = rproc->dev.parent->of_node;
>>>> +
>>>> +                       /* Associate reserved memory to vdev device */
>>>> +                       ret = of_reserved_mem_device_init_by_idx(dev, np,
>>>> +                                                                mem->of_resm_idx);
>>>> +                       if (ret) {
>>>> +                               dev_err(dev, "Can't associate reserved memory\n");
>>>> +                               goto out;
>>>> +                       }
>>>> +               } else {
>>>> +                       if (mem->va) {
>>>> +                               dev_warn(dev, "vdev %d buffer already mapped\n",
>>>> +                                        rvdev->index);
>>>> +                               pa = rproc_va_to_pa(mem->va);
>>>> +                       } else {
>>>> +                               /* Use dma address as carveout no memmapped yet */
>>>> +                               pa = (phys_addr_t)mem->dma;
>>>> +                       }
>>>> +
>>>> +                       /* Associate vdev buffer memory pool to vdev subdev */
>>>> +                       ret = dmam_declare_coherent_memory(dev, pa,
>>>> +                                                          mem->da,
>>>> +                                                          mem->len,
>>>> +                                                          DMA_MEMORY_EXCLUSIVE);
>>>> +                       if (ret < 0) {
>>>> +                               dev_err(dev, "Failed to associate buffer\n");
>>>> +                               goto out;
>>>> +                       }
>>>> +               }
>>>> +       }
>>>> +
>>>>         vdev->id.device = id,
>>>>         vdev->config = &rproc_virtio_config_ops,
>>>>         vdev->dev.parent = dev;
>>>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
>>>> index 6b3a234..2921dd2 100644
>>>> --- a/include/linux/remoteproc.h
>>>> +++ b/include/linux/remoteproc.h
>>>> @@ -547,6 +547,7 @@ struct rproc_vdev {
>>>>         struct kref refcount;
>>>>
>>>>         struct rproc_subdev subdev;
>>>> +       struct device dev;
>>>>
>>>>         unsigned int id;
>>>>         struct list_head node;
>>>> --
>>>> 1.9.1
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-remoteproc"
>>> in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-10-10 19:17     ` Loic PALLARDY
@ 2018-10-24  1:27       ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-24  1:27 UTC (permalink / raw)
  To: Loic PALLARDY, Bjorn Andersson
  Cc: ohad, linux-remoteproc, linux-kernel, Arnaud POULIQUEN,
	benjamin.gaignard, Anup Patel

On 10/10/18 2:17 PM, Loic PALLARDY wrote:
> 
> 
>> -----Original Message-----
>> From: Bjorn Andersson [mailto:bjorn.andersson@linaro.org]
>> Sent: mercredi 10 octobre 2018 07:58
>> To: Loic PALLARDY <loic.pallardy@st.com>
>> Cc: ohad@wizery.com; linux-remoteproc@vger.kernel.org; linux-
>> kernel@vger.kernel.org; Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
>> benjamin.gaignard@linaro.org; s-anna@ti.com
>> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
>> specific dma memory pool
>>
>> On Fri 27 Jul 06:14 PDT 2018, Loic Pallardy wrote:
>>
>>> This patch creates a dedicated vdev subdevice for each vdev declared
>>> in firmware resource table and associates carveout named "vdev%dbuffer"
>>> (with %d vdev index in resource table) if any as dma coherent memory
>> pool.
>>>
>>> Then vdev subdevice is used as parent for virtio device.
>>>
>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>> ---
>>>  drivers/remoteproc/remoteproc_core.c     | 35
>> +++++++++++++++++++++++---
>>>  drivers/remoteproc/remoteproc_internal.h |  1 +
>>>  drivers/remoteproc/remoteproc_virtio.c   | 42
>> +++++++++++++++++++++++++++++++-
>>>  include/linux/remoteproc.h               |  1 +
>>>  4 files changed, 75 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>> b/drivers/remoteproc/remoteproc_core.c
>>> index 4edc6f0..adcc66e 100644
>>> --- a/drivers/remoteproc/remoteproc_core.c
>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>> @@ -39,6 +39,7 @@
>>>  #include <linux/idr.h>
>>>  #include <linux/elf.h>
>>>  #include <linux/crc32.h>
>>> +#include <linux/of_reserved_mem.h>
>>>  #include <linux/virtio_ids.h>
>>>  #include <linux/virtio_ring.h>
>>>  #include <asm/byteorder.h>
>>> @@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc
>> *rproc)
>>>  	iommu_domain_free(domain);
>>>  }
>>>
>>> -static phys_addr_t rproc_va_to_pa(void *cpu_addr)
>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr)
>>>  {
>>>  	/*
>>>  	 * Return physical address according to virtual address location
>>> @@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void
>> *cpu_addr)
>>>  	WARN_ON(!virt_addr_valid(cpu_addr));
>>>  	return virt_to_phys(cpu_addr);
>>>  }
>>> +EXPORT_SYMBOL(rproc_va_to_pa);
>>>
>>>  /**
>>>   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc
>> address
>>> @@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct
>> rproc_subdev *subdev, bool crashed)
>>>  }
>>>
>>>  /**
>>> + * rproc_rvdev_release() - release the existence of a rvdev
>>> + *
>>> + * @dev: the subdevice's dev
>>> + */
>>> +static void rproc_rvdev_release(struct device *dev)
>>> +{
>>> +	struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev,
>> dev);
>>> +
>>> +	of_reserved_mem_device_release(dev);
>>> +
>>> +	kfree(rvdev);
>>> +}
>>> +
>>> +/**
>>>   * rproc_handle_vdev() - handle a vdev fw resource
>>>   * @rproc: the remote processor
>>>   * @rsc: the vring resource descriptor
>>> @@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
>> struct fw_rsc_vdev *rsc,
>>>  	struct device *dev = &rproc->dev;
>>>  	struct rproc_vdev *rvdev;
>>>  	int i, ret;
>>> +	char name[16];
>>>
>>>  	/* make sure resource isn't truncated */
>>>  	if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct
>> fw_rsc_vdev_vring)
>>> @@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc *rproc,
>> struct fw_rsc_vdev *rsc,
>>>  	rvdev->rproc = rproc;
>>>  	rvdev->index = rproc->nb_vdev++;
>>>
>>> +	/* Initialise vdev subdevice */
>>> +	snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
>>> +	rvdev->dev.parent = rproc->dev.parent;
>>> +	rvdev->dev.release = rproc_rvdev_release;
>>> +	dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev-
>>> dev.parent), name);
>>> +	dev_set_drvdata(&rvdev->dev, rvdev);
>>> +	dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
>>> +
>>> +	ret = device_register(&rvdev->dev);
>>> +	if (ret)
>>> +		goto free_rvdev;
>>> +
>>>  	/* parse the vrings */
>>>  	for (i = 0; i < rsc->num_of_vrings; i++) {
>>>  		ret = rproc_parse_vring(rvdev, rsc, i);
>>> @@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
>> struct fw_rsc_vdev *rsc,
>>>  	for (i--; i >= 0; i--)
>>>  		rproc_free_vring(&rvdev->vring[i]);
>>>  free_rvdev:
>>> -	kfree(rvdev);
>>> +	device_unregister(&rvdev->dev);
>>>  	return ret;
>>>  }
>>>
>>> @@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
>>>
>>>  	rproc_remove_subdev(rproc, &rvdev->subdev);
>>>  	list_del(&rvdev->node);
>>> -	kfree(rvdev);
>>> +	device_unregister(&rvdev->dev);
>>>  }
>>>
>>>  /**
>>> diff --git a/drivers/remoteproc/remoteproc_internal.h
>> b/drivers/remoteproc/remoteproc_internal.h
>>> index f6cad24..bfeacfd 100644
>>> --- a/drivers/remoteproc/remoteproc_internal.h
>>> +++ b/drivers/remoteproc/remoteproc_internal.h
>>> @@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const char
>> *name, struct rproc *rproc,
>>>  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
>>>
>>>  void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr);
>>>  int rproc_trigger_recovery(struct rproc *rproc);
>>>
>>>  int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware *fw);
>>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
>> b/drivers/remoteproc/remoteproc_virtio.c
>>> index de21f62..9ee63c0 100644
>>> --- a/drivers/remoteproc/remoteproc_virtio.c
>>> +++ b/drivers/remoteproc/remoteproc_virtio.c
>>> @@ -17,7 +17,9 @@
>>>   * GNU General Public License for more details.
>>>   */
>>>
>>> +#include <linux/dma-mapping.h>
>>>  #include <linux/export.h>
>>> +#include <linux/of_reserved_mem.h>
>>>  #include <linux/remoteproc.h>
>>>  #include <linux/virtio.h>
>>>  #include <linux/virtio_config.h>
>>> @@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct device
>> *dev)
>>>  int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
>>>  {
>>>  	struct rproc *rproc = rvdev->rproc;
>>> -	struct device *dev = &rproc->dev;
>>> +	struct device *dev = &rvdev->dev;
>>>  	struct virtio_device *vdev = &rvdev->vdev;
>>> +	struct rproc_mem_entry *mem;
>>>  	int ret;
>>>
>>> +	/* Try to find dedicated vdev buffer carveout */
>>> +	mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer",
>> rvdev->index);
>>> +	if (mem) {
>>> +		phys_addr_t pa;
>>> +
>>> +		if (mem->of_resm_idx != -1) {
>>> +			struct device_node *np = rproc->dev.parent-
>>> of_node;
>>> +
>>> +			/* Associate reserved memory to vdev device */
>>> +			ret = of_reserved_mem_device_init_by_idx(dev, np,
>>> +								 mem-
>>> of_resm_idx);
>>> +			if (ret) {
>>> +				dev_err(dev, "Can't associate reserved
>> memory\n");
>>> +				goto out;
>>> +			}
>>> +		} else {
>>> +			if (mem->va) {
>>> +				dev_warn(dev, "vdev %d buffer already
>> mapped\n",
>>> +					 rvdev->index);
>>> +				pa = rproc_va_to_pa(mem->va);
>>> +			} else {
>>> +				/* Use dma address as carveout no
>> memmapped yet */
>>> +				pa = (phys_addr_t)mem->dma;
>>> +			}
>>> +
>>> +			/* Associate vdev buffer memory pool to vdev
>> subdev */
>>> +			ret = dmam_declare_coherent_memory(dev, pa,
>>> +							   mem->da,
>>> +							   mem->len,
>>> +
>> DMA_MEMORY_EXCLUSIVE);
>>
>> Is it not possible to associate the dma_mem with vdev->dev directly,
>> instead of injecting yet another device in-between the remoteproc
>> platform device and the virtio device?
> 
> Thank you for the review.
> I'll check how to use virtio device. An additional ops for memory registration may be needed at virtio level. 
> 
>>
>> That way the resulting allocation in e.g. virtio_rpmsg would be on the
>> virtio device itself, not its parent.

> Yes agree, it will simplify allocation/inheritance...

+ Anup.

Anup,
Since you are currently interested in patch 17 (dma allocations from
vdev parent), may we get some information on how your device hierarchy
is looking like inside the VM/Guest?

regards
Suman

> 
> Regards,
> Loic
>>
>> Regards,
>> Bjorn
>>
>>> +			if (ret < 0) {
>>> +				dev_err(dev, "Failed to associate buffer\n");
>>> +				goto out;
>>> +			}
>>> +		}
>>> +	}
>>> +
>>>  	vdev->id.device	= id,
>>>  	vdev->config = &rproc_virtio_config_ops,
>>>  	vdev->dev.parent = dev;
>>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
>>> index 6b3a234..2921dd2 100644
>>> --- a/include/linux/remoteproc.h
>>> +++ b/include/linux/remoteproc.h
>>> @@ -547,6 +547,7 @@ struct rproc_vdev {
>>>  	struct kref refcount;
>>>
>>>  	struct rproc_subdev subdev;
>>> +	struct device dev;
>>>
>>>  	unsigned int id;
>>>  	struct list_head node;
>>> --
>>> 1.9.1
>>>


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-10-24  1:22         ` Suman Anna
@ 2018-10-24  1:48           ` Suman Anna
  2018-10-24 12:42             ` Loic PALLARDY
  2018-10-24 12:40           ` Loic PALLARDY
  1 sibling, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-24  1:48 UTC (permalink / raw)
  To: Wendy Liang, Loic Pallardy
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, arnaud.pouliquen, benjamin.gaignard

On 10/23/18 8:22 PM, Suman Anna wrote:
> On 9/27/18 3:18 PM, Wendy Liang wrote:
>> Hi Loic,
>>
>>
>> On Thu, Sep 27, 2018 at 12:22 PM Loic PALLARDY <loic.pallardy@st.com> wrote:
>>>
>>> Hi Wendy
>>>
>>>> -----Original Message-----
>>>> From: Wendy Liang <sunnyliangjy@gmail.com>
>>>> Sent: Thursday, September 27, 2018 7:17 PM
>>>> To: Loic PALLARDY <loic.pallardy@st.com>
>>>> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
>>>> <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; Linux Kernel
>>>> Mailing List <linux-kernel@vger.kernel.org>; Arnaud POULIQUEN
>>>> <arnaud.pouliquen@st.com>; benjamin.gaignard@linaro.org; Suman Anna
>>>> <s-anna@ti.com>
>>>> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
>>>> specific dma memory pool
>>>>
>>>> On Fri, Jul 27, 2018 at 6:16 AM Loic Pallardy <loic.pallardy@st.com> wrote:
>>>>>
>>>>> This patch creates a dedicated vdev subdevice for each vdev declared
>>>>> in firmware resource table and associates carveout named "vdev%dbuffer"
>>>>> (with %d vdev index in resource table) if any as dma coherent memory
>>>> pool.
>>>>>
>>>>> Then vdev subdevice is used as parent for virtio device.
>>>>>
>>>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>>>> ---
>>>>>  drivers/remoteproc/remoteproc_core.c     | 35
>>>> +++++++++++++++++++++++---
>>>>>  drivers/remoteproc/remoteproc_internal.h |  1 +
>>>>>  drivers/remoteproc/remoteproc_virtio.c   | 42
>>>> +++++++++++++++++++++++++++++++-
>>>>>  include/linux/remoteproc.h               |  1 +
>>>>>  4 files changed, 75 insertions(+), 4 deletions(-)
>>>>>
>>>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>>>> b/drivers/remoteproc/remoteproc_core.c
>>>>> index 4edc6f0..adcc66e 100644
>>>>> --- a/drivers/remoteproc/remoteproc_core.c
>>>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>>>> @@ -39,6 +39,7 @@
>>>>>  #include <linux/idr.h>
>>>>>  #include <linux/elf.h>
>>>>>  #include <linux/crc32.h>
>>>>> +#include <linux/of_reserved_mem.h>
>>>>>  #include <linux/virtio_ids.h>
>>>>>  #include <linux/virtio_ring.h>
>>>>>  #include <asm/byteorder.h>
>>>>> @@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc
>>>> *rproc)
>>>>>         iommu_domain_free(domain);
>>>>>  }
>>>>>
>>>>> -static phys_addr_t rproc_va_to_pa(void *cpu_addr)
>>>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr)
>>>>>  {
>>>>>         /*
>>>>>          * Return physical address according to virtual address location
>>>>> @@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void
>>>> *cpu_addr)
>>>>>         WARN_ON(!virt_addr_valid(cpu_addr));
>>>>>         return virt_to_phys(cpu_addr);
>>>>>  }
>>>>> +EXPORT_SYMBOL(rproc_va_to_pa);
>>>>>
>>>>>  /**
>>>>>   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc
>>>> address
>>>>> @@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct
>>>> rproc_subdev *subdev, bool crashed)
>>>>>  }
>>>>>
>>>>>  /**
>>>>> + * rproc_rvdev_release() - release the existence of a rvdev
>>>>> + *
>>>>> + * @dev: the subdevice's dev
>>>>> + */
>>>>> +static void rproc_rvdev_release(struct device *dev)
>>>>> +{
>>>>> +       struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev, dev);
>>>>> +
>>>>> +       of_reserved_mem_device_release(dev);
>>>>> +
>>>>> +       kfree(rvdev);
>>>>> +}
>>>>> +
>>>>> +/**
>>>>>   * rproc_handle_vdev() - handle a vdev fw resource
>>>>>   * @rproc: the remote processor
>>>>>   * @rsc: the vring resource descriptor
>>>>> @@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
>>>> struct fw_rsc_vdev *rsc,
>>>>>         struct device *dev = &rproc->dev;
>>>>>         struct rproc_vdev *rvdev;
>>>>>         int i, ret;
>>>>> +       char name[16];
>>>>>
>>>>>         /* make sure resource isn't truncated */
>>>>>         if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct
>>>> fw_rsc_vdev_vring)
>>>>> @@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc *rproc,
>>>> struct fw_rsc_vdev *rsc,
>>>>>         rvdev->rproc = rproc;
>>>>>         rvdev->index = rproc->nb_vdev++;
>>>>>
>>>>> +       /* Initialise vdev subdevice */
>>>>> +       snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
>>>>> +       rvdev->dev.parent = rproc->dev.parent;
>>>>> +       rvdev->dev.release = rproc_rvdev_release;
>>>>> +       dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev-
>>>>> dev.parent), name);
>>>>> +       dev_set_drvdata(&rvdev->dev, rvdev);
>>>>> +       dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
>>>> I tried the latest kernel, this function will not set the DMA coherent mask as
>>>> dma_supported() of the &rvdev->dev will return false.
>>>> As this is a device created at run time, should it be force to support DMA?
>>>> should it directly set the dma_coherent_mask?
>>>
>>> Thanks for pointing me this issue. I tested on top of 4.18-rc1 few months ago...
>>> Could you please give me kernel version on which you are testing the series?
>>> Is you platform 32bit or 64bit ?
>>> I'll rebase and check on my side.
>>
>> I am testing with 4.19-rc4 on aarch64 platform.
> 
> Btw, I ran into this on my v7 platform as well (4.19-rc6). The
> dma_set_coherent_mask fails with error EIO. I did get my allocations
> through though.

Correction, that was before Patch 17. With patch 17, this fails.

regards
Suman

> 
> regards
> Suman
> 
>>
>> Best Regards,
>> Wendy
>>>
>>> Regards,
>>> Loic
>>>
>>>>
>>>>> +
>>>>> +       ret = device_register(&rvdev->dev);
>>>>> +       if (ret)
>>>>> +               goto free_rvdev;
>>>>> +
>>>>>         /* parse the vrings */
>>>>>         for (i = 0; i < rsc->num_of_vrings; i++) {
>>>>>                 ret = rproc_parse_vring(rvdev, rsc, i);
>>>>> @@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
>>>> struct fw_rsc_vdev *rsc,
>>>>>         for (i--; i >= 0; i--)
>>>>>                 rproc_free_vring(&rvdev->vring[i]);
>>>>>  free_rvdev:
>>>>> -       kfree(rvdev);
>>>>> +       device_unregister(&rvdev->dev);
>>>>>         return ret;
>>>>>  }
>>>>>
>>>>> @@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
>>>>>
>>>>>         rproc_remove_subdev(rproc, &rvdev->subdev);
>>>>>         list_del(&rvdev->node);
>>>>> -       kfree(rvdev);
>>>>> +       device_unregister(&rvdev->dev);
>>>>>  }
>>>>>
>>>>>  /**
>>>>> diff --git a/drivers/remoteproc/remoteproc_internal.h
>>>> b/drivers/remoteproc/remoteproc_internal.h
>>>>> index f6cad24..bfeacfd 100644
>>>>> --- a/drivers/remoteproc/remoteproc_internal.h
>>>>> +++ b/drivers/remoteproc/remoteproc_internal.h
>>>>> @@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const char
>>>> *name, struct rproc *rproc,
>>>>>  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
>>>>>
>>>>>  void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
>>>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr);
>>>>>  int rproc_trigger_recovery(struct rproc *rproc);
>>>>>
>>>>>  int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware *fw);
>>>>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
>>>> b/drivers/remoteproc/remoteproc_virtio.c
>>>>> index de21f62..9ee63c0 100644
>>>>> --- a/drivers/remoteproc/remoteproc_virtio.c
>>>>> +++ b/drivers/remoteproc/remoteproc_virtio.c
>>>>> @@ -17,7 +17,9 @@
>>>>>   * GNU General Public License for more details.
>>>>>   */
>>>>>
>>>>> +#include <linux/dma-mapping.h>
>>>>>  #include <linux/export.h>
>>>>> +#include <linux/of_reserved_mem.h>
>>>>>  #include <linux/remoteproc.h>
>>>>>  #include <linux/virtio.h>
>>>>>  #include <linux/virtio_config.h>
>>>>> @@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct device
>>>> *dev)
>>>>>  int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
>>>>>  {
>>>>>         struct rproc *rproc = rvdev->rproc;
>>>>> -       struct device *dev = &rproc->dev;
>>>>> +       struct device *dev = &rvdev->dev;
>>>>>         struct virtio_device *vdev = &rvdev->vdev;
>>>>> +       struct rproc_mem_entry *mem;
>>>>>         int ret;
>>>>>
>>>>> +       /* Try to find dedicated vdev buffer carveout */
>>>>> +       mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer", rvdev-
>>>>> index);
>>>>> +       if (mem) {
>>>>> +               phys_addr_t pa;
>>>>> +
>>>>> +               if (mem->of_resm_idx != -1) {
>>>>> +                       struct device_node *np = rproc->dev.parent->of_node;
>>>>> +
>>>>> +                       /* Associate reserved memory to vdev device */
>>>>> +                       ret = of_reserved_mem_device_init_by_idx(dev, np,
>>>>> +                                                                mem->of_resm_idx);
>>>>> +                       if (ret) {
>>>>> +                               dev_err(dev, "Can't associate reserved memory\n");
>>>>> +                               goto out;
>>>>> +                       }
>>>>> +               } else {
>>>>> +                       if (mem->va) {
>>>>> +                               dev_warn(dev, "vdev %d buffer already mapped\n",
>>>>> +                                        rvdev->index);
>>>>> +                               pa = rproc_va_to_pa(mem->va);
>>>>> +                       } else {
>>>>> +                               /* Use dma address as carveout no memmapped yet */
>>>>> +                               pa = (phys_addr_t)mem->dma;
>>>>> +                       }
>>>>> +
>>>>> +                       /* Associate vdev buffer memory pool to vdev subdev */
>>>>> +                       ret = dmam_declare_coherent_memory(dev, pa,
>>>>> +                                                          mem->da,
>>>>> +                                                          mem->len,
>>>>> +                                                          DMA_MEMORY_EXCLUSIVE);
>>>>> +                       if (ret < 0) {
>>>>> +                               dev_err(dev, "Failed to associate buffer\n");
>>>>> +                               goto out;
>>>>> +                       }
>>>>> +               }
>>>>> +       }
>>>>> +
>>>>>         vdev->id.device = id,
>>>>>         vdev->config = &rproc_virtio_config_ops,
>>>>>         vdev->dev.parent = dev;
>>>>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
>>>>> index 6b3a234..2921dd2 100644
>>>>> --- a/include/linux/remoteproc.h
>>>>> +++ b/include/linux/remoteproc.h
>>>>> @@ -547,6 +547,7 @@ struct rproc_vdev {
>>>>>         struct kref refcount;
>>>>>
>>>>>         struct rproc_subdev subdev;
>>>>> +       struct device dev;
>>>>>
>>>>>         unsigned int id;
>>>>>         struct list_head node;
>>>>> --
>>>>> 1.9.1
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-remoteproc"
>>>> in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 15/17] remoteproc: da8xx: declare reserved memory region for vdev device
  2018-07-27 13:14 ` [PATCH v4 15/17] remoteproc: da8xx: " Loic Pallardy
@ 2018-10-24  2:57   ` Suman Anna
  2018-10-24 13:19     ` Loic PALLARDY
  0 siblings, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-24  2:57 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

Hi Loic,

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> This patch introduces da8xx_rproc_parse_fw() to declare a
> carveout region based on reserved memory for vdev buffer
> allocation.
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> ---
>  drivers/remoteproc/da8xx_remoteproc.c | 38 +++++++++++++++++++++++++++++++++++
>  1 file changed, 38 insertions(+)
> 
> diff --git a/drivers/remoteproc/da8xx_remoteproc.c b/drivers/remoteproc/da8xx_remoteproc.c
> index b668e32..679a076 100644
> --- a/drivers/remoteproc/da8xx_remoteproc.c
> +++ b/drivers/remoteproc/da8xx_remoteproc.c
> @@ -16,6 +16,7 @@
>  #include <linux/irq.h>
>  #include <linux/kernel.h>
>  #include <linux/module.h>
> +#include <linux/of_address.h>
>  #include <linux/of_reserved_mem.h>
>  #include <linux/platform_device.h>
>  #include <linux/remoteproc.h>
> @@ -179,10 +180,47 @@ static void da8xx_rproc_kick(struct rproc *rproc, int vqid)
>  	writel(SYSCFG_CHIPSIG2, drproc->chipsig);
>  }
>  
> +static int da8xx_rproc_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> +	struct device *dev = rproc->dev.parent;
> +	struct rproc_mem_entry *mem;
> +	struct device_node *node;
> +	struct resource res;
> +	int err;
> +
> +	node = of_parse_phandle(dev->of_node, "memory-region", 0);
> +	if (!node) {
> +		dev_err(dev, "No memory-region specified\n");
> +		return -EINVAL;
> +	}
> +
> +	err = of_address_to_resource(node, 0, &res);
> +	if (err) {
> +		dev_err(dev, "Bad memory-region definition\n");
> +		return err;
> +	}
> +
> +	/* Register memory region for vdev buffer allocation */
> +	mem = rproc_of_resm_mem_entry_init(dev, 0, resource_size(&res),
> +					   res.start, "vdev0buffer");> +
> +	if (!mem)
> +		return -ENOMEM;
> +
> +	rproc_add_carveout(rproc, mem);
> +
> +	return rproc_elf_load_rsc_table(rproc, fw);
> +}

Thanks for the patch, but this creates a kernel crash for me due to
overlaps with manually created carveouts. I currently have a single
memory-region and all allocations come from the same DMA pool, but the
rproc_of_resm_mem_entry_init() creates an overall mem entry without the
va being set (no alloc function plumbed in). In general, it is permitted
to use the same reserved-memory node with multiple devices, so the index
usage should have allowed it to do DMA allocations with vdev devices,
but the loading is performed even before the vdev allocations and the
da_to_va matches the first entry with no va set causing the crash.

Here's my debugfs output of the carveout_memories for reference,

Carveout memory entry:
        Name: vdev0buffer
        Virtual address: 00000000
        DMA address: 0x00000000
        Device address: 0xc3000000
        Length: 0x1000000 Bytes

Carveout memory entry:
        Name: vdev0vring0
        Virtual address: c3000000
        DMA address: 0xc3000000
        Device address: 0xc3000000
        Length: 0x3000 Bytes

Carveout memory entry:
        Name: vdev0vring1
        Virtual address: c3004000
        DMA address: 0xc3004000
        Device address: 0xc3004000
        Length: 0x3000 Bytes

Carveout memory entry:
        Name: DSP_MEM_DATA
        Virtual address: c3100000
        DMA address: 0xc3100000
        Device address: 0xc3100000
        Length: 0xf00000 Bytes

You can drop both this patch and the keystone_remoteproc patch from the
series. I did not run into any issues there since I did not have any
RSC_CARVEOUT entries there. Also, see my comments on the next patch (the
changes in ST) in general regarding these API. Looks like this needs
some more time in ironing out the issues.

regards
Suman



> +
>  static const struct rproc_ops da8xx_rproc_ops = {
>  	.start = da8xx_rproc_start,
>  	.stop = da8xx_rproc_stop,
>  	.kick = da8xx_rproc_kick,
> +	.parse_fw = da8xx_rproc_parse_fw,
> +	.load = rproc_elf_load_segments,
> +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> +	.sanity_check = rproc_elf_sanity_check,
> +	.get_boot_addr = rproc_elf_get_boot_addr,
>  };
>  
>  static int da8xx_rproc_get_internal_memories(struct platform_device *pdev,
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 16/17] remoteproc: st: add reserved memory support
  2018-07-27 13:14 ` [PATCH v4 16/17] remoteproc: st: add reserved memory support Loic Pallardy
@ 2018-10-24  3:01   ` Suman Anna
  2018-10-24 12:37     ` Loic PALLARDY
  0 siblings, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-24  3:01 UTC (permalink / raw)
  To: Loic Pallardy, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, arnaud.pouliquen, benjamin.gaignard

Hi Loic,

On 7/27/18 8:14 AM, Loic Pallardy wrote:
> ST remote processor needs some specified memory regions for
> firmware and IPC.
> Memory regions are defined as reserved memory and should
> be registered in remoteproc core thanks to rproc_add_carveout
> function before rproc_start. For this, st rproc driver implements
> prepare ops.
> 
> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> ---
>  drivers/remoteproc/st_remoteproc.c | 96 +++++++++++++++++++++++++++++++++-----
>  1 file changed, 85 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/remoteproc/st_remoteproc.c b/drivers/remoteproc/st_remoteproc.c
> index aacef0e..45958d5 100644
> --- a/drivers/remoteproc/st_remoteproc.c
> +++ b/drivers/remoteproc/st_remoteproc.c
> @@ -19,6 +19,7 @@
>  #include <linux/mfd/syscon.h>
>  #include <linux/module.h>
>  #include <linux/of.h>
> +#include <linux/of_address.h>
>  #include <linux/of_device.h>
>  #include <linux/of_reserved_mem.h>
>  #include <linux/platform_device.h>
> @@ -91,6 +92,82 @@ static void st_rproc_kick(struct rproc *rproc, int vqid)
>  		dev_err(dev, "failed to send message via mbox: %d\n", ret);
>  }
>  
> +static int st_rproc_mem_alloc(struct rproc *rproc,
> +			      struct rproc_mem_entry *mem)
> +{
> +	struct device *dev = rproc->dev.parent;
> +	void *va;
> +
> +	va = ioremap_wc(mem->dma, mem->len);
> +	if (!va) {
> +		dev_err(dev, "Unable to map memory region: %pa+%zx\n",
> +			&mem->dma, mem->len);
> +		return -ENOMEM;
> +	}
> +
> +	/* Update memory entry va */
> +	mem->va = va;
> +
> +	return 0;
> +}
> +
> +static int st_rproc_mem_release(struct rproc *rproc,
> +				struct rproc_mem_entry *mem)
> +{
> +	iounmap(mem->va);
> +
> +	return 0;
> +}
> +
> +static int st_rproc_parse_fw(struct rproc *rproc, const struct firmware *fw)
> +{
> +	struct device *dev = rproc->dev.parent;
> +	struct device_node *np = dev->of_node;
> +	struct rproc_mem_entry *mem;
> +	void *va;
> +	struct reserved_mem *rmem;
> +	struct of_phandle_iterator it;
> +	int err, index = 0;
> +
> +	of_phandle_iterator_init(&it, np, "memory-region", NULL, 0);
> +	while ((err = of_phandle_iterator_next(&it)) == 0) {
> +		va = NULL;
> +		rmem = of_reserved_mem_lookup(it.node);
> +
> +		/*  No need to map vdev buffer */
> +		if (strcmp(it.node->name, "vdev0buffer")) {
> +			va = devm_ioremap_wc(dev, rmem->base, rmem->size);
> +			if (!va) {
> +				dev_err(dev, "Unable to map memory region: %pa+%zx\n",
> +					&rmem->base, rmem->size);
> +				return -ENOMEM;
> +			}
> +
> +			/* Register memory region */
> +			mem = rproc_mem_entry_init(dev, va,
> +						   (dma_addr_t)rmem->base,
> +						   rmem->size, rmem->base,
> +						   st_rproc_mem_alloc,
> +						   st_rproc_mem_release,

You seem to be double-mapping, first just above to supply the va to
mem_entry_init and then through the alloc function.

This is DT parsing and is fixed irrespective of firmware and only needs
to be done once really in the platform driver probe. I think this whole
logic is unnecessarily complicated in the name of letting the remoteproc
core handle the alloc and free for fixed memory carveouts during every
boot and shutdown, and having to redo the mapping everytime during the
firmware parse. I am not a big fan of overloading the parse_fw to do
this runtime remapping everytime for something that should have been
done once. I think you have done this in your v2, and somewhere along in
v3 got modified into this.

regards
Suman

> +						   it.node->name);
> +		} else {
> +			/* Register reserved memory for vdev buffer allocation */
> +			mem = rproc_of_resm_mem_entry_init(dev, index,
> +							   rmem->size,
> +							   rmem->base,
> +							   it.node->name);
> +		}
> +
> +		if (!mem)
> +			return -ENOMEM;
> +
> +		rproc_add_carveout(rproc, mem);
> +		index++;
> +	}
> +
> +	return rproc_elf_load_rsc_table(rproc, fw);
> +}
> +
>  static int st_rproc_start(struct rproc *rproc)
>  {
>  	struct st_rproc *ddata = rproc->priv;
> @@ -158,9 +235,14 @@ static int st_rproc_stop(struct rproc *rproc)
>  }
>  
>  static const struct rproc_ops st_rproc_ops = {
> -	.kick		= st_rproc_kick,
> -	.start		= st_rproc_start,
> -	.stop		= st_rproc_stop,
> +	.kick			= st_rproc_kick,
> +	.start			= st_rproc_start,
> +	.stop			= st_rproc_stop,
> +	.parse_fw		= st_rproc_parse_fw,
> +	.load			= rproc_elf_load_segments,
> +	.find_loaded_rsc_table	= rproc_elf_find_loaded_rsc_table,
> +	.sanity_check		= rproc_elf_sanity_check,
> +	.get_boot_addr		= rproc_elf_get_boot_addr,
>  };
>  
>  /*
> @@ -254,12 +336,6 @@ static int st_rproc_parse_dt(struct platform_device *pdev)
>  		return -EINVAL;
>  	}
>  
> -	err = of_reserved_mem_device_init(dev);
> -	if (err) {
> -		dev_err(dev, "Failed to obtain shared memory\n");
> -		return err;
> -	}
> -
>  	err = clk_prepare(ddata->clk);
>  	if (err)
>  		dev_err(dev, "failed to get clock\n");
> @@ -387,8 +463,6 @@ static int st_rproc_remove(struct platform_device *pdev)
>  
>  	clk_disable_unprepare(ddata->clk);
>  
> -	of_reserved_mem_device_release(&pdev->dev);
> -
>  	for (i = 0; i < ST_RPROC_MAX_VRING * MBOX_MAX; i++)
>  		mbox_free_channel(ddata->mbox_chan[i]);
>  
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function
  2018-10-23 19:51     ` Loic PALLARDY
@ 2018-10-24  3:19       ` Suman Anna
  2018-10-24 12:58         ` Loic PALLARDY
  0 siblings, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-24  3:19 UTC (permalink / raw)
  To: Loic PALLARDY, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

On 10/23/18 2:51 PM, Loic PALLARDY wrote:
> Hi Suman,
> 
>> -----Original Message-----
>> From: Suman Anna <s-anna@ti.com>
>> Sent: mardi 23 octobre 2018 18:51
>> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
>> ohad@wizery.com
>> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
>> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
>> benjamin.gaignard@linaro.org
>> Subject: Re: [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function
>>
>> Hi Loic, Bjorn,
>>
>> On 7/27/18 8:14 AM, Loic Pallardy wrote:
>>> This new function translates CPU virtual address in
>>> CPU physical one according to virtual address location.
>>>
>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>> Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>
>>> ---
>>>  drivers/remoteproc/remoteproc_core.c | 18 +++++++++++++++++-
>>>  1 file changed, 17 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>> b/drivers/remoteproc/remoteproc_core.c
>>> index 437fabf..8e5fe1e 100644
>>> --- a/drivers/remoteproc/remoteproc_core.c
>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>> @@ -140,6 +140,22 @@ static void rproc_disable_iommu(struct rproc
>> *rproc)
>>>  	iommu_domain_free(domain);
>>>  }
>>>
>>> +static phys_addr_t rproc_va_to_pa(void *cpu_addr)
>>> +{
>>> +	/*
>>> +	 * Return physical address according to virtual address location
>>> +	 * - in vmalloc: if region ioremapped or defined as
>> dma_alloc_coherent
>>> +	 * - in kernel: if region allocated in generic dma memory pool
>>> +	 */
>>> +	if (is_vmalloc_addr(cpu_addr)) {
>>> +		return page_to_phys(vmalloc_to_page(cpu_addr)) +
>>> +				    offset_in_page(cpu_addr);
>>> +	}
>>> +
>>> +	WARN_ON(!virt_addr_valid(cpu_addr));
>>> +	return virt_to_phys(cpu_addr);
>>> +}
>>> +
>>>  /**
>>>   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc
>> address
>>>   * @rproc: handle of a remote processor
>>> @@ -711,7 +727,7 @@ static int rproc_handle_carveout(struct rproc
>> *rproc,
>>>  	 * In this case, the device address and the physical address
>>>  	 * are the same.
>>>  	 */
>>> -	rsc->pa = dma;
>>> +	rsc->pa = (u32)rproc_va_to_pa(va);
>>
>> While I agree with the direction here, we ought to add a check here
>> warning users if some address bits are getting lost as a result of the
>> typecast. Granted the issue may have been present previously with
>> dma_addr_t as well, but most platforms were using 32-bit dma addresses,
>> so this was kinda masked. There are ARMv7 platforms with LPAE enabled
>> allowing physical addresses > 32-bits.
>>
>> In anycase, we definitely have a need for a v2 for the fw_rsc_carveout
>> structure to deal with 64-bit addresses.
>>
> 
> Agree with you.
> Assumption for this series was to keep resource table as it is. Resource table improvement is planned in a second step.

Perhaps, we should add a WARN_ON for the time being until we enhance the
resource table for 64-bit platforms/addresses.

regards
Suman

> Regards,
> Loic
> 
>> regards
>> Suman
>>
>>>
>>>  	carveout->va = va;
>>>  	carveout->len = rsc->len;
>>>
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 01/17] remoteproc: configure IOMMU only if device address requested
  2018-10-23 19:40     ` Loic PALLARDY
@ 2018-10-24  3:46       ` Suman Anna
  2018-10-24 12:56         ` Loic PALLARDY
  0 siblings, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-24  3:46 UTC (permalink / raw)
  To: Loic PALLARDY, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

On 10/23/18 2:40 PM, Loic PALLARDY wrote:
> Hi Suman,
> 
>> -----Original Message-----
>> From: Suman Anna <s-anna@ti.com>
>> Sent: mardi 23 octobre 2018 19:26
>> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
>> ohad@wizery.com
>> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
>> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
>> benjamin.gaignard@linaro.org
>> Subject: Re: [PATCH v4 01/17] remoteproc: configure IOMMU only if device
>> address requested
>>
>> Hi Loic,
>>
>> On 7/27/18 8:14 AM, Loic Pallardy wrote:
>>> If there is no IOMMU associate to remote processor device,
>>> remoteproc_core won't be able to satisfy device address requested
>>> in firmware resource table.
>>> Return an error as configuration won't be coherent.
>>>
>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>
>> This patch is breaking my Davinci platforms. It is not really required
>> that you _should_ have IOMMUs when a valid DA is mentioned. Please see
>> the existing description (paras 4 and 5) on the fw_rsc_carveout
>> kerneldoc in remoteproc.h file.
> 
> Thanks for pointing this comment. Indeed sMMU is not mandatory, and at first sight I agree we should remove the restriction introduced by the patch.
> Driver porting on the series should be done before adding this.
>>
>> We do have platforms where we have some internal sub-modules within the
>> remote processor sub-system that provides some linear
>> address-translation (most common case with 32-bit processors supporting
>> 64-bit addresses). Also, we have some upcoming SoCs where we have an
>> MMU
>> but is not programmable by Linux.
>>
>> There is one comment there, but I don't think this is actually handled
>> in the current remoteproc core.
>> "If @da is set to
>>  * FW_RSC_ADDR_ANY, then the host will dynamically allocate it, and then
>>  * overwrite @da with the dynamically allocated address."
>>
> I don't remember it was implemented like described.

Yes, it was missing, and one of your patches seem to add this behavior
now. That said, I really don't think the remoteproc core can dictate the
 da. Even if the individual remoteproc driver were to furnish this, how
would you get such data without forcing a fixed behavior for all
possible firmwares (not desirable). We should get rid of this comment,
and any code that seems to do this.

> 
> I have remarks about the comment:
> "* We will always use @da to negotiate the device addresses, even if it
>  * isn't using an iommu. In that case, though, it will obviously contain
>  * physical addresses."
> 
> When there is no sMMU, we can't consider that da contains a physical address because coprocessor can have its own memory map just because it is a 32bit processor accessing only a part of the memory and the main is 64bit one. The 2 processors won't see the internal memory at the same base address for example.

Agreed, believe it was valid when it was written (32-bit platforms
supporting 32-bit addresses). I think this is akin to an IPA
(Intermediate Physical Address).

> So what should we do when carveout allocated by host is not fitting with resource table request?
> - put a warning and overwrite da address in the resource table?

Hmm, why da? This goes to my earlier comment about how you are able to
decide the da. Atleast your current ST driver seems to be assigning the
same value as the physical bus address for da, which would prompt why
you would still need a carveout entry in the resource table if it is
truly one-to-one.

Eg, I have an upcoming usecase with R5Fs on newer TI SoCs where we
actually have a sub-module called Region Address Translator (RAT) which
can only be programmed by the R5F for translating the 32-bit CPU
addresses to larger physical address space, and yet I need the da and pa
to be able to do loading. I cannot dictate the da since that is what the
firmware images are linked against. So, have to rely on the firmware
providing this data for me.

> - stop rproc probe as no match detected?

I think that is the safest approach.

> 
> Later in the series, carveout allocation is changed. Resource table carveout are either linked with an existing carveout registered by driver or added to carveout list for allocations.
> In the case you described, TI driver should first register the specific carveout regions thank to the helper.

The current series should still continue to work without having to
enforce new name assignments (unless needed and being defined to use the
new features being added).

> 
> Regards,
> Loic
> 
>> regards
>> Suman
>>
>>> ---
>>>  drivers/remoteproc/remoteproc_core.c | 10 +++++++++-
>>>  1 file changed, 9 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>> b/drivers/remoteproc/remoteproc_core.c
>>> index 4cd1a8e..437fabf 100644
>>> --- a/drivers/remoteproc/remoteproc_core.c
>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>> @@ -657,7 +657,15 @@ static int rproc_handle_carveout(struct rproc
>> *rproc,
>>>  	 * to use the iommu-based DMA API: we expect 'dma' to contain the
>>>  	 * physical address in this case.
>>>  	 */
>>> -	if (rproc->domain) {
>>> +
>>> +	if (rsc->da != FW_RSC_ADDR_ANY && !rproc->domain) {

Maybe this should really be a reverse check when you have MMUs,
if (rproc->domain && rsc->da == FW_RSC_ADDR_ANY)

regards
Suman

>>> +		dev_err(dev->parent,
>>> +			"Bad carveout rsc configuration\n");
>>> +		ret = -ENOMEM;
>>> +		goto dma_free;
>>> +	}
>>> +
>>> +	if (rsc->da != FW_RSC_ADDR_ANY && rproc->domain) {
>>>  		mapping = kzalloc(sizeof(*mapping), GFP_KERNEL);
>>>  		if (!mapping) {
>>>  			ret = -ENOMEM;
>>>
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 16/17] remoteproc: st: add reserved memory support
  2018-10-24  3:01   ` Suman Anna
@ 2018-10-24 12:37     ` Loic PALLARDY
  0 siblings, 0 replies; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-24 12:37 UTC (permalink / raw)
  To: Suman Anna, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

Hi Suman,

> -----Original Message-----
> From: Suman Anna <s-anna@ti.com>
> Sent: mercredi 24 octobre 2018 05:02
> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> ohad@wizery.com
> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> benjamin.gaignard@linaro.org
> Subject: Re: [PATCH v4 16/17] remoteproc: st: add reserved memory support
> 
> Hi Loic,
> 
> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> > ST remote processor needs some specified memory regions for
> > firmware and IPC.
> > Memory regions are defined as reserved memory and should
> > be registered in remoteproc core thanks to rproc_add_carveout
> > function before rproc_start. For this, st rproc driver implements
> > prepare ops.
> >
> > Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > ---
> >  drivers/remoteproc/st_remoteproc.c | 96
> +++++++++++++++++++++++++++++++++-----
> >  1 file changed, 85 insertions(+), 11 deletions(-)
> >
> > diff --git a/drivers/remoteproc/st_remoteproc.c
> b/drivers/remoteproc/st_remoteproc.c
> > index aacef0e..45958d5 100644
> > --- a/drivers/remoteproc/st_remoteproc.c
> > +++ b/drivers/remoteproc/st_remoteproc.c
> > @@ -19,6 +19,7 @@
> >  #include <linux/mfd/syscon.h>
> >  #include <linux/module.h>
> >  #include <linux/of.h>
> > +#include <linux/of_address.h>
> >  #include <linux/of_device.h>
> >  #include <linux/of_reserved_mem.h>
> >  #include <linux/platform_device.h>
> > @@ -91,6 +92,82 @@ static void st_rproc_kick(struct rproc *rproc, int vqid)
> >  		dev_err(dev, "failed to send message via mbox: %d\n", ret);
> >  }
> >
> > +static int st_rproc_mem_alloc(struct rproc *rproc,
> > +			      struct rproc_mem_entry *mem)
> > +{
> > +	struct device *dev = rproc->dev.parent;
> > +	void *va;
> > +
> > +	va = ioremap_wc(mem->dma, mem->len);
> > +	if (!va) {
> > +		dev_err(dev, "Unable to map memory region: %pa+%zx\n",
> > +			&mem->dma, mem->len);
> > +		return -ENOMEM;
> > +	}
> > +
> > +	/* Update memory entry va */
> > +	mem->va = va;
> > +
> > +	return 0;
> > +}
> > +
> > +static int st_rproc_mem_release(struct rproc *rproc,
> > +				struct rproc_mem_entry *mem)
> > +{
> > +	iounmap(mem->va);
> > +
> > +	return 0;
> > +}
> > +
> > +static int st_rproc_parse_fw(struct rproc *rproc, const struct firmware
> *fw)
> > +{
> > +	struct device *dev = rproc->dev.parent;
> > +	struct device_node *np = dev->of_node;
> > +	struct rproc_mem_entry *mem;
> > +	void *va;
> > +	struct reserved_mem *rmem;
> > +	struct of_phandle_iterator it;
> > +	int err, index = 0;
> > +
> > +	of_phandle_iterator_init(&it, np, "memory-region", NULL, 0);
> > +	while ((err = of_phandle_iterator_next(&it)) == 0) {
> > +		va = NULL;
> > +		rmem = of_reserved_mem_lookup(it.node);
> > +
> > +		/*  No need to map vdev buffer */
> > +		if (strcmp(it.node->name, "vdev0buffer")) {
> > +			va = devm_ioremap_wc(dev, rmem->base, rmem-
> >size);
> > +			if (!va) {
> > +				dev_err(dev, "Unable to map memory
> region: %pa+%zx\n",
> > +					&rmem->base, rmem->size);
> > +				return -ENOMEM;
> > +			}
> > +
> > +			/* Register memory region */
> > +			mem = rproc_mem_entry_init(dev, va,
> > +						   (dma_addr_t)rmem->base,
> > +						   rmem->size, rmem->base,
> > +						   st_rproc_mem_alloc,
> > +						   st_rproc_mem_release,
> 
> You seem to be double-mapping, first just above to supply the va to
> mem_entry_init and then through the alloc function.
> 
> This is DT parsing and is fixed irrespective of firmware and only needs
> to be done once really in the platform driver probe. I think this whole
> logic is unnecessarily complicated in the name of letting the remoteproc
> core handle the alloc and free for fixed memory carveouts during every
> boot and shutdown, and having to redo the mapping everytime during the
> firmware parse. I am not a big fan of overloading the parse_fw to do
> this runtime remapping everytime for something that should have been
> done once. I think you have done this in your v2, and somewhere along in
> v3 got modified into this.
 
Double mapping is definitively an error. Forget it.
Being able to allocate all carveouts at coprocessor start was a remark from Bjorn.
That's why I introduced alloc and free ops to do the allocation at each coprocessor start and so st_rproc_mem_alloc() in this driver.

Regards,
Loic
> 
> regards
> Suman
> 
> > +						   it.node->name);
> > +		} else {
> > +			/* Register reserved memory for vdev buffer
> allocation */
> > +			mem = rproc_of_resm_mem_entry_init(dev, index,
> > +							   rmem->size,
> > +							   rmem->base,
> > +							   it.node->name);
> > +		}
> > +
> > +		if (!mem)
> > +			return -ENOMEM;
> > +
> > +		rproc_add_carveout(rproc, mem);
> > +		index++;
> > +	}
> > +
> > +	return rproc_elf_load_rsc_table(rproc, fw);
> > +}
> > +
> >  static int st_rproc_start(struct rproc *rproc)
> >  {
> >  	struct st_rproc *ddata = rproc->priv;
> > @@ -158,9 +235,14 @@ static int st_rproc_stop(struct rproc *rproc)
> >  }
> >
> >  static const struct rproc_ops st_rproc_ops = {
> > -	.kick		= st_rproc_kick,
> > -	.start		= st_rproc_start,
> > -	.stop		= st_rproc_stop,
> > +	.kick			= st_rproc_kick,
> > +	.start			= st_rproc_start,
> > +	.stop			= st_rproc_stop,
> > +	.parse_fw		= st_rproc_parse_fw,
> > +	.load			= rproc_elf_load_segments,
> > +	.find_loaded_rsc_table	= rproc_elf_find_loaded_rsc_table,
> > +	.sanity_check		= rproc_elf_sanity_check,
> > +	.get_boot_addr		= rproc_elf_get_boot_addr,
> >  };
> >
> >  /*
> > @@ -254,12 +336,6 @@ static int st_rproc_parse_dt(struct
> platform_device *pdev)
> >  		return -EINVAL;
> >  	}
> >
> > -	err = of_reserved_mem_device_init(dev);
> > -	if (err) {
> > -		dev_err(dev, "Failed to obtain shared memory\n");
> > -		return err;
> > -	}
> > -
> >  	err = clk_prepare(ddata->clk);
> >  	if (err)
> >  		dev_err(dev, "failed to get clock\n");
> > @@ -387,8 +463,6 @@ static int st_rproc_remove(struct platform_device
> *pdev)
> >
> >  	clk_disable_unprepare(ddata->clk);
> >
> > -	of_reserved_mem_device_release(&pdev->dev);
> > -
> >  	for (i = 0; i < ST_RPROC_MAX_VRING * MBOX_MAX; i++)
> >  		mbox_free_channel(ddata->mbox_chan[i]);
> >
> >


^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-10-24  1:22         ` Suman Anna
  2018-10-24  1:48           ` Suman Anna
@ 2018-10-24 12:40           ` Loic PALLARDY
  2018-10-25 20:16             ` Suman Anna
  1 sibling, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-24 12:40 UTC (permalink / raw)
  To: Suman Anna, Wendy Liang
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, Arnaud POULIQUEN, benjamin.gaignard

Hi Suman,

> -----Original Message-----
> From: Suman Anna <s-anna@ti.com>
> Sent: mercredi 24 octobre 2018 03:22
> To: Wendy Liang <sunnyliangjy@gmail.com>; Loic PALLARDY
> <loic.pallardy@st.com>
> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
> <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; Linux Kernel
> Mailing List <linux-kernel@vger.kernel.org>; Arnaud POULIQUEN
> <arnaud.pouliquen@st.com>; benjamin.gaignard@linaro.org
> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
> specific dma memory pool
> 
> On 9/27/18 3:18 PM, Wendy Liang wrote:
> > Hi Loic,
> >
> >
> > On Thu, Sep 27, 2018 at 12:22 PM Loic PALLARDY <loic.pallardy@st.com>
> wrote:
> >>
> >> Hi Wendy
> >>
> >>> -----Original Message-----
> >>> From: Wendy Liang <sunnyliangjy@gmail.com>
> >>> Sent: Thursday, September 27, 2018 7:17 PM
> >>> To: Loic PALLARDY <loic.pallardy@st.com>
> >>> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
> >>> <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; Linux Kernel
> >>> Mailing List <linux-kernel@vger.kernel.org>; Arnaud POULIQUEN
> >>> <arnaud.pouliquen@st.com>; benjamin.gaignard@linaro.org; Suman
> Anna
> >>> <s-anna@ti.com>
> >>> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
> >>> specific dma memory pool
> >>>
> >>> On Fri, Jul 27, 2018 at 6:16 AM Loic Pallardy <loic.pallardy@st.com>
> wrote:
> >>>>
> >>>> This patch creates a dedicated vdev subdevice for each vdev declared
> >>>> in firmware resource table and associates carveout named
> "vdev%dbuffer"
> >>>> (with %d vdev index in resource table) if any as dma coherent memory
> >>> pool.
> >>>>
> >>>> Then vdev subdevice is used as parent for virtio device.
> >>>>
> >>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> >>>> ---
> >>>>  drivers/remoteproc/remoteproc_core.c     | 35
> >>> +++++++++++++++++++++++---
> >>>>  drivers/remoteproc/remoteproc_internal.h |  1 +
> >>>>  drivers/remoteproc/remoteproc_virtio.c   | 42
> >>> +++++++++++++++++++++++++++++++-
> >>>>  include/linux/remoteproc.h               |  1 +
> >>>>  4 files changed, 75 insertions(+), 4 deletions(-)
> >>>>
> >>>> diff --git a/drivers/remoteproc/remoteproc_core.c
> >>> b/drivers/remoteproc/remoteproc_core.c
> >>>> index 4edc6f0..adcc66e 100644
> >>>> --- a/drivers/remoteproc/remoteproc_core.c
> >>>> +++ b/drivers/remoteproc/remoteproc_core.c
> >>>> @@ -39,6 +39,7 @@
> >>>>  #include <linux/idr.h>
> >>>>  #include <linux/elf.h>
> >>>>  #include <linux/crc32.h>
> >>>> +#include <linux/of_reserved_mem.h>
> >>>>  #include <linux/virtio_ids.h>
> >>>>  #include <linux/virtio_ring.h>
> >>>>  #include <asm/byteorder.h>
> >>>> @@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc
> >>> *rproc)
> >>>>         iommu_domain_free(domain);
> >>>>  }
> >>>>
> >>>> -static phys_addr_t rproc_va_to_pa(void *cpu_addr)
> >>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr)
> >>>>  {
> >>>>         /*
> >>>>          * Return physical address according to virtual address location
> >>>> @@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void
> >>> *cpu_addr)
> >>>>         WARN_ON(!virt_addr_valid(cpu_addr));
> >>>>         return virt_to_phys(cpu_addr);
> >>>>  }
> >>>> +EXPORT_SYMBOL(rproc_va_to_pa);
> >>>>
> >>>>  /**
> >>>>   * rproc_da_to_va() - lookup the kernel virtual address for a
> remoteproc
> >>> address
> >>>> @@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct
> >>> rproc_subdev *subdev, bool crashed)
> >>>>  }
> >>>>
> >>>>  /**
> >>>> + * rproc_rvdev_release() - release the existence of a rvdev
> >>>> + *
> >>>> + * @dev: the subdevice's dev
> >>>> + */
> >>>> +static void rproc_rvdev_release(struct device *dev)
> >>>> +{
> >>>> +       struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev,
> dev);
> >>>> +
> >>>> +       of_reserved_mem_device_release(dev);
> >>>> +
> >>>> +       kfree(rvdev);
> >>>> +}
> >>>> +
> >>>> +/**
> >>>>   * rproc_handle_vdev() - handle a vdev fw resource
> >>>>   * @rproc: the remote processor
> >>>>   * @rsc: the vring resource descriptor
> >>>> @@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc
> *rproc,
> >>> struct fw_rsc_vdev *rsc,
> >>>>         struct device *dev = &rproc->dev;
> >>>>         struct rproc_vdev *rvdev;
> >>>>         int i, ret;
> >>>> +       char name[16];
> >>>>
> >>>>         /* make sure resource isn't truncated */
> >>>>         if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct
> >>> fw_rsc_vdev_vring)
> >>>> @@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc
> *rproc,
> >>> struct fw_rsc_vdev *rsc,
> >>>>         rvdev->rproc = rproc;
> >>>>         rvdev->index = rproc->nb_vdev++;
> >>>>
> >>>> +       /* Initialise vdev subdevice */
> >>>> +       snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
> >>>> +       rvdev->dev.parent = rproc->dev.parent;
> >>>> +       rvdev->dev.release = rproc_rvdev_release;
> >>>> +       dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev-
> >>>> dev.parent), name);
> >>>> +       dev_set_drvdata(&rvdev->dev, rvdev);
> >>>> +       dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
> >>> I tried the latest kernel, this function will not set the DMA coherent mask
> as
> >>> dma_supported() of the &rvdev->dev will return false.
> >>> As this is a device created at run time, should it be force to support
> DMA?
> >>> should it directly set the dma_coherent_mask?
> >>
> >> Thanks for pointing me this issue. I tested on top of 4.18-rc1 few months
> ago...
> >> Could you please give me kernel version on which you are testing the
> series?
> >> Is you platform 32bit or 64bit ?
> >> I'll rebase and check on my side.
> >
> > I am testing with 4.19-rc4 on aarch64 platform.
> 
> Btw, I ran into this on my v7 platform as well (4.19-rc6). The
> dma_set_coherent_mask fails with error EIO. I did get my allocations
> through though.

I don't have issue on v7 platform. Anyway I have now an Xilinx Ultra96 board running on my desk. It looks like vdev device doesn't have dma support, so not possible to set mask.
Need to continue the investigations...
Regards,
Loic
> 
> regards
> Suman
> 
> >
> > Best Regards,
> > Wendy
> >>
> >> Regards,
> >> Loic
> >>
> >>>
> >>>> +
> >>>> +       ret = device_register(&rvdev->dev);
> >>>> +       if (ret)
> >>>> +               goto free_rvdev;
> >>>> +
> >>>>         /* parse the vrings */
> >>>>         for (i = 0; i < rsc->num_of_vrings; i++) {
> >>>>                 ret = rproc_parse_vring(rvdev, rsc, i);
> >>>> @@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc
> *rproc,
> >>> struct fw_rsc_vdev *rsc,
> >>>>         for (i--; i >= 0; i--)
> >>>>                 rproc_free_vring(&rvdev->vring[i]);
> >>>>  free_rvdev:
> >>>> -       kfree(rvdev);
> >>>> +       device_unregister(&rvdev->dev);
> >>>>         return ret;
> >>>>  }
> >>>>
> >>>> @@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
> >>>>
> >>>>         rproc_remove_subdev(rproc, &rvdev->subdev);
> >>>>         list_del(&rvdev->node);
> >>>> -       kfree(rvdev);
> >>>> +       device_unregister(&rvdev->dev);
> >>>>  }
> >>>>
> >>>>  /**
> >>>> diff --git a/drivers/remoteproc/remoteproc_internal.h
> >>> b/drivers/remoteproc/remoteproc_internal.h
> >>>> index f6cad24..bfeacfd 100644
> >>>> --- a/drivers/remoteproc/remoteproc_internal.h
> >>>> +++ b/drivers/remoteproc/remoteproc_internal.h
> >>>> @@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const char
> >>> *name, struct rproc *rproc,
> >>>>  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
> >>>>
> >>>>  void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
> >>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr);
> >>>>  int rproc_trigger_recovery(struct rproc *rproc);
> >>>>
> >>>>  int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware
> *fw);
> >>>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
> >>> b/drivers/remoteproc/remoteproc_virtio.c
> >>>> index de21f62..9ee63c0 100644
> >>>> --- a/drivers/remoteproc/remoteproc_virtio.c
> >>>> +++ b/drivers/remoteproc/remoteproc_virtio.c
> >>>> @@ -17,7 +17,9 @@
> >>>>   * GNU General Public License for more details.
> >>>>   */
> >>>>
> >>>> +#include <linux/dma-mapping.h>
> >>>>  #include <linux/export.h>
> >>>> +#include <linux/of_reserved_mem.h>
> >>>>  #include <linux/remoteproc.h>
> >>>>  #include <linux/virtio.h>
> >>>>  #include <linux/virtio_config.h>
> >>>> @@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct
> device
> >>> *dev)
> >>>>  int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
> >>>>  {
> >>>>         struct rproc *rproc = rvdev->rproc;
> >>>> -       struct device *dev = &rproc->dev;
> >>>> +       struct device *dev = &rvdev->dev;
> >>>>         struct virtio_device *vdev = &rvdev->vdev;
> >>>> +       struct rproc_mem_entry *mem;
> >>>>         int ret;
> >>>>
> >>>> +       /* Try to find dedicated vdev buffer carveout */
> >>>> +       mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer",
> rvdev-
> >>>> index);
> >>>> +       if (mem) {
> >>>> +               phys_addr_t pa;
> >>>> +
> >>>> +               if (mem->of_resm_idx != -1) {
> >>>> +                       struct device_node *np = rproc->dev.parent->of_node;
> >>>> +
> >>>> +                       /* Associate reserved memory to vdev device */
> >>>> +                       ret = of_reserved_mem_device_init_by_idx(dev, np,
> >>>> +                                                                mem->of_resm_idx);
> >>>> +                       if (ret) {
> >>>> +                               dev_err(dev, "Can't associate reserved memory\n");
> >>>> +                               goto out;
> >>>> +                       }
> >>>> +               } else {
> >>>> +                       if (mem->va) {
> >>>> +                               dev_warn(dev, "vdev %d buffer already mapped\n",
> >>>> +                                        rvdev->index);
> >>>> +                               pa = rproc_va_to_pa(mem->va);
> >>>> +                       } else {
> >>>> +                               /* Use dma address as carveout no memmapped yet
> */
> >>>> +                               pa = (phys_addr_t)mem->dma;
> >>>> +                       }
> >>>> +
> >>>> +                       /* Associate vdev buffer memory pool to vdev subdev */
> >>>> +                       ret = dmam_declare_coherent_memory(dev, pa,
> >>>> +                                                          mem->da,
> >>>> +                                                          mem->len,
> >>>> +                                                          DMA_MEMORY_EXCLUSIVE);
> >>>> +                       if (ret < 0) {
> >>>> +                               dev_err(dev, "Failed to associate buffer\n");
> >>>> +                               goto out;
> >>>> +                       }
> >>>> +               }
> >>>> +       }
> >>>> +
> >>>>         vdev->id.device = id,
> >>>>         vdev->config = &rproc_virtio_config_ops,
> >>>>         vdev->dev.parent = dev;
> >>>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> >>>> index 6b3a234..2921dd2 100644
> >>>> --- a/include/linux/remoteproc.h
> >>>> +++ b/include/linux/remoteproc.h
> >>>> @@ -547,6 +547,7 @@ struct rproc_vdev {
> >>>>         struct kref refcount;
> >>>>
> >>>>         struct rproc_subdev subdev;
> >>>> +       struct device dev;
> >>>>
> >>>>         unsigned int id;
> >>>>         struct list_head node;
> >>>> --
> >>>> 1.9.1
> >>>>
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe linux-
> remoteproc"
> >>> in
> >>>> the body of a message to majordomo@vger.kernel.org
> >>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html


^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-10-24  1:48           ` Suman Anna
@ 2018-10-24 12:42             ` Loic PALLARDY
  2018-10-25 22:06               ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-24 12:42 UTC (permalink / raw)
  To: Suman Anna, Wendy Liang
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, Arnaud POULIQUEN, benjamin.gaignard



> -----Original Message-----
> From: Suman Anna <s-anna@ti.com>
> Sent: mercredi 24 octobre 2018 03:49
> To: Wendy Liang <sunnyliangjy@gmail.com>; Loic PALLARDY
> <loic.pallardy@st.com>
> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
> <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; Linux Kernel
> Mailing List <linux-kernel@vger.kernel.org>; Arnaud POULIQUEN
> <arnaud.pouliquen@st.com>; benjamin.gaignard@linaro.org
> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
> specific dma memory pool
> 
> On 10/23/18 8:22 PM, Suman Anna wrote:
> > On 9/27/18 3:18 PM, Wendy Liang wrote:
> >> Hi Loic,
> >>
> >>
> >> On Thu, Sep 27, 2018 at 12:22 PM Loic PALLARDY <loic.pallardy@st.com>
> wrote:
> >>>
> >>> Hi Wendy
> >>>
> >>>> -----Original Message-----
> >>>> From: Wendy Liang <sunnyliangjy@gmail.com>
> >>>> Sent: Thursday, September 27, 2018 7:17 PM
> >>>> To: Loic PALLARDY <loic.pallardy@st.com>
> >>>> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
> >>>> <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; Linux Kernel
> >>>> Mailing List <linux-kernel@vger.kernel.org>; Arnaud POULIQUEN
> >>>> <arnaud.pouliquen@st.com>; benjamin.gaignard@linaro.org; Suman
> Anna
> >>>> <s-anna@ti.com>
> >>>> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
> >>>> specific dma memory pool
> >>>>
> >>>> On Fri, Jul 27, 2018 at 6:16 AM Loic Pallardy <loic.pallardy@st.com>
> wrote:
> >>>>>
> >>>>> This patch creates a dedicated vdev subdevice for each vdev declared
> >>>>> in firmware resource table and associates carveout named
> "vdev%dbuffer"
> >>>>> (with %d vdev index in resource table) if any as dma coherent
> memory
> >>>> pool.
> >>>>>
> >>>>> Then vdev subdevice is used as parent for virtio device.
> >>>>>
> >>>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> >>>>> ---
> >>>>>  drivers/remoteproc/remoteproc_core.c     | 35
> >>>> +++++++++++++++++++++++---
> >>>>>  drivers/remoteproc/remoteproc_internal.h |  1 +
> >>>>>  drivers/remoteproc/remoteproc_virtio.c   | 42
> >>>> +++++++++++++++++++++++++++++++-
> >>>>>  include/linux/remoteproc.h               |  1 +
> >>>>>  4 files changed, 75 insertions(+), 4 deletions(-)
> >>>>>
> >>>>> diff --git a/drivers/remoteproc/remoteproc_core.c
> >>>> b/drivers/remoteproc/remoteproc_core.c
> >>>>> index 4edc6f0..adcc66e 100644
> >>>>> --- a/drivers/remoteproc/remoteproc_core.c
> >>>>> +++ b/drivers/remoteproc/remoteproc_core.c
> >>>>> @@ -39,6 +39,7 @@
> >>>>>  #include <linux/idr.h>
> >>>>>  #include <linux/elf.h>
> >>>>>  #include <linux/crc32.h>
> >>>>> +#include <linux/of_reserved_mem.h>
> >>>>>  #include <linux/virtio_ids.h>
> >>>>>  #include <linux/virtio_ring.h>
> >>>>>  #include <asm/byteorder.h>
> >>>>> @@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc
> >>>> *rproc)
> >>>>>         iommu_domain_free(domain);
> >>>>>  }
> >>>>>
> >>>>> -static phys_addr_t rproc_va_to_pa(void *cpu_addr)
> >>>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr)
> >>>>>  {
> >>>>>         /*
> >>>>>          * Return physical address according to virtual address location
> >>>>> @@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void
> >>>> *cpu_addr)
> >>>>>         WARN_ON(!virt_addr_valid(cpu_addr));
> >>>>>         return virt_to_phys(cpu_addr);
> >>>>>  }
> >>>>> +EXPORT_SYMBOL(rproc_va_to_pa);
> >>>>>
> >>>>>  /**
> >>>>>   * rproc_da_to_va() - lookup the kernel virtual address for a
> remoteproc
> >>>> address
> >>>>> @@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct
> >>>> rproc_subdev *subdev, bool crashed)
> >>>>>  }
> >>>>>
> >>>>>  /**
> >>>>> + * rproc_rvdev_release() - release the existence of a rvdev
> >>>>> + *
> >>>>> + * @dev: the subdevice's dev
> >>>>> + */
> >>>>> +static void rproc_rvdev_release(struct device *dev)
> >>>>> +{
> >>>>> +       struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev,
> dev);
> >>>>> +
> >>>>> +       of_reserved_mem_device_release(dev);
> >>>>> +
> >>>>> +       kfree(rvdev);
> >>>>> +}
> >>>>> +
> >>>>> +/**
> >>>>>   * rproc_handle_vdev() - handle a vdev fw resource
> >>>>>   * @rproc: the remote processor
> >>>>>   * @rsc: the vring resource descriptor
> >>>>> @@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc
> *rproc,
> >>>> struct fw_rsc_vdev *rsc,
> >>>>>         struct device *dev = &rproc->dev;
> >>>>>         struct rproc_vdev *rvdev;
> >>>>>         int i, ret;
> >>>>> +       char name[16];
> >>>>>
> >>>>>         /* make sure resource isn't truncated */
> >>>>>         if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct
> >>>> fw_rsc_vdev_vring)
> >>>>> @@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc
> *rproc,
> >>>> struct fw_rsc_vdev *rsc,
> >>>>>         rvdev->rproc = rproc;
> >>>>>         rvdev->index = rproc->nb_vdev++;
> >>>>>
> >>>>> +       /* Initialise vdev subdevice */
> >>>>> +       snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
> >>>>> +       rvdev->dev.parent = rproc->dev.parent;
> >>>>> +       rvdev->dev.release = rproc_rvdev_release;
> >>>>> +       dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev-
> >>>>> dev.parent), name);
> >>>>> +       dev_set_drvdata(&rvdev->dev, rvdev);
> >>>>> +       dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
> >>>> I tried the latest kernel, this function will not set the DMA coherent
> mask as
> >>>> dma_supported() of the &rvdev->dev will return false.
> >>>> As this is a device created at run time, should it be force to support
> DMA?
> >>>> should it directly set the dma_coherent_mask?
> >>>
> >>> Thanks for pointing me this issue. I tested on top of 4.18-rc1 few months
> ago...
> >>> Could you please give me kernel version on which you are testing the
> series?
> >>> Is you platform 32bit or 64bit ?
> >>> I'll rebase and check on my side.
> >>
> >> I am testing with 4.19-rc4 on aarch64 platform.
> >
> > Btw, I ran into this on my v7 platform as well (4.19-rc6). The
> > dma_set_coherent_mask fails with error EIO. I did get my allocations
> > through though.
> 
> Correction, that was before Patch 17. With patch 17, this fails.

Yes normal as device for allocation is changed Patch17. 
Regards,
Loic
> 
> regards
> Suman
> 
> >
> > regards
> > Suman
> >
> >>
> >> Best Regards,
> >> Wendy
> >>>
> >>> Regards,
> >>> Loic
> >>>
> >>>>
> >>>>> +
> >>>>> +       ret = device_register(&rvdev->dev);
> >>>>> +       if (ret)
> >>>>> +               goto free_rvdev;
> >>>>> +
> >>>>>         /* parse the vrings */
> >>>>>         for (i = 0; i < rsc->num_of_vrings; i++) {
> >>>>>                 ret = rproc_parse_vring(rvdev, rsc, i);
> >>>>> @@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc
> *rproc,
> >>>> struct fw_rsc_vdev *rsc,
> >>>>>         for (i--; i >= 0; i--)
> >>>>>                 rproc_free_vring(&rvdev->vring[i]);
> >>>>>  free_rvdev:
> >>>>> -       kfree(rvdev);
> >>>>> +       device_unregister(&rvdev->dev);
> >>>>>         return ret;
> >>>>>  }
> >>>>>
> >>>>> @@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
> >>>>>
> >>>>>         rproc_remove_subdev(rproc, &rvdev->subdev);
> >>>>>         list_del(&rvdev->node);
> >>>>> -       kfree(rvdev);
> >>>>> +       device_unregister(&rvdev->dev);
> >>>>>  }
> >>>>>
> >>>>>  /**
> >>>>> diff --git a/drivers/remoteproc/remoteproc_internal.h
> >>>> b/drivers/remoteproc/remoteproc_internal.h
> >>>>> index f6cad24..bfeacfd 100644
> >>>>> --- a/drivers/remoteproc/remoteproc_internal.h
> >>>>> +++ b/drivers/remoteproc/remoteproc_internal.h
> >>>>> @@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const
> char
> >>>> *name, struct rproc *rproc,
> >>>>>  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
> >>>>>
> >>>>>  void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
> >>>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr);
> >>>>>  int rproc_trigger_recovery(struct rproc *rproc);
> >>>>>
> >>>>>  int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware
> *fw);
> >>>>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
> >>>> b/drivers/remoteproc/remoteproc_virtio.c
> >>>>> index de21f62..9ee63c0 100644
> >>>>> --- a/drivers/remoteproc/remoteproc_virtio.c
> >>>>> +++ b/drivers/remoteproc/remoteproc_virtio.c
> >>>>> @@ -17,7 +17,9 @@
> >>>>>   * GNU General Public License for more details.
> >>>>>   */
> >>>>>
> >>>>> +#include <linux/dma-mapping.h>
> >>>>>  #include <linux/export.h>
> >>>>> +#include <linux/of_reserved_mem.h>
> >>>>>  #include <linux/remoteproc.h>
> >>>>>  #include <linux/virtio.h>
> >>>>>  #include <linux/virtio_config.h>
> >>>>> @@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct
> device
> >>>> *dev)
> >>>>>  int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
> >>>>>  {
> >>>>>         struct rproc *rproc = rvdev->rproc;
> >>>>> -       struct device *dev = &rproc->dev;
> >>>>> +       struct device *dev = &rvdev->dev;
> >>>>>         struct virtio_device *vdev = &rvdev->vdev;
> >>>>> +       struct rproc_mem_entry *mem;
> >>>>>         int ret;
> >>>>>
> >>>>> +       /* Try to find dedicated vdev buffer carveout */
> >>>>> +       mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer",
> rvdev-
> >>>>> index);
> >>>>> +       if (mem) {
> >>>>> +               phys_addr_t pa;
> >>>>> +
> >>>>> +               if (mem->of_resm_idx != -1) {
> >>>>> +                       struct device_node *np = rproc->dev.parent->of_node;
> >>>>> +
> >>>>> +                       /* Associate reserved memory to vdev device */
> >>>>> +                       ret = of_reserved_mem_device_init_by_idx(dev, np,
> >>>>> +                                                                mem->of_resm_idx);
> >>>>> +                       if (ret) {
> >>>>> +                               dev_err(dev, "Can't associate reserved memory\n");
> >>>>> +                               goto out;
> >>>>> +                       }
> >>>>> +               } else {
> >>>>> +                       if (mem->va) {
> >>>>> +                               dev_warn(dev, "vdev %d buffer already mapped\n",
> >>>>> +                                        rvdev->index);
> >>>>> +                               pa = rproc_va_to_pa(mem->va);
> >>>>> +                       } else {
> >>>>> +                               /* Use dma address as carveout no memmapped yet
> */
> >>>>> +                               pa = (phys_addr_t)mem->dma;
> >>>>> +                       }
> >>>>> +
> >>>>> +                       /* Associate vdev buffer memory pool to vdev subdev */
> >>>>> +                       ret = dmam_declare_coherent_memory(dev, pa,
> >>>>> +                                                          mem->da,
> >>>>> +                                                          mem->len,
> >>>>> +                                                          DMA_MEMORY_EXCLUSIVE);
> >>>>> +                       if (ret < 0) {
> >>>>> +                               dev_err(dev, "Failed to associate buffer\n");
> >>>>> +                               goto out;
> >>>>> +                       }
> >>>>> +               }
> >>>>> +       }
> >>>>> +
> >>>>>         vdev->id.device = id,
> >>>>>         vdev->config = &rproc_virtio_config_ops,
> >>>>>         vdev->dev.parent = dev;
> >>>>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> >>>>> index 6b3a234..2921dd2 100644
> >>>>> --- a/include/linux/remoteproc.h
> >>>>> +++ b/include/linux/remoteproc.h
> >>>>> @@ -547,6 +547,7 @@ struct rproc_vdev {
> >>>>>         struct kref refcount;
> >>>>>
> >>>>>         struct rproc_subdev subdev;
> >>>>> +       struct device dev;
> >>>>>
> >>>>>         unsigned int id;
> >>>>>         struct list_head node;
> >>>>> --
> >>>>> 1.9.1
> >>>>>
> >>>>> --
> >>>>> To unsubscribe from this list: send the line "unsubscribe linux-
> remoteproc"
> >>>> in
> >>>>> the body of a message to majordomo@vger.kernel.org
> >>>>> More majordomo info at  http://vger.kernel.org/majordomo-
> info.html
> >


^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 01/17] remoteproc: configure IOMMU only if device address requested
  2018-10-24  3:46       ` Suman Anna
@ 2018-10-24 12:56         ` Loic PALLARDY
  2018-10-26  0:46           ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-24 12:56 UTC (permalink / raw)
  To: Suman Anna, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard



> -----Original Message-----
> From: Suman Anna <s-anna@ti.com>
> Sent: mercredi 24 octobre 2018 05:47
> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> ohad@wizery.com
> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> benjamin.gaignard@linaro.org
> Subject: Re: [PATCH v4 01/17] remoteproc: configure IOMMU only if device
> address requested
> 
> On 10/23/18 2:40 PM, Loic PALLARDY wrote:
> > Hi Suman,
> >
> >> -----Original Message-----
> >> From: Suman Anna <s-anna@ti.com>
> >> Sent: mardi 23 octobre 2018 19:26
> >> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> >> ohad@wizery.com
> >> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> >> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> >> benjamin.gaignard@linaro.org
> >> Subject: Re: [PATCH v4 01/17] remoteproc: configure IOMMU only if
> device
> >> address requested
> >>
> >> Hi Loic,
> >>
> >> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> >>> If there is no IOMMU associate to remote processor device,
> >>> remoteproc_core won't be able to satisfy device address requested
> >>> in firmware resource table.
> >>> Return an error as configuration won't be coherent.
> >>>
> >>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> >>
> >> This patch is breaking my Davinci platforms. It is not really required
> >> that you _should_ have IOMMUs when a valid DA is mentioned. Please
> see
> >> the existing description (paras 4 and 5) on the fw_rsc_carveout
> >> kerneldoc in remoteproc.h file.
> >
> > Thanks for pointing this comment. Indeed sMMU is not mandatory, and at
> first sight I agree we should remove the restriction introduced by the patch.
> > Driver porting on the series should be done before adding this.
> >>
> >> We do have platforms where we have some internal sub-modules within
> the
> >> remote processor sub-system that provides some linear
> >> address-translation (most common case with 32-bit processors supporting
> >> 64-bit addresses). Also, we have some upcoming SoCs where we have an
> >> MMU
> >> but is not programmable by Linux.
> >>
> >> There is one comment there, but I don't think this is actually handled
> >> in the current remoteproc core.
> >> "If @da is set to
> >>  * FW_RSC_ADDR_ANY, then the host will dynamically allocate it, and
> then
> >>  * overwrite @da with the dynamically allocated address."
> >>
> > I don't remember it was implemented like described.
> 
> Yes, it was missing, and one of your patches seem to add this behavior
> now. That said, I really don't think the remoteproc core can dictate the
>  da. Even if the individual remoteproc driver were to furnish this, how
> would you get such data without forcing a fixed behavior for all
> possible firmwares (not desirable). We should get rid of this comment,
> and any code that seems to do this.

Agree that if some rules are fixed by firmware, rproc core can't overwrite them.
It was the goal of the patch as without sMMU if you don't have a match between rproc core carveout allocation and firmware rsc request, rproc core can't answer the request.
In the rest of the series, da in rsc table is updated only in some specific cases (i.e. da is equal to FW_RSC_ADDR_ANY). If there is a force update I agree we should not allow it.
> 
> >
> > I have remarks about the comment:
> > "* We will always use @da to negotiate the device addresses, even if it
> >  * isn't using an iommu. In that case, though, it will obviously contain
> >  * physical addresses."
> >
> > When there is no sMMU, we can't consider that da contains a physical
> address because coprocessor can have its own memory map just because it
> is a 32bit processor accessing only a part of the memory and the main is 64bit
> one. The 2 processors won't see the internal memory at the same base
> address for example.
> 
> Agreed, believe it was valid when it was written (32-bit platforms
> supporting 32-bit addresses). I think this is akin to an IPA
> (Intermediate Physical Address).
> 
> > So what should we do when carveout allocated by host is not fitting with
> resource table request?
> > - put a warning and overwrite da address in the resource table?
> 
> Hmm, why da? This goes to my earlier comment about how you are able to
> decide the da. Atleast your current ST driver seems to be assigning the
> same value as the physical bus address for da, which would prompt why
> you would still need a carveout entry in the resource table if it is
> truly one-to-one.
> 
> Eg, I have an upcoming usecase with R5Fs on newer TI SoCs where we
> actually have a sub-module called Region Address Translator (RAT) which
> can only be programmed by the R5F for translating the 32-bit CPU
> addresses to larger physical address space, and yet I need the da and pa
> to be able to do loading. I cannot dictate the da since that is what the
> firmware images are linked against. So, have to rely on the firmware
> providing this data for me.
> 
> > - stop rproc probe as no match detected?
> 
> I think that is the safest approach.

So we agree on the patch if a WARN is added and comment updated.
Are we aligned ?

> 
> >
> > Later in the series, carveout allocation is changed. Resource table carveout
> are either linked with an existing carveout registered by driver or added to
> carveout list for allocations.
> > In the case you described, TI driver should first register the specific
> carveout regions thank to the helper.
> 
> The current series should still continue to work without having to
> enforce new name assignments (unless needed and being defined to use
> the
> new features being added).
> 
OK but if we implement above solution, if da is different from "dma" address (from dma_alloc_coherent) and no sMMU registered, rproc start procedure will failed.

Regards,
Loic
> >
> > Regards,
> > Loic
> >
> >> regards
> >> Suman
> >>
> >>> ---
> >>>  drivers/remoteproc/remoteproc_core.c | 10 +++++++++-
> >>>  1 file changed, 9 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/drivers/remoteproc/remoteproc_core.c
> >> b/drivers/remoteproc/remoteproc_core.c
> >>> index 4cd1a8e..437fabf 100644
> >>> --- a/drivers/remoteproc/remoteproc_core.c
> >>> +++ b/drivers/remoteproc/remoteproc_core.c
> >>> @@ -657,7 +657,15 @@ static int rproc_handle_carveout(struct rproc
> >> *rproc,
> >>>  	 * to use the iommu-based DMA API: we expect 'dma' to contain the
> >>>  	 * physical address in this case.
> >>>  	 */
> >>> -	if (rproc->domain) {
> >>> +
> >>> +	if (rsc->da != FW_RSC_ADDR_ANY && !rproc->domain) {
> 
> Maybe this should really be a reverse check when you have MMUs,
> if (rproc->domain && rsc->da == FW_RSC_ADDR_ANY)
> 
> regards
> Suman
> 
> >>> +		dev_err(dev->parent,
> >>> +			"Bad carveout rsc configuration\n");
> >>> +		ret = -ENOMEM;
> >>> +		goto dma_free;
> >>> +	}
> >>> +
> >>> +	if (rsc->da != FW_RSC_ADDR_ANY && rproc->domain) {
> >>>  		mapping = kzalloc(sizeof(*mapping), GFP_KERNEL);
> >>>  		if (!mapping) {
> >>>  			ret = -ENOMEM;
> >>>
> >


^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function
  2018-10-24  3:19       ` Suman Anna
@ 2018-10-24 12:58         ` Loic PALLARDY
  2018-10-25 22:50           ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-24 12:58 UTC (permalink / raw)
  To: Suman Anna, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard



> -----Original Message-----
> From: Suman Anna <s-anna@ti.com>
> Sent: mercredi 24 octobre 2018 05:19
> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> ohad@wizery.com
> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> benjamin.gaignard@linaro.org
> Subject: Re: [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function
> 
> On 10/23/18 2:51 PM, Loic PALLARDY wrote:
> > Hi Suman,
> >
> >> -----Original Message-----
> >> From: Suman Anna <s-anna@ti.com>
> >> Sent: mardi 23 octobre 2018 18:51
> >> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> >> ohad@wizery.com
> >> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> >> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> >> benjamin.gaignard@linaro.org
> >> Subject: Re: [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function
> >>
> >> Hi Loic, Bjorn,
> >>
> >> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> >>> This new function translates CPU virtual address in
> >>> CPU physical one according to virtual address location.
> >>>
> >>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> >>> Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>
> >>> ---
> >>>  drivers/remoteproc/remoteproc_core.c | 18 +++++++++++++++++-
> >>>  1 file changed, 17 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/drivers/remoteproc/remoteproc_core.c
> >> b/drivers/remoteproc/remoteproc_core.c
> >>> index 437fabf..8e5fe1e 100644
> >>> --- a/drivers/remoteproc/remoteproc_core.c
> >>> +++ b/drivers/remoteproc/remoteproc_core.c
> >>> @@ -140,6 +140,22 @@ static void rproc_disable_iommu(struct rproc
> >> *rproc)
> >>>  	iommu_domain_free(domain);
> >>>  }
> >>>
> >>> +static phys_addr_t rproc_va_to_pa(void *cpu_addr)
> >>> +{
> >>> +	/*
> >>> +	 * Return physical address according to virtual address location
> >>> +	 * - in vmalloc: if region ioremapped or defined as
> >> dma_alloc_coherent
> >>> +	 * - in kernel: if region allocated in generic dma memory pool
> >>> +	 */
> >>> +	if (is_vmalloc_addr(cpu_addr)) {
> >>> +		return page_to_phys(vmalloc_to_page(cpu_addr)) +
> >>> +				    offset_in_page(cpu_addr);
> >>> +	}
> >>> +
> >>> +	WARN_ON(!virt_addr_valid(cpu_addr));
> >>> +	return virt_to_phys(cpu_addr);
> >>> +}
> >>> +
> >>>  /**
> >>>   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc
> >> address
> >>>   * @rproc: handle of a remote processor
> >>> @@ -711,7 +727,7 @@ static int rproc_handle_carveout(struct rproc
> >> *rproc,
> >>>  	 * In this case, the device address and the physical address
> >>>  	 * are the same.
> >>>  	 */
> >>> -	rsc->pa = dma;
> >>> +	rsc->pa = (u32)rproc_va_to_pa(va);
> >>
> >> While I agree with the direction here, we ought to add a check here
> >> warning users if some address bits are getting lost as a result of the
> >> typecast. Granted the issue may have been present previously with
> >> dma_addr_t as well, but most platforms were using 32-bit dma addresses,
> >> so this was kinda masked. There are ARMv7 platforms with LPAE enabled
> >> allowing physical addresses > 32-bits.
> >>
> >> In anycase, we definitely have a need for a v2 for the fw_rsc_carveout
> >> structure to deal with 64-bit addresses.
> >>
> >
> > Agree with you.
> > Assumption for this series was to keep resource table as it is. Resource
> table improvement is planned in a second step.
> 
> Perhaps, we should add a WARN_ON for the time being until we enhance
> the
> resource table for 64-bit platforms/addresses.

OK I will propose a patch to add WARN_ON on cast applied on resource table field
Regards,
Loic
> 
> regards
> Suman
> 
> > Regards,
> > Loic
> >
> >> regards
> >> Suman
> >>
> >>>
> >>>  	carveout->va = va;
> >>>  	carveout->len = rsc->len;
> >>>
> >


^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 15/17] remoteproc: da8xx: declare reserved memory region for vdev device
  2018-10-24  2:57   ` Suman Anna
@ 2018-10-24 13:19     ` Loic PALLARDY
  2018-10-25 22:11       ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-24 13:19 UTC (permalink / raw)
  To: Suman Anna, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard



> -----Original Message-----
> From: Suman Anna <s-anna@ti.com>
> Sent: mercredi 24 octobre 2018 04:58
> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> ohad@wizery.com
> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> benjamin.gaignard@linaro.org
> Subject: Re: [PATCH v4 15/17] remoteproc: da8xx: declare reserved memory
> region for vdev device
> 
> Hi Loic,
> 
> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> > This patch introduces da8xx_rproc_parse_fw() to declare a
> > carveout region based on reserved memory for vdev buffer
> > allocation.
> >
> > Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > ---
> >  drivers/remoteproc/da8xx_remoteproc.c | 38
> +++++++++++++++++++++++++++++++++++
> >  1 file changed, 38 insertions(+)
> >
> > diff --git a/drivers/remoteproc/da8xx_remoteproc.c
> b/drivers/remoteproc/da8xx_remoteproc.c
> > index b668e32..679a076 100644
> > --- a/drivers/remoteproc/da8xx_remoteproc.c
> > +++ b/drivers/remoteproc/da8xx_remoteproc.c
> > @@ -16,6 +16,7 @@
> >  #include <linux/irq.h>
> >  #include <linux/kernel.h>
> >  #include <linux/module.h>
> > +#include <linux/of_address.h>
> >  #include <linux/of_reserved_mem.h>
> >  #include <linux/platform_device.h>
> >  #include <linux/remoteproc.h>
> > @@ -179,10 +180,47 @@ static void da8xx_rproc_kick(struct rproc *rproc,
> int vqid)
> >  	writel(SYSCFG_CHIPSIG2, drproc->chipsig);
> >  }
> >
> > +static int da8xx_rproc_parse_fw(struct rproc *rproc, const struct firmware
> *fw)
> > +{
> > +	struct device *dev = rproc->dev.parent;
> > +	struct rproc_mem_entry *mem;
> > +	struct device_node *node;
> > +	struct resource res;
> > +	int err;
> > +
> > +	node = of_parse_phandle(dev->of_node, "memory-region", 0);
> > +	if (!node) {
> > +		dev_err(dev, "No memory-region specified\n");
> > +		return -EINVAL;
> > +	}
> > +
> > +	err = of_address_to_resource(node, 0, &res);
> > +	if (err) {
> > +		dev_err(dev, "Bad memory-region definition\n");
> > +		return err;
> > +	}
> > +
> > +	/* Register memory region for vdev buffer allocation */
> > +	mem = rproc_of_resm_mem_entry_init(dev, 0,
> resource_size(&res),
> > +					   res.start, "vdev0buffer");> +
> > +	if (!mem)
> > +		return -ENOMEM;
> > +
> > +	rproc_add_carveout(rproc, mem);
> > +
> > +	return rproc_elf_load_rsc_table(rproc, fw);
> > +}
> 
> Thanks for the patch, but this creates a kernel crash for me due to
> overlaps with manually created carveouts. I currently have a single
> memory-region and all allocations come from the same DMA pool, but the
> rproc_of_resm_mem_entry_init() creates an overall mem entry without the
> va being set (no alloc function plumbed in). In general, it is permitted
> to use the same reserved-memory node with multiple devices, so the index
> usage should have allowed it to do DMA allocations with vdev devices,
> but the loading is performed even before the vdev allocations and the
> da_to_va matches the first entry with no va set causing the crash.

Hummm, I didn't fall in this case, but clearly da_to_va should not crashed. 
Not allocated carveout should be bypassed in the loop. Thanks for pointing this. I need to fix it.

The rproc_of_resm_mem_entry_init() is simply registering the reserved memory to be attached to vdev device.
So that normal it won't be allocated by rproc core (there is no alloc/free function specificied in this helper). 

Regards,
Loic
> 
> Here's my debugfs output of the carveout_memories for reference,
> 
> Carveout memory entry:
>         Name: vdev0buffer
>         Virtual address: 00000000
>         DMA address: 0x00000000
>         Device address: 0xc3000000
>         Length: 0x1000000 Bytes
> 
> Carveout memory entry:
>         Name: vdev0vring0
>         Virtual address: c3000000
>         DMA address: 0xc3000000
>         Device address: 0xc3000000
>         Length: 0x3000 Bytes
> 
> Carveout memory entry:
>         Name: vdev0vring1
>         Virtual address: c3004000
>         DMA address: 0xc3004000
>         Device address: 0xc3004000
>         Length: 0x3000 Bytes
> 
> Carveout memory entry:
>         Name: DSP_MEM_DATA
>         Virtual address: c3100000
>         DMA address: 0xc3100000
>         Device address: 0xc3100000
>         Length: 0xf00000 Bytes
> 
> You can drop both this patch and the keystone_remoteproc patch from the
> series. I did not run into any issues there since I did not have any
> RSC_CARVEOUT entries there. Also, see my comments on the next patch
> (the
> changes in ST) in general regarding these API. Looks like this needs
> some more time in ironing out the issues.
> 
> regards
> Suman
> 
> 
> 
> > +
> >  static const struct rproc_ops da8xx_rproc_ops = {
> >  	.start = da8xx_rproc_start,
> >  	.stop = da8xx_rproc_stop,
> >  	.kick = da8xx_rproc_kick,
> > +	.parse_fw = da8xx_rproc_parse_fw,
> > +	.load = rproc_elf_load_segments,
> > +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
> > +	.sanity_check = rproc_elf_sanity_check,
> > +	.get_boot_addr = rproc_elf_get_boot_addr,
> >  };
> >
> >  static int da8xx_rproc_get_internal_memories(struct platform_device
> *pdev,
> >


^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-10-24  0:14   ` Suman Anna
@ 2018-10-24 15:14     ` Loic PALLARDY
  2018-10-29 20:17       ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-24 15:14 UTC (permalink / raw)
  To: Suman Anna, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

Hi Suman,

> -----Original Message-----
> From: Suman Anna <s-anna@ti.com>
> Sent: mercredi 24 octobre 2018 02:14
> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> ohad@wizery.com
> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> benjamin.gaignard@linaro.org
> Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on
> centralized carveout allocator
> 
> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> > Current version of rproc_alloc_vring function supports only dynamic vring
> > allocation.
> >
> > This patch allows to allocate vrings based on memory region declatation.
> > Vrings are now manage like memory carveouts, to communize memory
> management
> > code in rproc_alloc_registered_carveouts().
> >
> > Allocated buffer is retrieved in rp_find_vq() thanks to
> > rproc_find_carveout_by_name() functions for.
> >
> > This patch sets vrings names to vdev"x"vring"y" with x vdev index in
> > resource table and y vring index in vdev. This will be updated when
> > name will be associated to vdev in firmware resource table.
> >
> > Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > ---
> >  drivers/remoteproc/remoteproc_core.c     | 61 +++++++++++++++++------
> ---------
> >  drivers/remoteproc/remoteproc_internal.h |  2 ++
> >  drivers/remoteproc/remoteproc_virtio.c   | 14 +++++++-
> >  include/linux/remoteproc.h               |  6 ++--
> >  4 files changed, 51 insertions(+), 32 deletions(-)
> >
> > diff --git a/drivers/remoteproc/remoteproc_core.c
> b/drivers/remoteproc/remoteproc_core.c
> > index c543d04..4edc6f0 100644
> > --- a/drivers/remoteproc/remoteproc_core.c
> > +++ b/drivers/remoteproc/remoteproc_core.c
> > @@ -53,6 +53,11 @@ typedef int (*rproc_handle_resources_t)(struct
> rproc *rproc,
> >  typedef int (*rproc_handle_resource_t)(struct rproc *rproc,
> >  				 void *, int offset, int avail);
> >
> > +static int rproc_alloc_carveout(struct rproc *rproc,
> > +				struct rproc_mem_entry *mem);
> > +static int rproc_release_carveout(struct rproc *rproc,
> > +				  struct rproc_mem_entry *mem);
> > +
> >  /* Unique indices for remoteproc devices */
> >  static DEFINE_IDA(rproc_dev_index);
> >
> > @@ -312,21 +317,33 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev,
> int i)
> >  	struct device *dev = &rproc->dev;
> >  	struct rproc_vring *rvring = &rvdev->vring[i];
> >  	struct fw_rsc_vdev *rsc;
> > -	dma_addr_t dma;
> > -	void *va;
> >  	int ret, size, notifyid;
> > +	struct rproc_mem_entry *mem;
> >
> >  	/* actual size of vring (in bytes) */
> >  	size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
> >
> > -	/*
> > -	 * Allocate non-cacheable memory for the vring. In the future
> > -	 * this call will also configure the IOMMU for us
> > -	 */
> > -	va = dma_alloc_coherent(dev->parent, size, &dma, GFP_KERNEL);
> > -	if (!va) {
> > -		dev_err(dev->parent, "dma_alloc_coherent failed\n");
> > -		return -EINVAL;
> > +	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > +
> > +	/* Search for pre-registered carveout */
> > +	mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d",
> rvdev->index,
> > +					  i);
> > +	if (mem) {
> > +		if (rproc_check_carveout_da(rproc, mem, rsc->vring[i].da,
> size))
> > +			return -ENOMEM;
> > +	} else {
> > +		/* Register carveout in in list */
> > +		mem = rproc_mem_entry_init(dev, 0, 0, size, rsc-
> >vring[i].da,
> > +					   rproc_alloc_carveout,
> > +					   rproc_release_carveout,
> > +					   "vdev%dvring%d",
> > +					   rvdev->index, i);
> > +		if (!mem) {
> > +			dev_err(dev, "Can't allocate memory entry
> structure\n");
> > +			return -ENOMEM;
> > +		}
> > +
> > +		rproc_add_carveout(rproc, mem);
> >  	}
> >
> >  	/*
> > @@ -337,7 +354,6 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
> >  	ret = idr_alloc(&rproc->notifyids, rvring, 0, 0, GFP_KERNEL);
> >  	if (ret < 0) {
> >  		dev_err(dev, "idr_alloc failed: %d\n", ret);
> > -		dma_free_coherent(dev->parent, size, va, dma);
> >  		return ret;
> >  	}
> >  	notifyid = ret;
> > @@ -346,21 +362,9 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int
> i)
> >  	if (notifyid > rproc->max_notifyid)
> >  		rproc->max_notifyid = notifyid;
> >
> > -	dev_dbg(dev, "vring%d: va %pK dma %pad size 0x%x idr %d\n",
> > -		i, va, &dma, size, notifyid);
> > -
> > -	rvring->va = va;
> > -	rvring->dma = dma;
> >  	rvring->notifyid = notifyid;
> >
> > -	/*
> > -	 * Let the rproc know the notifyid and da of this vring.
> > -	 * Not all platforms use dma_alloc_coherent to automatically
> > -	 * set up the iommu. In this case the device address (da) will
> > -	 * hold the physical address and not the device address.
> > -	 */
> > -	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > -	rsc->vring[i].da = dma;
> > +	/* Let the rproc know the notifyid of this vring.*/
> >  	rsc->vring[i].notifyid = notifyid;
> >  	return 0;
> >  }
> > @@ -392,12 +396,10 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev,
> int i)
> >
> >  void rproc_free_vring(struct rproc_vring *rvring)
> >  {
> > -	int size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
> >  	struct rproc *rproc = rvring->rvdev->rproc;
> >  	int idx = rvring->rvdev->vring - rvring;
> >  	struct fw_rsc_vdev *rsc;
> >
> > -	dma_free_coherent(rproc->dev.parent, size, rvring->va, rvring-
> >dma);
> >  	idr_remove(&rproc->notifyids, rvring->notifyid);
> 
> Also, I am not sure if FW_RSC_ADDR_ANY semantics were enforced
> previously on the vring da. It was simply overwritten irrespective of
> the value. Now, I am running again into the "bad carveout rsc
> configuration value" due to the iommu_domain_check if
> !FW_RSC_ADDR_ANY.

When are you running into the "bad carveout rsc configuration value" ? 
This patch is creating one carveout per vring to rely on generic carveout allocator.
Then carveout is retrieved from carveout  list and vring resource table information updated.

If the da of the carveout was fixed in the resource table, it is normal you have this error.
To solve that ST driver is registering one fixed carveout per vring (with the right name today)

It is the same discussion as with Patch 1. If we consider we can't change fixed coprocessor address requests, rproc core should stop its execution.
It is the responsibility of platform driver to register the right memory regions.
That's what we discussed with you and Bill in OpenAMP forum.
TI usecase was to have the same DSP firmware with the same resource table being able to run on any DSP.
In that case each DSP platform driver has to provide the right memory region configuration with the correct pa to da.

> 
> FWIW, the rproc_free_vring was actually using the value 0 when resetting.
 
It is no more needed as the carveout list is cleared at each stop and recreated at each start.
Moreover resource table (and firmware) is reloaded at each coprocessor start.

Regards,
Loic
> 
> regards
> Suman
> 
> >
> >  	/* reset resource entry info */
> > @@ -484,6 +486,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
> struct fw_rsc_vdev *rsc,
> >
> >  	rvdev->id = rsc->id;
> >  	rvdev->rproc = rproc;
> > +	rvdev->index = rproc->nb_vdev++;
> >
> >  	/* parse the vrings */
> >  	for (i = 0; i < rsc->num_of_vrings; i++) {
> > @@ -528,9 +531,6 @@ void rproc_vdev_release(struct kref *ref)
> >
> >  	for (id = 0; id < ARRAY_SIZE(rvdev->vring); id++) {
> >  		rvring = &rvdev->vring[id];
> > -		if (!rvring->va)
> > -			continue;
> > -
> >  		rproc_free_vring(rvring);
> >  	}
> >
> > @@ -1322,6 +1322,9 @@ static int rproc_fw_boot(struct rproc *rproc,
> const struct firmware *fw)
> >  	/* reset max_notifyid */
> >  	rproc->max_notifyid = -1;
> >
> > +	/* reset handled vdev */
> > +	rproc->nb_vdev = 0;
> > +
> >  	/* handle fw resources which are required to boot rproc */
> >  	ret = rproc_handle_resources(rproc, rproc_loading_handlers);
> >  	if (ret) {
> > diff --git a/drivers/remoteproc/remoteproc_internal.h
> b/drivers/remoteproc/remoteproc_internal.h
> > index 7570beb..f6cad24 100644
> > --- a/drivers/remoteproc/remoteproc_internal.h
> > +++ b/drivers/remoteproc/remoteproc_internal.h
> > @@ -60,6 +60,8 @@ struct dentry *rproc_create_trace_file(const char
> *name, struct rproc *rproc,
> >  int rproc_elf_load_rsc_table(struct rproc *rproc, const struct firmware
> *fw);
> >  struct resource_table *rproc_elf_find_loaded_rsc_table(struct rproc
> *rproc,
> >  						       const struct firmware
> *fw);
> > +struct rproc_mem_entry *
> > +rproc_find_carveout_by_name(struct rproc *rproc, const char *name,
> ...);
> >
> >  static inline
> >  int rproc_fw_sanity_check(struct rproc *rproc, const struct firmware *fw)
> > diff --git a/drivers/remoteproc/remoteproc_virtio.c
> b/drivers/remoteproc/remoteproc_virtio.c
> > index bbecd44..de21f62 100644
> > --- a/drivers/remoteproc/remoteproc_virtio.c
> > +++ b/drivers/remoteproc/remoteproc_virtio.c
> > @@ -76,7 +76,9 @@ static struct virtqueue *rp_find_vq(struct
> virtio_device *vdev,
> >  	struct rproc_vdev *rvdev = vdev_to_rvdev(vdev);
> >  	struct rproc *rproc = vdev_to_rproc(vdev);
> >  	struct device *dev = &rproc->dev;
> > +	struct rproc_mem_entry *mem;
> >  	struct rproc_vring *rvring;
> > +	struct fw_rsc_vdev *rsc;
> >  	struct virtqueue *vq;
> >  	void *addr;
> >  	int len, size;
> > @@ -88,8 +90,14 @@ static struct virtqueue *rp_find_vq(struct
> virtio_device *vdev,
> >  	if (!name)
> >  		return NULL;
> >
> > +	/* Search allocated memory region by name */
> > +	mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d",
> rvdev->index,
> > +					  id);
> > +	if (!mem || !mem->va)
> > +		return ERR_PTR(-ENOMEM);
> > +
> >  	rvring = &rvdev->vring[id];
> > -	addr = rvring->va;
> > +	addr = mem->va;
> >  	len = rvring->len;
> >
> >  	/* zero vring */
> > @@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct
> virtio_device *vdev,
> >  	rvring->vq = vq;
> >  	vq->priv = rvring;
> >
> > +	/* Update vring in resource table */
> > +	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > +	rsc->vring[id].da = mem->da;
> > +
> >  	return vq;
> >  }
> >
> > diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> > index 4cdd0c6..6b3a234 100644
> > --- a/include/linux/remoteproc.h
> > +++ b/include/linux/remoteproc.h
> > @@ -453,6 +453,7 @@ struct rproc_dump_segment {
> >   * @table_sz: size of @cached_table
> >   * @has_iommu: flag to indicate if remote processor is behind an MMU
> >   * @dump_segments: list of segments in the firmware
> > + * @nb_vdev: number of vdev currently handled by rproc
> >   */
> >  struct rproc {
> >  	struct list_head node;
> > @@ -485,6 +486,7 @@ struct rproc {
> >  	bool has_iommu;
> >  	bool auto_boot;
> >  	struct list_head dump_segments;
> > +	int nb_vdev;
> >  };
> >
> >  /**
> > @@ -512,7 +514,6 @@ struct rproc_subdev {
> >  /**
> >   * struct rproc_vring - remoteproc vring state
> >   * @va:	virtual address
> > - * @dma: dma address
> >   * @len: length, in bytes
> >   * @da: device address
> >   * @align: vring alignment
> > @@ -522,7 +523,6 @@ struct rproc_subdev {
> >   */
> >  struct rproc_vring {
> >  	void *va;
> > -	dma_addr_t dma;
> >  	int len;
> >  	u32 da;
> >  	u32 align;
> > @@ -541,6 +541,7 @@ struct rproc_vring {
> >   * @vdev: the virio device
> >   * @vring: the vrings for this vdev
> >   * @rsc_offset: offset of the vdev's resource entry
> > + * @index: vdev position versus other vdev declared in resource table
> >   */
> >  struct rproc_vdev {
> >  	struct kref refcount;
> > @@ -553,6 +554,7 @@ struct rproc_vdev {
> >  	struct virtio_device vdev;
> >  	struct rproc_vring vring[RVDEV_NUM_VRINGS];
> >  	u32 rsc_offset;
> > +	u32 index;
> >  };
> >
> >  struct rproc *rproc_get_by_phandle(phandle phandle);
> >


^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 10/17] remoteproc: add helper function to check carveout device address
  2018-10-23 22:14   ` Suman Anna
@ 2018-10-24 15:24     ` Loic PALLARDY
  2018-10-25 22:50       ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-24 15:24 UTC (permalink / raw)
  To: Suman Anna, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

Hi Suman,

> -----Original Message-----
> From: Suman Anna <s-anna@ti.com>
> Sent: mercredi 24 octobre 2018 00:14
> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> ohad@wizery.com
> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> benjamin.gaignard@linaro.org
> Subject: Re: [PATCH v4 10/17] remoteproc: add helper function to check
> carveout device address
> 
> Hi Loic,
> 
> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> > This patch introduces a function to verify that a specified carveout
> > is fitting request device address and associated length
> >
> > Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > ---
> >  drivers/remoteproc/remoteproc_core.c | 47
> ++++++++++++++++++++++++++++++++++++
> >  1 file changed, 47 insertions(+)
> >
> > diff --git a/drivers/remoteproc/remoteproc_core.c
> b/drivers/remoteproc/remoteproc_core.c
> > index 1e0fe3e..5dd5edf 100644
> > --- a/drivers/remoteproc/remoteproc_core.c
> > +++ b/drivers/remoteproc/remoteproc_core.c
> > @@ -259,6 +259,53 @@ struct rproc_mem_entry *
> >  	return mem;
> >  }
> >
> > +/**
> > + * rproc_check_carveout_da() - Check specified carveout da configuration
> > + * @rproc: handle of a remote processor
> > + * @mem: pointer on carveout to check
> > + * @da: area device address
> > + * @len: associated area size
> > + *
> > + * This function is a helper function to verify requested device area
> (couple
> > + * da, len) is part of specified carevout.
> 
> %s/carevout/carveout/
OK
> 
> > + *
> > + * Return: 0 if carveout matchs request else -ENOMEM
> 
> %s/matchs/matches/
OK
> 
> > + */
> > +int rproc_check_carveout_da(struct rproc *rproc, struct
> rproc_mem_entry *mem,
> 
> static int since this seems to be only a local function.
OK
> 
> > +			    u32 da, u32 len)
> > +{
> > +	struct device *dev = &rproc->dev;
> > +	int delta = 0;
> > +
> > +	/* Check requested resource length */
> > +	if (len > mem->len) {
> > +		dev_err(dev, "Registered carveout doesn't fit len
> request\n");
> > +		return -ENOMEM;
> 
> ENOMEM not typically used for these kind of errors, you were probably
> inclined to used this since it is dealing with memory.

-EINVAL will be better
> 
> > +	}
> > +
> 
> Both the below codepaths are exercised only when da is not
> FW_RSC_ADDR_ANY, and you are returning 0 otherwise (which is the case of
> matches as per your description above). Is that what you really want -
> should it be an error

Yes when da is equal to FW_RSC_ADDR_ANY we should check the length too

> 
> > +	if (da != FW_RSC_ADDR_ANY && mem->da == FW_RSC_ADDR_ANY)
> {
> > +		/* Update existing carveout da */
> > +		mem->da = da;
> 
> Where would you need to update this?
In that case, we have 2 carveouts with the same name.
One has some fixed requests. The other one has none.
The goal here is to align both on the one which has the strongest requirements.
I think length is missing.

Regards,
Loic

> 
> regards
> Suman
> 
> > +	} else if (da != FW_RSC_ADDR_ANY && mem->da !=
> FW_RSC_ADDR_ANY) {
> > +		delta = da - mem->da;
> > +
> > +		/* Check requested resource belongs to registered carveout
> */
> > +		if (delta < 0) {
> > +			dev_err(dev,
> > +				"Registered carveout doesn't fit da
> request\n");
> > +			return -ENOMEM;
> > +		}
> > +
> > +		if (delta + len > mem->len) {
> > +			dev_err(dev,
> > +				"Registered carveout doesn't fit len
> request\n");
> > +			return -ENOMEM;
> > +		}
> > +	}
> > +
> > +	return 0;
> 
> 
> > +}
> > +
> >  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
> >  {
> >  	struct rproc *rproc = rvdev->rproc;
> >


^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 08/17] remoteproc: add alloc ops in rproc_mem_entry struct
  2018-10-23 21:20   ` Suman Anna
@ 2018-10-24 16:00     ` Loic PALLARDY
  2018-10-25 22:37       ` Suman Anna
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-10-24 16:00 UTC (permalink / raw)
  To: Suman Anna, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard



> -----Original Message-----
> From: Suman Anna <s-anna@ti.com>
> Sent: mardi 23 octobre 2018 23:20
> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> ohad@wizery.com
> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> benjamin.gaignard@linaro.org
> Subject: Re: [PATCH v4 08/17] remoteproc: add alloc ops in
> rproc_mem_entry struct
> 
> Hi Loic,
> 
> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> > Memory entry could be allocated in different ways (ioremap,
> > dma_alloc_coherent, internal RAM allocator...).
> > This patch introduces an alloc ops in rproc_mem_entry structure
> > to associate dedicated allocation mechanism to each memory entry
> > descriptor in order to do remote core agnostic from memory allocators.
> >
> > The introduction of this ops allows to perform allocation of all registered
> > carveout at the same time, just before calling rproc_start().
> > It simplifies and makes uniform carveout management whatever origin.
> 
> This patch is causing a kernel crash with trace entries. Please see
> further below for the cause.
> 
> >
> > Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > ---
> >  drivers/remoteproc/remoteproc_core.c | 261
> ++++++++++++++++++++++-------------
> >  include/linux/remoteproc.h           |   7 +
> >  2 files changed, 175 insertions(+), 93 deletions(-)
> >
> > diff --git a/drivers/remoteproc/remoteproc_core.c
> b/drivers/remoteproc/remoteproc_core.c
> > index 77b39ba..2c51549 100644
> > --- a/drivers/remoteproc/remoteproc_core.c
> > +++ b/drivers/remoteproc/remoteproc_core.c
> > @@ -642,74 +642,31 @@ static int rproc_handle_devmem(struct rproc
> *rproc, struct fw_rsc_devmem *rsc,
> >  }
> >
> >  /**
> > - * rproc_release_carveout() - release acquired carveout
> > + * rproc_alloc_carveout() - allocated specified carveout
> >   * @rproc: rproc handle
> > - * @mem: the memory entry to release
> > - *
> > - * This function releases specified memory entry @mem allocated via
> > - * dma_alloc_coherent() function by @rproc.
> > - */
> > -static int rproc_release_carveout(struct rproc *rproc,
> > -				  struct rproc_mem_entry *mem)
> > -{
> > -	struct device *dev = &rproc->dev;
> > -
> > -	/* clean up carveout allocations */
> > -	dma_free_coherent(dev->parent, mem->len, mem->va, mem-
> >dma);
> > -	return 0;
> > -}
> > -
> > -/**
> > - * rproc_handle_carveout() - handle phys contig memory allocation
> requests
> > - * @rproc: rproc handle
> > - * @rsc: the resource entry
> > - * @avail: size of available data (for image validation)
> > - *
> > - * This function will handle firmware requests for allocation of physically
> > - * contiguous memory regions.
> > - *
> > - * These request entries should come first in the firmware's resource
> table,
> > - * as other firmware entries might request placing other data objects
> inside
> > - * these memory regions (e.g. data/code segments, trace resource
> entries, ...).
> > + * @mem: the memory entry to allocate
> >   *
> > - * Allocating memory this way helps utilizing the reserved physical memory
> > - * (e.g. CMA) more efficiently, and also minimizes the number of TLB
> entries
> > - * needed to map it (in case @rproc is using an IOMMU). Reducing the TLB
> > - * pressure is important; it may have a substantial impact on performance.
> > + * This function allocate specified memory entry @mem using
> > + * dma_alloc_coherent() as default allocator
> >   */
> > -static int rproc_handle_carveout(struct rproc *rproc,
> > -				 struct fw_rsc_carveout *rsc,
> > -				 int offset, int avail)
> > +static int rproc_alloc_carveout(struct rproc *rproc,
> > +				struct rproc_mem_entry *mem)
> >  {
> > -	struct rproc_mem_entry *carveout, *mapping = NULL;
> > +	struct rproc_mem_entry *mapping = NULL;
> >  	struct device *dev = &rproc->dev;
> >  	dma_addr_t dma;
> >  	void *va;
> >  	int ret;
> >
> > -	if (sizeof(*rsc) > avail) {
> > -		dev_err(dev, "carveout rsc is truncated\n");
> > -		return -EINVAL;
> > -	}
> > -
> > -	/* make sure reserved bytes are zeroes */
> > -	if (rsc->reserved) {
> > -		dev_err(dev, "carveout rsc has non zero reserved bytes\n");
> > -		return -EINVAL;
> > -	}
> > -
> > -	dev_dbg(dev, "carveout rsc: name: %s, da 0x%x, pa 0x%x, len 0x%x,
> flags 0x%x\n",
> > -		rsc->name, rsc->da, rsc->pa, rsc->len, rsc->flags);
> > -
> > -	va = dma_alloc_coherent(dev->parent, rsc->len, &dma,
> GFP_KERNEL);
> > +	va = dma_alloc_coherent(dev->parent, mem->len, &dma,
> GFP_KERNEL);
> >  	if (!va) {
> >  		dev_err(dev->parent,
> > -			"failed to allocate dma memory: len 0x%x\n", rsc-
> >len);
> > +			"failed to allocate dma memory: len 0x%x\n", mem-
> >len);
> >  		return -ENOMEM;
> >  	}
> >
> >  	dev_dbg(dev, "carveout va %pK, dma %pad, len 0x%x\n",
> > -		va, &dma, rsc->len);
> > +		va, &dma, mem->len);
> >
> >  	/*
> >  	 * Ok, this is non-standard.
> > @@ -729,22 +686,22 @@ static int rproc_handle_carveout(struct rproc
> *rproc,
> >  	 * physical address in this case.
> >  	 */
> >
> > -	if (rsc->da != FW_RSC_ADDR_ANY && !rproc->domain) {
> > -		dev_err(dev->parent,
> > -			"Bad carveout rsc configuration\n");
> > -		ret = -ENOMEM;
> > -		goto dma_free;
> > -	}
> > +	if (mem->da != FW_RSC_ADDR_ANY) {
> > +		if (!rproc->domain) {
> > +			dev_err(dev->parent,
> > +				"Bad carveout rsc configuration\n");
> > +			ret = -ENOMEM;
> > +			goto dma_free;
> > +		}
> 
> Same comment from Patch 1.
> 
> >
> > -	if (rsc->da != FW_RSC_ADDR_ANY && rproc->domain) {
> >  		mapping = kzalloc(sizeof(*mapping), GFP_KERNEL);
> >  		if (!mapping) {
> >  			ret = -ENOMEM;
> >  			goto dma_free;
> >  		}
> >
> > -		ret = iommu_map(rproc->domain, rsc->da, dma, rsc->len,
> > -				rsc->flags);
> > +		ret = iommu_map(rproc->domain, mem->da, dma, mem-
> >len,
> > +				mem->flags);
> >  		if (ret) {
> >  			dev_err(dev, "iommu_map failed: %d\n", ret);
> >  			goto free_mapping;
> > @@ -757,52 +714,102 @@ static int rproc_handle_carveout(struct rproc
> *rproc,
> >  		 * We can't trust the remote processor not to change the
> >  		 * resource table, so we must maintain this info
> independently.
> >  		 */
> > -		mapping->da = rsc->da;
> > -		mapping->len = rsc->len;
> > +		mapping->da = mem->da;
> > +		mapping->len = mem->len;
> >  		list_add_tail(&mapping->node, &rproc->mappings);
> >
> >  		dev_dbg(dev, "carveout mapped 0x%x to %pad\n",
> > -			rsc->da, &dma);
> > +			mem->da, &dma);
> > +	} else {
> > +		mem->da = (u32)dma;
> 
> Hmm, what was the purpose of this? So, this appears to be handling the
> missing implementation for the comment in the fw_rsc_carveout about
> FW_RSC_ADDR_ANY.

It is needed to update da in rsc table when da is equal to FW_RSC_ADDR_ANY. No force update when da is fixed to another value.
It is the only way to provide information to coprocessor about carveout dynamically allocated by rproc core.
Coprocessor doesn't care about pa. Only da is valid from his pov.

> 
> >  	}
> >
> > -	/*
> > -	 * Some remote processors might need to know the pa
> > -	 * even though they are behind an IOMMU. E.g., OMAP4's
> > -	 * remote M3 processor needs this so it can control
> > -	 * on-chip hardware accelerators that are not behind
> > -	 * the IOMMU, and therefor must know the pa.
> > -	 *
> > -	 * Generally we don't want to expose physical addresses
> > -	 * if we don't have to (remote processors are generally
> > -	 * _not_ trusted), so we might want to do this only for
> > -	 * remote processor that _must_ have this (e.g. OMAP4's
> > -	 * dual M3 subsystem).
> > -	 *
> > -	 * Non-IOMMU processors might also want to have this info.
> > -	 * In this case, the device address and the physical address
> > -	 * are the same.
> > -	 */
> > -	rsc->pa = (u32)rproc_va_to_pa(va);
> > -
> > -	carveout = rproc_mem_entry_init(dev, va, dma, rsc->len, rsc->da,
> > -					rproc_release_carveout, rsc->name);
> > -	if (!carveout)
> > -		goto free_carv;
> > -
> > -	rproc_add_carveout(rproc, carveout);
> > +	mem->dma = (u32)dma;
> 
> We don't need the typecast, mem->dma is already of type dma_addr_t.
> Same
> comment above on the else part as well.

Ok for cast
> 
> > +	mem->va = va;
> >
> >  	return 0;
> >
> > -free_carv:
> > -	kfree(carveout);
> >  free_mapping:
> >  	kfree(mapping);
> >  dma_free:
> > -	dma_free_coherent(dev->parent, rsc->len, va, dma);
> > +	dma_free_coherent(dev->parent, mem->len, va, dma);
> >  	return ret;
> >  }
> >
> >  /**
> > + * rproc_release_carveout() - release acquired carveout
> > + * @rproc: rproc handle
> > + * @mem: the memory entry to release
> > + *
> > + * This function releases specified memory entry @mem allocated via
> > + * rproc_alloc_carveout() function by @rproc.
> > + */
> > +static int rproc_release_carveout(struct rproc *rproc,
> > +				  struct rproc_mem_entry *mem)
> > +{
> > +	struct device *dev = &rproc->dev;
> > +
> > +	/* clean up carveout allocations */
> > +	dma_free_coherent(dev->parent, mem->len, mem->va, mem-
> >dma);
> > +	return 0;
> > +}
> > +
> > +/**
> > + * rproc_handle_carveout() - handle phys contig memory allocation
> requests
> > + * @rproc: rproc handle
> > + * @rsc: the resource entry
> > + * @avail: size of available data (for image validation)
> > + *
> > + * This function will handle firmware requests for allocation of physically
> > + * contiguous memory regions.
> > + *
> > + * These request entries should come first in the firmware's resource
> table,
> > + * as other firmware entries might request placing other data objects
> inside
> > + * these memory regions (e.g. data/code segments, trace resource
> entries, ...).
> > + *
> > + * Allocating memory this way helps utilizing the reserved physical
> memory
> > + * (e.g. CMA) more efficiently, and also minimizes the number of TLB
> entries
> > + * needed to map it (in case @rproc is using an IOMMU). Reducing the TLB
> > + * pressure is important; it may have a substantial impact on performance.
> > + */
> > +static int rproc_handle_carveout(struct rproc *rproc,
> > +				 struct fw_rsc_carveout *rsc,
> > +				 int offset, int avail)
> > +{
> > +	struct rproc_mem_entry *carveout;
> > +	struct device *dev = &rproc->dev;
> > +
> > +	if (sizeof(*rsc) > avail) {
> > +		dev_err(dev, "carveout rsc is truncated\n");
> > +		return -EINVAL;
> > +	}
> > +
> > +	/* make sure reserved bytes are zeroes */
> > +	if (rsc->reserved) {
> > +		dev_err(dev, "carveout rsc has non zero reserved bytes\n");
> > +		return -EINVAL;
> > +	}
> > +
> > +	dev_dbg(dev, "carveout rsc: name: %s, da 0x%x, pa 0x%x, len 0x%x,
> flags 0x%x\n",
> > +		rsc->name, rsc->da, rsc->pa, rsc->len, rsc->flags);
> > +
> > +	/* Register carveout in in list */
> > +	carveout = rproc_mem_entry_init(dev, 0, 0, rsc->len, rsc->da,
> > +					rproc_alloc_carveout,
> > +					rproc_release_carveout, rsc->name);
> > +	if (!carveout) {
> > +		dev_err(dev, "Can't allocate memory entry structure\n");
> > +		return -ENOMEM;
> > +	}
> > +
> > +	carveout->flags = rsc->flags;
> > +	carveout->rsc_offset = offset;
> > +	rproc_add_carveout(rproc, carveout);
> 
> Once we get rid of rproc_add_carveout, the list addition will mostly be
> handled in rproc_mem_entry_init itself.

Yes I have to figure out what could be the impact to call rproc_add_carveout() from rproc_mem_entry_init(). It should work...

> 
> > +
> > +	return 0;
> > +}
> > +
> > +/**
> >   * rproc_add_carveout() - register an allocated carveout region
> >   * @rproc: rproc handle
> >   * @mem: memory entry to register
> > @@ -832,6 +839,7 @@ void rproc_add_carveout(struct rproc *rproc, struct
> rproc_mem_entry *mem)
> >  struct rproc_mem_entry *
> >  rproc_mem_entry_init(struct device *dev,
> >  		     void *va, dma_addr_t dma, int len, u32 da,
> > +		     int (*alloc)(struct rproc *, struct rproc_mem_entry *),
> >  		     int (*release)(struct rproc *, struct rproc_mem_entry *),
> >  		     const char *name, ...)
> >  {
> > @@ -846,7 +854,9 @@ struct rproc_mem_entry *
> >  	mem->dma = dma;
> >  	mem->da = da;
> >  	mem->len = len;
> > +	mem->alloc = alloc;
> >  	mem->release = release;
> > +	mem->rsc_offset = FW_RSC_ADDR_ANY;
> >
> >  	va_start(args, name);
> >  	vsnprintf(mem->name, sizeof(mem->name), name, args);
> > @@ -978,6 +988,63 @@ static void rproc_unprepare_subdevices(struct
> rproc *rproc)
> >  }
> >
> >  /**
> > + * rproc_alloc_registered_carveouts() - allocate all carveouts registered
> > + * in the list
> > + * @rproc: the remote processor handle
> > + *
> > + * This function parses registered carveout list, performs allocation
> > + * if alloc() ops registered and updates resource table information
> > + * if rsc_offset set.
> > + *
> > + * Return: 0 on success
> > + */
> > +static int rproc_alloc_registered_carveouts(struct rproc *rproc)
> > +{
> > +	struct rproc_mem_entry *entry, *tmp;
> > +	struct fw_rsc_carveout *rsc;
> > +	struct device *dev = &rproc->dev;
> > +	int ret;
> > +
> > +	list_for_each_entry_safe(entry, tmp, &rproc->carveouts, node) {
> > +		if (entry->alloc) {
> > +			ret = entry->alloc(rproc, entry);
> > +			if (ret) {
> > +				dev_err(dev, "Unable to allocate carveout
> %s: %d\n",
> > +					entry->name, ret);
> > +				return -ENOMEM;
> > +			}
> > +		}
> > +
> > +		if (entry->rsc_offset != FW_RSC_ADDR_ANY) {
> > +			/* update resource table */
> > +			rsc = (void *)rproc->table_ptr + entry->rsc_offset;
> > +
> > +			/*
> > +			 * Some remote processors might need to know the
> pa
> > +			 * even though they are behind an IOMMU. E.g.,
> OMAP4's
> > +			 * remote M3 processor needs this so it can control
> > +			 * on-chip hardware accelerators that are not behind
> > +			 * the IOMMU, and therefor must know the pa.
> > +			 *
> > +			 * Generally we don't want to expose physical
> addresses
> > +			 * if we don't have to (remote processors are
> generally
> > +			 * _not_ trusted), so we might want to do this only
> for
> > +			 * remote processor that _must_ have this (e.g.
> OMAP4's
> > +			 * dual M3 subsystem).
> > +			 *
> > +			 * Non-IOMMU processors might also want to have
> this info.
> > +			 * In this case, the device address and the physical
> address
> > +			 * are the same.
> > +			 */
> > +			if (entry->va)
> > +				rsc->pa = (u32)rproc_va_to_pa(entry->va);
> > +		}
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +/**
> >   * rproc_coredump_cleanup() - clean up dump_segments list
> >   * @rproc: the remote processor handle
> >   */
> > @@ -1148,6 +1215,14 @@ static int rproc_fw_boot(struct rproc *rproc,
> const struct firmware *fw)
> >  		goto clean_up_resources;
> >  	}
> >
> > +	/* Allocate carveout resources associated to rproc */
> > +	ret = rproc_alloc_registered_carveouts(rproc);
> > +	if (ret) {
> > +		dev_err(dev, "Failed to allocate associated carveouts: %d\n",
> > +			ret);
> > +		goto clean_up_resources;
> > +	}
> 
> This is causing an issue with RSC_TRACE on where the trace region on the
> remote processor is actually backed by a DDR carveout address. The
> allocations are now being done after processing the resources from the
> rproc_loading_handlers, which causes the RSC_TRACE to be configured with
> an incorrect kernel va, and accessing it through debugfs then results in
> a kernel crash.

Ok I understand the issue. Mask by the double mapping you pointing in my driver.
I need to correct that.
Thanks for pointing it.

Regards,
Loic

> 
> regards
> Suman
> 
> > +
> >  	ret = rproc_start(rproc, fw);
> >  	if (ret)
> >  		goto clean_up_resources;
> > diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> > index 55f30fc..ea95b04 100644
> > --- a/include/linux/remoteproc.h
> > +++ b/include/linux/remoteproc.h
> > @@ -317,6 +317,9 @@ struct fw_rsc_vdev {
> >   * @priv: associated data
> >   * @name: associated memory region name (optional)
> >   * @node: list node
> > + * @rsc_offset: offset in resource table
> > + * @flags: iommu protection flags
> > + * @alloc: specific memory allocator function
> >   */
> >  struct rproc_mem_entry {
> >  	void *va;
> > @@ -326,6 +329,9 @@ struct rproc_mem_entry {
> >  	void *priv;
> >  	char name[32];
> >  	struct list_head node;
> > +	u32 rsc_offset;
> > +	u32 flags;
> > +	int (*alloc)(struct rproc *rproc, struct rproc_mem_entry *mem);
> >  	int (*release)(struct rproc *rproc, struct rproc_mem_entry *mem);
> >  };
> >
> > @@ -563,6 +569,7 @@ struct rproc *rproc_alloc(struct device *dev, const
> char *name,
> >  struct rproc_mem_entry *
> >  rproc_mem_entry_init(struct device *dev,
> >  		     void *va, dma_addr_t dma, int len, u32 da,
> > +		     int (*alloc)(struct rproc *, struct rproc_mem_entry *),
> >  		     int (*release)(struct rproc *, struct rproc_mem_entry *),
> >  		     const char *name, ...);
> >
> >


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-10-24 12:40           ` Loic PALLARDY
@ 2018-10-25 20:16             ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-25 20:16 UTC (permalink / raw)
  To: Loic PALLARDY, Wendy Liang
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, Arnaud POULIQUEN, benjamin.gaignard

On 10/24/18 7:40 AM, Loic PALLARDY wrote:
> Hi Suman,
> 
>> -----Original Message-----
>> From: Suman Anna <s-anna@ti.com>
>> Sent: mercredi 24 octobre 2018 03:22
>> To: Wendy Liang <sunnyliangjy@gmail.com>; Loic PALLARDY
>> <loic.pallardy@st.com>
>> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
>> <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; Linux Kernel
>> Mailing List <linux-kernel@vger.kernel.org>; Arnaud POULIQUEN
>> <arnaud.pouliquen@st.com>; benjamin.gaignard@linaro.org
>> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
>> specific dma memory pool
>>
>> On 9/27/18 3:18 PM, Wendy Liang wrote:
>>> Hi Loic,
>>>
>>>
>>> On Thu, Sep 27, 2018 at 12:22 PM Loic PALLARDY <loic.pallardy@st.com>
>> wrote:
>>>>
>>>> Hi Wendy
>>>>
>>>>> -----Original Message-----
>>>>> From: Wendy Liang <sunnyliangjy@gmail.com>
>>>>> Sent: Thursday, September 27, 2018 7:17 PM
>>>>> To: Loic PALLARDY <loic.pallardy@st.com>
>>>>> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
>>>>> <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; Linux Kernel
>>>>> Mailing List <linux-kernel@vger.kernel.org>; Arnaud POULIQUEN
>>>>> <arnaud.pouliquen@st.com>; benjamin.gaignard@linaro.org; Suman
>> Anna
>>>>> <s-anna@ti.com>
>>>>> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
>>>>> specific dma memory pool
>>>>>
>>>>> On Fri, Jul 27, 2018 at 6:16 AM Loic Pallardy <loic.pallardy@st.com>
>> wrote:
>>>>>>
>>>>>> This patch creates a dedicated vdev subdevice for each vdev declared
>>>>>> in firmware resource table and associates carveout named
>> "vdev%dbuffer"
>>>>>> (with %d vdev index in resource table) if any as dma coherent memory
>>>>> pool.
>>>>>>
>>>>>> Then vdev subdevice is used as parent for virtio device.
>>>>>>
>>>>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>>>>> ---
>>>>>>  drivers/remoteproc/remoteproc_core.c     | 35
>>>>> +++++++++++++++++++++++---
>>>>>>  drivers/remoteproc/remoteproc_internal.h |  1 +
>>>>>>  drivers/remoteproc/remoteproc_virtio.c   | 42
>>>>> +++++++++++++++++++++++++++++++-
>>>>>>  include/linux/remoteproc.h               |  1 +
>>>>>>  4 files changed, 75 insertions(+), 4 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>>>>> b/drivers/remoteproc/remoteproc_core.c
>>>>>> index 4edc6f0..adcc66e 100644
>>>>>> --- a/drivers/remoteproc/remoteproc_core.c
>>>>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>>>>> @@ -39,6 +39,7 @@
>>>>>>  #include <linux/idr.h>
>>>>>>  #include <linux/elf.h>
>>>>>>  #include <linux/crc32.h>
>>>>>> +#include <linux/of_reserved_mem.h>
>>>>>>  #include <linux/virtio_ids.h>
>>>>>>  #include <linux/virtio_ring.h>
>>>>>>  #include <asm/byteorder.h>
>>>>>> @@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc
>>>>> *rproc)
>>>>>>         iommu_domain_free(domain);
>>>>>>  }
>>>>>>
>>>>>> -static phys_addr_t rproc_va_to_pa(void *cpu_addr)
>>>>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr)
>>>>>>  {
>>>>>>         /*
>>>>>>          * Return physical address according to virtual address location
>>>>>> @@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void
>>>>> *cpu_addr)
>>>>>>         WARN_ON(!virt_addr_valid(cpu_addr));
>>>>>>         return virt_to_phys(cpu_addr);
>>>>>>  }
>>>>>> +EXPORT_SYMBOL(rproc_va_to_pa);
>>>>>>
>>>>>>  /**
>>>>>>   * rproc_da_to_va() - lookup the kernel virtual address for a
>> remoteproc
>>>>> address
>>>>>> @@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct
>>>>> rproc_subdev *subdev, bool crashed)
>>>>>>  }
>>>>>>
>>>>>>  /**
>>>>>> + * rproc_rvdev_release() - release the existence of a rvdev
>>>>>> + *
>>>>>> + * @dev: the subdevice's dev
>>>>>> + */
>>>>>> +static void rproc_rvdev_release(struct device *dev)
>>>>>> +{
>>>>>> +       struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev,
>> dev);
>>>>>> +
>>>>>> +       of_reserved_mem_device_release(dev);
>>>>>> +
>>>>>> +       kfree(rvdev);
>>>>>> +}
>>>>>> +
>>>>>> +/**
>>>>>>   * rproc_handle_vdev() - handle a vdev fw resource
>>>>>>   * @rproc: the remote processor
>>>>>>   * @rsc: the vring resource descriptor
>>>>>> @@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc
>> *rproc,
>>>>> struct fw_rsc_vdev *rsc,
>>>>>>         struct device *dev = &rproc->dev;
>>>>>>         struct rproc_vdev *rvdev;
>>>>>>         int i, ret;
>>>>>> +       char name[16];
>>>>>>
>>>>>>         /* make sure resource isn't truncated */
>>>>>>         if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct
>>>>> fw_rsc_vdev_vring)
>>>>>> @@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc
>> *rproc,
>>>>> struct fw_rsc_vdev *rsc,
>>>>>>         rvdev->rproc = rproc;
>>>>>>         rvdev->index = rproc->nb_vdev++;
>>>>>>
>>>>>> +       /* Initialise vdev subdevice */
>>>>>> +       snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
>>>>>> +       rvdev->dev.parent = rproc->dev.parent;
>>>>>> +       rvdev->dev.release = rproc_rvdev_release;
>>>>>> +       dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev-
>>>>>> dev.parent), name);
>>>>>> +       dev_set_drvdata(&rvdev->dev, rvdev);
>>>>>> +       dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
>>>>> I tried the latest kernel, this function will not set the DMA coherent mask
>> as
>>>>> dma_supported() of the &rvdev->dev will return false.
>>>>> As this is a device created at run time, should it be force to support
>> DMA?
>>>>> should it directly set the dma_coherent_mask?
>>>>
>>>> Thanks for pointing me this issue. I tested on top of 4.18-rc1 few months
>> ago...
>>>> Could you please give me kernel version on which you are testing the
>> series?
>>>> Is you platform 32bit or 64bit ?
>>>> I'll rebase and check on my side.
>>>
>>> I am testing with 4.19-rc4 on aarch64 platform.
>>
>> Btw, I ran into this on my v7 platform as well (4.19-rc6). The
>> dma_set_coherent_mask fails with error EIO. I did get my allocations
>> through though.
> 
> I don't have issue on v7 platform. Anyway I have now an Xilinx Ultra96 board running on my desk. It looks like vdev device doesn't have dma support, so not possible to set mask.
> Need to continue the investigations...

Strange how it is working for you, the dma support on vdev device should
be the same for all of us right - vdevs are created during runtime.

Think somehow, dma_supported() function is returning non-zero value for you.

regards
Suman

> Regards,
> Loic
>>
>> regards
>> Suman
>>
>>>
>>> Best Regards,
>>> Wendy
>>>>
>>>> Regards,
>>>> Loic
>>>>
>>>>>
>>>>>> +
>>>>>> +       ret = device_register(&rvdev->dev);
>>>>>> +       if (ret)
>>>>>> +               goto free_rvdev;
>>>>>> +
>>>>>>         /* parse the vrings */
>>>>>>         for (i = 0; i < rsc->num_of_vrings; i++) {
>>>>>>                 ret = rproc_parse_vring(rvdev, rsc, i);
>>>>>> @@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc
>> *rproc,
>>>>> struct fw_rsc_vdev *rsc,
>>>>>>         for (i--; i >= 0; i--)
>>>>>>                 rproc_free_vring(&rvdev->vring[i]);
>>>>>>  free_rvdev:
>>>>>> -       kfree(rvdev);
>>>>>> +       device_unregister(&rvdev->dev);
>>>>>>         return ret;
>>>>>>  }
>>>>>>
>>>>>> @@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
>>>>>>
>>>>>>         rproc_remove_subdev(rproc, &rvdev->subdev);
>>>>>>         list_del(&rvdev->node);
>>>>>> -       kfree(rvdev);
>>>>>> +       device_unregister(&rvdev->dev);
>>>>>>  }
>>>>>>
>>>>>>  /**
>>>>>> diff --git a/drivers/remoteproc/remoteproc_internal.h
>>>>> b/drivers/remoteproc/remoteproc_internal.h
>>>>>> index f6cad24..bfeacfd 100644
>>>>>> --- a/drivers/remoteproc/remoteproc_internal.h
>>>>>> +++ b/drivers/remoteproc/remoteproc_internal.h
>>>>>> @@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const char
>>>>> *name, struct rproc *rproc,
>>>>>>  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
>>>>>>
>>>>>>  void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
>>>>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr);
>>>>>>  int rproc_trigger_recovery(struct rproc *rproc);
>>>>>>
>>>>>>  int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware
>> *fw);
>>>>>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
>>>>> b/drivers/remoteproc/remoteproc_virtio.c
>>>>>> index de21f62..9ee63c0 100644
>>>>>> --- a/drivers/remoteproc/remoteproc_virtio.c
>>>>>> +++ b/drivers/remoteproc/remoteproc_virtio.c
>>>>>> @@ -17,7 +17,9 @@
>>>>>>   * GNU General Public License for more details.
>>>>>>   */
>>>>>>
>>>>>> +#include <linux/dma-mapping.h>
>>>>>>  #include <linux/export.h>
>>>>>> +#include <linux/of_reserved_mem.h>
>>>>>>  #include <linux/remoteproc.h>
>>>>>>  #include <linux/virtio.h>
>>>>>>  #include <linux/virtio_config.h>
>>>>>> @@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct
>> device
>>>>> *dev)
>>>>>>  int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
>>>>>>  {
>>>>>>         struct rproc *rproc = rvdev->rproc;
>>>>>> -       struct device *dev = &rproc->dev;
>>>>>> +       struct device *dev = &rvdev->dev;
>>>>>>         struct virtio_device *vdev = &rvdev->vdev;
>>>>>> +       struct rproc_mem_entry *mem;
>>>>>>         int ret;
>>>>>>
>>>>>> +       /* Try to find dedicated vdev buffer carveout */
>>>>>> +       mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer",
>> rvdev-
>>>>>> index);
>>>>>> +       if (mem) {
>>>>>> +               phys_addr_t pa;
>>>>>> +
>>>>>> +               if (mem->of_resm_idx != -1) {
>>>>>> +                       struct device_node *np = rproc->dev.parent->of_node;
>>>>>> +
>>>>>> +                       /* Associate reserved memory to vdev device */
>>>>>> +                       ret = of_reserved_mem_device_init_by_idx(dev, np,
>>>>>> +                                                                mem->of_resm_idx);
>>>>>> +                       if (ret) {
>>>>>> +                               dev_err(dev, "Can't associate reserved memory\n");
>>>>>> +                               goto out;
>>>>>> +                       }
>>>>>> +               } else {
>>>>>> +                       if (mem->va) {
>>>>>> +                               dev_warn(dev, "vdev %d buffer already mapped\n",
>>>>>> +                                        rvdev->index);
>>>>>> +                               pa = rproc_va_to_pa(mem->va);
>>>>>> +                       } else {
>>>>>> +                               /* Use dma address as carveout no memmapped yet
>> */
>>>>>> +                               pa = (phys_addr_t)mem->dma;
>>>>>> +                       }
>>>>>> +
>>>>>> +                       /* Associate vdev buffer memory pool to vdev subdev */
>>>>>> +                       ret = dmam_declare_coherent_memory(dev, pa,
>>>>>> +                                                          mem->da,
>>>>>> +                                                          mem->len,
>>>>>> +                                                          DMA_MEMORY_EXCLUSIVE);
>>>>>> +                       if (ret < 0) {
>>>>>> +                               dev_err(dev, "Failed to associate buffer\n");
>>>>>> +                               goto out;
>>>>>> +                       }
>>>>>> +               }
>>>>>> +       }
>>>>>> +
>>>>>>         vdev->id.device = id,
>>>>>>         vdev->config = &rproc_virtio_config_ops,
>>>>>>         vdev->dev.parent = dev;
>>>>>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
>>>>>> index 6b3a234..2921dd2 100644
>>>>>> --- a/include/linux/remoteproc.h
>>>>>> +++ b/include/linux/remoteproc.h
>>>>>> @@ -547,6 +547,7 @@ struct rproc_vdev {
>>>>>>         struct kref refcount;
>>>>>>
>>>>>>         struct rproc_subdev subdev;
>>>>>> +       struct device dev;
>>>>>>
>>>>>>         unsigned int id;
>>>>>>         struct list_head node;
>>>>>> --
>>>>>> 1.9.1
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-
>> remoteproc"
>>>>> in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool
  2018-10-24 12:42             ` Loic PALLARDY
@ 2018-10-25 22:06               ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-25 22:06 UTC (permalink / raw)
  To: Loic PALLARDY, Wendy Liang
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, Arnaud POULIQUEN, benjamin.gaignard

On 10/24/18 7:42 AM, Loic PALLARDY wrote:
> 
> 
>> -----Original Message-----
>> From: Suman Anna <s-anna@ti.com>
>> Sent: mercredi 24 octobre 2018 03:49
>> To: Wendy Liang <sunnyliangjy@gmail.com>; Loic PALLARDY
>> <loic.pallardy@st.com>
>> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
>> <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; Linux Kernel
>> Mailing List <linux-kernel@vger.kernel.org>; Arnaud POULIQUEN
>> <arnaud.pouliquen@st.com>; benjamin.gaignard@linaro.org
>> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
>> specific dma memory pool
>>
>> On 10/23/18 8:22 PM, Suman Anna wrote:
>>> On 9/27/18 3:18 PM, Wendy Liang wrote:
>>>> Hi Loic,
>>>>
>>>>
>>>> On Thu, Sep 27, 2018 at 12:22 PM Loic PALLARDY <loic.pallardy@st.com>
>> wrote:
>>>>>
>>>>> Hi Wendy
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Wendy Liang <sunnyliangjy@gmail.com>
>>>>>> Sent: Thursday, September 27, 2018 7:17 PM
>>>>>> To: Loic PALLARDY <loic.pallardy@st.com>
>>>>>> Cc: Bjorn Andersson <bjorn.andersson@linaro.org>; Ohad Ben-Cohen
>>>>>> <ohad@wizery.com>; linux-remoteproc@vger.kernel.org; Linux Kernel
>>>>>> Mailing List <linux-kernel@vger.kernel.org>; Arnaud POULIQUEN
>>>>>> <arnaud.pouliquen@st.com>; benjamin.gaignard@linaro.org; Suman
>> Anna
>>>>>> <s-anna@ti.com>
>>>>>> Subject: Re: [PATCH v4 13/17] remoteproc: create vdev subdevice with
>>>>>> specific dma memory pool
>>>>>>
>>>>>> On Fri, Jul 27, 2018 at 6:16 AM Loic Pallardy <loic.pallardy@st.com>
>> wrote:
>>>>>>>
>>>>>>> This patch creates a dedicated vdev subdevice for each vdev declared
>>>>>>> in firmware resource table and associates carveout named
>> "vdev%dbuffer"
>>>>>>> (with %d vdev index in resource table) if any as dma coherent
>> memory
>>>>>> pool.
>>>>>>>
>>>>>>> Then vdev subdevice is used as parent for virtio device.
>>>>>>>
>>>>>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>>>>>> ---
>>>>>>>  drivers/remoteproc/remoteproc_core.c     | 35
>>>>>> +++++++++++++++++++++++---
>>>>>>>  drivers/remoteproc/remoteproc_internal.h |  1 +
>>>>>>>  drivers/remoteproc/remoteproc_virtio.c   | 42
>>>>>> +++++++++++++++++++++++++++++++-
>>>>>>>  include/linux/remoteproc.h               |  1 +
>>>>>>>  4 files changed, 75 insertions(+), 4 deletions(-)
>>>>>>>
>>>>>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>>>>>> b/drivers/remoteproc/remoteproc_core.c
>>>>>>> index 4edc6f0..adcc66e 100644
>>>>>>> --- a/drivers/remoteproc/remoteproc_core.c
>>>>>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>>>>>> @@ -39,6 +39,7 @@
>>>>>>>  #include <linux/idr.h>
>>>>>>>  #include <linux/elf.h>
>>>>>>>  #include <linux/crc32.h>
>>>>>>> +#include <linux/of_reserved_mem.h>
>>>>>>>  #include <linux/virtio_ids.h>
>>>>>>>  #include <linux/virtio_ring.h>
>>>>>>>  #include <asm/byteorder.h>
>>>>>>> @@ -145,7 +146,7 @@ static void rproc_disable_iommu(struct rproc
>>>>>> *rproc)
>>>>>>>         iommu_domain_free(domain);
>>>>>>>  }
>>>>>>>
>>>>>>> -static phys_addr_t rproc_va_to_pa(void *cpu_addr)
>>>>>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr)
>>>>>>>  {
>>>>>>>         /*
>>>>>>>          * Return physical address according to virtual address location
>>>>>>> @@ -160,6 +161,7 @@ static phys_addr_t rproc_va_to_pa(void
>>>>>> *cpu_addr)
>>>>>>>         WARN_ON(!virt_addr_valid(cpu_addr));
>>>>>>>         return virt_to_phys(cpu_addr);
>>>>>>>  }
>>>>>>> +EXPORT_SYMBOL(rproc_va_to_pa);
>>>>>>>
>>>>>>>  /**
>>>>>>>   * rproc_da_to_va() - lookup the kernel virtual address for a
>> remoteproc
>>>>>> address
>>>>>>> @@ -423,6 +425,20 @@ static void rproc_vdev_do_stop(struct
>>>>>> rproc_subdev *subdev, bool crashed)
>>>>>>>  }
>>>>>>>
>>>>>>>  /**
>>>>>>> + * rproc_rvdev_release() - release the existence of a rvdev
>>>>>>> + *
>>>>>>> + * @dev: the subdevice's dev
>>>>>>> + */
>>>>>>> +static void rproc_rvdev_release(struct device *dev)
>>>>>>> +{
>>>>>>> +       struct rproc_vdev *rvdev = container_of(dev, struct rproc_vdev,
>> dev);
>>>>>>> +
>>>>>>> +       of_reserved_mem_device_release(dev);
>>>>>>> +
>>>>>>> +       kfree(rvdev);
>>>>>>> +}
>>>>>>> +
>>>>>>> +/**
>>>>>>>   * rproc_handle_vdev() - handle a vdev fw resource
>>>>>>>   * @rproc: the remote processor
>>>>>>>   * @rsc: the vring resource descriptor
>>>>>>> @@ -455,6 +471,7 @@ static int rproc_handle_vdev(struct rproc
>> *rproc,
>>>>>> struct fw_rsc_vdev *rsc,
>>>>>>>         struct device *dev = &rproc->dev;
>>>>>>>         struct rproc_vdev *rvdev;
>>>>>>>         int i, ret;
>>>>>>> +       char name[16];
>>>>>>>
>>>>>>>         /* make sure resource isn't truncated */
>>>>>>>         if (sizeof(*rsc) + rsc->num_of_vrings * sizeof(struct
>>>>>> fw_rsc_vdev_vring)
>>>>>>> @@ -488,6 +505,18 @@ static int rproc_handle_vdev(struct rproc
>> *rproc,
>>>>>> struct fw_rsc_vdev *rsc,
>>>>>>>         rvdev->rproc = rproc;
>>>>>>>         rvdev->index = rproc->nb_vdev++;
>>>>>>>
>>>>>>> +       /* Initialise vdev subdevice */
>>>>>>> +       snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
>>>>>>> +       rvdev->dev.parent = rproc->dev.parent;
>>>>>>> +       rvdev->dev.release = rproc_rvdev_release;
>>>>>>> +       dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev-
>>>>>>> dev.parent), name);
>>>>>>> +       dev_set_drvdata(&rvdev->dev, rvdev);
>>>>>>> +       dma_set_coherent_mask(&rvdev->dev, DMA_BIT_MASK(32));
>>>>>> I tried the latest kernel, this function will not set the DMA coherent
>> mask as
>>>>>> dma_supported() of the &rvdev->dev will return false.
>>>>>> As this is a device created at run time, should it be force to support
>> DMA?
>>>>>> should it directly set the dma_coherent_mask?
>>>>>
>>>>> Thanks for pointing me this issue. I tested on top of 4.18-rc1 few months
>> ago...
>>>>> Could you please give me kernel version on which you are testing the
>> series?
>>>>> Is you platform 32bit or 64bit ?
>>>>> I'll rebase and check on my side.
>>>>
>>>> I am testing with 4.19-rc4 on aarch64 platform.
>>>
>>> Btw, I ran into this on my v7 platform as well (4.19-rc6). The
>>> dma_set_coherent_mask fails with error EIO. I did get my allocations
>>> through though.
>>
>> Correction, that was before Patch 17. With patch 17, this fails.
> 
> Yes normal as device for allocation is changed Patch17. 

The failure was due to dma_alloc failure in rpmsg_probe, your patch13
should have registered the DMA pool for this, and I think the root-cause
is back to the dma_set_coherent_mask failure and unsupported dma_ops on
this device.

regards
Suman

> Regards,
> Loic
>>
>> regards
>> Suman
>>
>>>
>>> regards
>>> Suman
>>>
>>>>
>>>> Best Regards,
>>>> Wendy
>>>>>
>>>>> Regards,
>>>>> Loic
>>>>>
>>>>>>
>>>>>>> +
>>>>>>> +       ret = device_register(&rvdev->dev);
>>>>>>> +       if (ret)
>>>>>>> +               goto free_rvdev;
>>>>>>> +
>>>>>>>         /* parse the vrings */
>>>>>>>         for (i = 0; i < rsc->num_of_vrings; i++) {
>>>>>>>                 ret = rproc_parse_vring(rvdev, rsc, i);
>>>>>>> @@ -518,7 +547,7 @@ static int rproc_handle_vdev(struct rproc
>> *rproc,
>>>>>> struct fw_rsc_vdev *rsc,
>>>>>>>         for (i--; i >= 0; i--)
>>>>>>>                 rproc_free_vring(&rvdev->vring[i]);
>>>>>>>  free_rvdev:
>>>>>>> -       kfree(rvdev);
>>>>>>> +       device_unregister(&rvdev->dev);
>>>>>>>         return ret;
>>>>>>>  }
>>>>>>>
>>>>>>> @@ -536,7 +565,7 @@ void rproc_vdev_release(struct kref *ref)
>>>>>>>
>>>>>>>         rproc_remove_subdev(rproc, &rvdev->subdev);
>>>>>>>         list_del(&rvdev->node);
>>>>>>> -       kfree(rvdev);
>>>>>>> +       device_unregister(&rvdev->dev);
>>>>>>>  }
>>>>>>>
>>>>>>>  /**
>>>>>>> diff --git a/drivers/remoteproc/remoteproc_internal.h
>>>>>> b/drivers/remoteproc/remoteproc_internal.h
>>>>>>> index f6cad24..bfeacfd 100644
>>>>>>> --- a/drivers/remoteproc/remoteproc_internal.h
>>>>>>> +++ b/drivers/remoteproc/remoteproc_internal.h
>>>>>>> @@ -52,6 +52,7 @@ struct dentry *rproc_create_trace_file(const
>> char
>>>>>> *name, struct rproc *rproc,
>>>>>>>  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i);
>>>>>>>
>>>>>>>  void *rproc_da_to_va(struct rproc *rproc, u64 da, int len);
>>>>>>> +phys_addr_t rproc_va_to_pa(void *cpu_addr);
>>>>>>>  int rproc_trigger_recovery(struct rproc *rproc);
>>>>>>>
>>>>>>>  int rproc_elf_sanity_check(struct rproc *rproc, const struct firmware
>> *fw);
>>>>>>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
>>>>>> b/drivers/remoteproc/remoteproc_virtio.c
>>>>>>> index de21f62..9ee63c0 100644
>>>>>>> --- a/drivers/remoteproc/remoteproc_virtio.c
>>>>>>> +++ b/drivers/remoteproc/remoteproc_virtio.c
>>>>>>> @@ -17,7 +17,9 @@
>>>>>>>   * GNU General Public License for more details.
>>>>>>>   */
>>>>>>>
>>>>>>> +#include <linux/dma-mapping.h>
>>>>>>>  #include <linux/export.h>
>>>>>>> +#include <linux/of_reserved_mem.h>
>>>>>>>  #include <linux/remoteproc.h>
>>>>>>>  #include <linux/virtio.h>
>>>>>>>  #include <linux/virtio_config.h>
>>>>>>> @@ -315,10 +317,48 @@ static void rproc_virtio_dev_release(struct
>> device
>>>>>> *dev)
>>>>>>>  int rproc_add_virtio_dev(struct rproc_vdev *rvdev, int id)
>>>>>>>  {
>>>>>>>         struct rproc *rproc = rvdev->rproc;
>>>>>>> -       struct device *dev = &rproc->dev;
>>>>>>> +       struct device *dev = &rvdev->dev;
>>>>>>>         struct virtio_device *vdev = &rvdev->vdev;
>>>>>>> +       struct rproc_mem_entry *mem;
>>>>>>>         int ret;
>>>>>>>
>>>>>>> +       /* Try to find dedicated vdev buffer carveout */
>>>>>>> +       mem = rproc_find_carveout_by_name(rproc, "vdev%dbuffer",
>> rvdev-
>>>>>>> index);
>>>>>>> +       if (mem) {
>>>>>>> +               phys_addr_t pa;
>>>>>>> +
>>>>>>> +               if (mem->of_resm_idx != -1) {
>>>>>>> +                       struct device_node *np = rproc->dev.parent->of_node;
>>>>>>> +
>>>>>>> +                       /* Associate reserved memory to vdev device */
>>>>>>> +                       ret = of_reserved_mem_device_init_by_idx(dev, np,
>>>>>>> +                                                                mem->of_resm_idx);
>>>>>>> +                       if (ret) {
>>>>>>> +                               dev_err(dev, "Can't associate reserved memory\n");
>>>>>>> +                               goto out;
>>>>>>> +                       }
>>>>>>> +               } else {
>>>>>>> +                       if (mem->va) {
>>>>>>> +                               dev_warn(dev, "vdev %d buffer already mapped\n",
>>>>>>> +                                        rvdev->index);
>>>>>>> +                               pa = rproc_va_to_pa(mem->va);
>>>>>>> +                       } else {
>>>>>>> +                               /* Use dma address as carveout no memmapped yet
>> */
>>>>>>> +                               pa = (phys_addr_t)mem->dma;
>>>>>>> +                       }
>>>>>>> +
>>>>>>> +                       /* Associate vdev buffer memory pool to vdev subdev */
>>>>>>> +                       ret = dmam_declare_coherent_memory(dev, pa,
>>>>>>> +                                                          mem->da,
>>>>>>> +                                                          mem->len,
>>>>>>> +                                                          DMA_MEMORY_EXCLUSIVE);
>>>>>>> +                       if (ret < 0) {
>>>>>>> +                               dev_err(dev, "Failed to associate buffer\n");
>>>>>>> +                               goto out;
>>>>>>> +                       }
>>>>>>> +               }
>>>>>>> +       }
>>>>>>> +
>>>>>>>         vdev->id.device = id,
>>>>>>>         vdev->config = &rproc_virtio_config_ops,
>>>>>>>         vdev->dev.parent = dev;
>>>>>>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
>>>>>>> index 6b3a234..2921dd2 100644
>>>>>>> --- a/include/linux/remoteproc.h
>>>>>>> +++ b/include/linux/remoteproc.h
>>>>>>> @@ -547,6 +547,7 @@ struct rproc_vdev {
>>>>>>>         struct kref refcount;
>>>>>>>
>>>>>>>         struct rproc_subdev subdev;
>>>>>>> +       struct device dev;
>>>>>>>
>>>>>>>         unsigned int id;
>>>>>>>         struct list_head node;
>>>>>>> --
>>>>>>> 1.9.1
>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-
>> remoteproc"
>>>>>> in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-
>> info.html
>>>
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 15/17] remoteproc: da8xx: declare reserved memory region for vdev device
  2018-10-24 13:19     ` Loic PALLARDY
@ 2018-10-25 22:11       ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-25 22:11 UTC (permalink / raw)
  To: Loic PALLARDY, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

On 10/24/18 8:19 AM, Loic PALLARDY wrote:
> 
> 
>> -----Original Message-----
>> From: Suman Anna <s-anna@ti.com>
>> Sent: mercredi 24 octobre 2018 04:58
>> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
>> ohad@wizery.com
>> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
>> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
>> benjamin.gaignard@linaro.org
>> Subject: Re: [PATCH v4 15/17] remoteproc: da8xx: declare reserved memory
>> region for vdev device
>>
>> Hi Loic,
>>
>> On 7/27/18 8:14 AM, Loic Pallardy wrote:
>>> This patch introduces da8xx_rproc_parse_fw() to declare a
>>> carveout region based on reserved memory for vdev buffer
>>> allocation.
>>>
>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>> ---
>>>  drivers/remoteproc/da8xx_remoteproc.c | 38
>> +++++++++++++++++++++++++++++++++++
>>>  1 file changed, 38 insertions(+)
>>>
>>> diff --git a/drivers/remoteproc/da8xx_remoteproc.c
>> b/drivers/remoteproc/da8xx_remoteproc.c
>>> index b668e32..679a076 100644
>>> --- a/drivers/remoteproc/da8xx_remoteproc.c
>>> +++ b/drivers/remoteproc/da8xx_remoteproc.c
>>> @@ -16,6 +16,7 @@
>>>  #include <linux/irq.h>
>>>  #include <linux/kernel.h>
>>>  #include <linux/module.h>
>>> +#include <linux/of_address.h>
>>>  #include <linux/of_reserved_mem.h>
>>>  #include <linux/platform_device.h>
>>>  #include <linux/remoteproc.h>
>>> @@ -179,10 +180,47 @@ static void da8xx_rproc_kick(struct rproc *rproc,
>> int vqid)
>>>  	writel(SYSCFG_CHIPSIG2, drproc->chipsig);
>>>  }
>>>
>>> +static int da8xx_rproc_parse_fw(struct rproc *rproc, const struct firmware
>> *fw)
>>> +{
>>> +	struct device *dev = rproc->dev.parent;
>>> +	struct rproc_mem_entry *mem;
>>> +	struct device_node *node;
>>> +	struct resource res;
>>> +	int err;
>>> +
>>> +	node = of_parse_phandle(dev->of_node, "memory-region", 0);
>>> +	if (!node) {
>>> +		dev_err(dev, "No memory-region specified\n");
>>> +		return -EINVAL;
>>> +	}
>>> +
>>> +	err = of_address_to_resource(node, 0, &res);
>>> +	if (err) {
>>> +		dev_err(dev, "Bad memory-region definition\n");
>>> +		return err;
>>> +	}
>>> +
>>> +	/* Register memory region for vdev buffer allocation */
>>> +	mem = rproc_of_resm_mem_entry_init(dev, 0,
>> resource_size(&res),
>>> +					   res.start, "vdev0buffer");> +
>>> +	if (!mem)
>>> +		return -ENOMEM;
>>> +
>>> +	rproc_add_carveout(rproc, mem);
>>> +
>>> +	return rproc_elf_load_rsc_table(rproc, fw);
>>> +}
>>
>> Thanks for the patch, but this creates a kernel crash for me due to
>> overlaps with manually created carveouts. I currently have a single
>> memory-region and all allocations come from the same DMA pool, but the
>> rproc_of_resm_mem_entry_init() creates an overall mem entry without the
>> va being set (no alloc function plumbed in). In general, it is permitted
>> to use the same reserved-memory node with multiple devices, so the index
>> usage should have allowed it to do DMA allocations with vdev devices,
>> but the loading is performed even before the vdev allocations and the
>> da_to_va matches the first entry with no va set causing the crash.
> 
> Hummm, I didn't fall in this case, but clearly da_to_va should not crashed. 
> Not allocated carveout should be bypassed in the loop. Thanks for pointing this. I need to fix it.

da_to_va didn't crash, it just returned a bogus va based on base address
0x0 (as you can see below) as it looks through all carveouts, and my
loading crashed. This brings us to fact that we now need to distinguish
allocated carveouts vs non-allocated carveouts.

regards
Suman

> 
> The rproc_of_resm_mem_entry_init() is simply registering the reserved memory to be attached to vdev device.
> So that normal it won't be allocated by rproc core (there is no alloc/free function specificied in this helper). 
> 
> Regards,
> Loic
>>
>> Here's my debugfs output of the carveout_memories for reference,
>>
>> Carveout memory entry:
>>         Name: vdev0buffer
>>         Virtual address: 00000000
>>         DMA address: 0x00000000
>>         Device address: 0xc3000000
>>         Length: 0x1000000 Bytes
>>
>> Carveout memory entry:
>>         Name: vdev0vring0
>>         Virtual address: c3000000
>>         DMA address: 0xc3000000
>>         Device address: 0xc3000000
>>         Length: 0x3000 Bytes
>>
>> Carveout memory entry:
>>         Name: vdev0vring1
>>         Virtual address: c3004000
>>         DMA address: 0xc3004000
>>         Device address: 0xc3004000
>>         Length: 0x3000 Bytes
>>
>> Carveout memory entry:
>>         Name: DSP_MEM_DATA
>>         Virtual address: c3100000
>>         DMA address: 0xc3100000
>>         Device address: 0xc3100000
>>         Length: 0xf00000 Bytes
>>
>> You can drop both this patch and the keystone_remoteproc patch from the
>> series. I did not run into any issues there since I did not have any
>> RSC_CARVEOUT entries there. Also, see my comments on the next patch
>> (the
>> changes in ST) in general regarding these API. Looks like this needs
>> some more time in ironing out the issues.
>>
>> regards
>> Suman
>>
>>
>>
>>> +
>>>  static const struct rproc_ops da8xx_rproc_ops = {
>>>  	.start = da8xx_rproc_start,
>>>  	.stop = da8xx_rproc_stop,
>>>  	.kick = da8xx_rproc_kick,
>>> +	.parse_fw = da8xx_rproc_parse_fw,
>>> +	.load = rproc_elf_load_segments,
>>> +	.find_loaded_rsc_table = rproc_elf_find_loaded_rsc_table,
>>> +	.sanity_check = rproc_elf_sanity_check,
>>> +	.get_boot_addr = rproc_elf_get_boot_addr,
>>>  };
>>>
>>>  static int da8xx_rproc_get_internal_memories(struct platform_device
>> *pdev,
>>>
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 08/17] remoteproc: add alloc ops in rproc_mem_entry struct
  2018-10-24 16:00     ` Loic PALLARDY
@ 2018-10-25 22:37       ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-25 22:37 UTC (permalink / raw)
  To: Loic PALLARDY, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

Hi Loic,

> 
>>
>> On 7/27/18 8:14 AM, Loic Pallardy wrote:
>>> Memory entry could be allocated in different ways (ioremap,
>>> dma_alloc_coherent, internal RAM allocator...).
>>> This patch introduces an alloc ops in rproc_mem_entry structure
>>> to associate dedicated allocation mechanism to each memory entry
>>> descriptor in order to do remote core agnostic from memory allocators.
>>>
>>> The introduction of this ops allows to perform allocation of all registered
>>> carveout at the same time, just before calling rproc_start().
>>> It simplifies and makes uniform carveout management whatever origin.
>>
>> This patch is causing a kernel crash with trace entries. Please see
>> further below for the cause.
>>
>>>
>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>> ---
>>>  drivers/remoteproc/remoteproc_core.c | 261
>> ++++++++++++++++++++++-------------
>>>  include/linux/remoteproc.h           |   7 +
>>>  2 files changed, 175 insertions(+), 93 deletions(-)
>>>
>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>> b/drivers/remoteproc/remoteproc_core.c
>>> index 77b39ba..2c51549 100644
>>> --- a/drivers/remoteproc/remoteproc_core.c
>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>> @@ -642,74 +642,31 @@ static int rproc_handle_devmem(struct rproc
>> *rproc, struct fw_rsc_devmem *rsc,
>>>  }
>>>
>>>  /**
>>> - * rproc_release_carveout() - release acquired carveout
>>> + * rproc_alloc_carveout() - allocated specified carveout
>>>   * @rproc: rproc handle
>>> - * @mem: the memory entry to release
>>> - *
>>> - * This function releases specified memory entry @mem allocated via
>>> - * dma_alloc_coherent() function by @rproc.
>>> - */
>>> -static int rproc_release_carveout(struct rproc *rproc,
>>> -				  struct rproc_mem_entry *mem)
>>> -{
>>> -	struct device *dev = &rproc->dev;
>>> -
>>> -	/* clean up carveout allocations */
>>> -	dma_free_coherent(dev->parent, mem->len, mem->va, mem-
>>> dma);
>>> -	return 0;
>>> -}
>>> -
>>> -/**
>>> - * rproc_handle_carveout() - handle phys contig memory allocation
>> requests
>>> - * @rproc: rproc handle
>>> - * @rsc: the resource entry
>>> - * @avail: size of available data (for image validation)
>>> - *
>>> - * This function will handle firmware requests for allocation of physically
>>> - * contiguous memory regions.
>>> - *
>>> - * These request entries should come first in the firmware's resource
>> table,
>>> - * as other firmware entries might request placing other data objects
>> inside
>>> - * these memory regions (e.g. data/code segments, trace resource
>> entries, ...).
>>> + * @mem: the memory entry to allocate
>>>   *
>>> - * Allocating memory this way helps utilizing the reserved physical memory
>>> - * (e.g. CMA) more efficiently, and also minimizes the number of TLB
>> entries
>>> - * needed to map it (in case @rproc is using an IOMMU). Reducing the TLB
>>> - * pressure is important; it may have a substantial impact on performance.
>>> + * This function allocate specified memory entry @mem using
>>> + * dma_alloc_coherent() as default allocator
>>>   */
>>> -static int rproc_handle_carveout(struct rproc *rproc,
>>> -				 struct fw_rsc_carveout *rsc,
>>> -				 int offset, int avail)
>>> +static int rproc_alloc_carveout(struct rproc *rproc,
>>> +				struct rproc_mem_entry *mem)
>>>  {
>>> -	struct rproc_mem_entry *carveout, *mapping = NULL;
>>> +	struct rproc_mem_entry *mapping = NULL;
>>>  	struct device *dev = &rproc->dev;
>>>  	dma_addr_t dma;
>>>  	void *va;
>>>  	int ret;
>>>
>>> -	if (sizeof(*rsc) > avail) {
>>> -		dev_err(dev, "carveout rsc is truncated\n");
>>> -		return -EINVAL;
>>> -	}
>>> -
>>> -	/* make sure reserved bytes are zeroes */
>>> -	if (rsc->reserved) {
>>> -		dev_err(dev, "carveout rsc has non zero reserved bytes\n");
>>> -		return -EINVAL;
>>> -	}
>>> -
>>> -	dev_dbg(dev, "carveout rsc: name: %s, da 0x%x, pa 0x%x, len 0x%x,
>> flags 0x%x\n",
>>> -		rsc->name, rsc->da, rsc->pa, rsc->len, rsc->flags);
>>> -
>>> -	va = dma_alloc_coherent(dev->parent, rsc->len, &dma,
>> GFP_KERNEL);
>>> +	va = dma_alloc_coherent(dev->parent, mem->len, &dma,
>> GFP_KERNEL);
>>>  	if (!va) {
>>>  		dev_err(dev->parent,
>>> -			"failed to allocate dma memory: len 0x%x\n", rsc-
>>> len);
>>> +			"failed to allocate dma memory: len 0x%x\n", mem-
>>> len);
>>>  		return -ENOMEM;
>>>  	}
>>>
>>>  	dev_dbg(dev, "carveout va %pK, dma %pad, len 0x%x\n",
>>> -		va, &dma, rsc->len);
>>> +		va, &dma, mem->len);
>>>
>>>  	/*
>>>  	 * Ok, this is non-standard.
>>> @@ -729,22 +686,22 @@ static int rproc_handle_carveout(struct rproc
>> *rproc,
>>>  	 * physical address in this case.
>>>  	 */
>>>
>>> -	if (rsc->da != FW_RSC_ADDR_ANY && !rproc->domain) {
>>> -		dev_err(dev->parent,
>>> -			"Bad carveout rsc configuration\n");
>>> -		ret = -ENOMEM;
>>> -		goto dma_free;
>>> -	}
>>> +	if (mem->da != FW_RSC_ADDR_ANY) {
>>> +		if (!rproc->domain) {
>>> +			dev_err(dev->parent,
>>> +				"Bad carveout rsc configuration\n");
>>> +			ret = -ENOMEM;
>>> +			goto dma_free;
>>> +		}
>>
>> Same comment from Patch 1.
>>
>>>
>>> -	if (rsc->da != FW_RSC_ADDR_ANY && rproc->domain) {
>>>  		mapping = kzalloc(sizeof(*mapping), GFP_KERNEL);
>>>  		if (!mapping) {
>>>  			ret = -ENOMEM;
>>>  			goto dma_free;
>>>  		}
>>>
>>> -		ret = iommu_map(rproc->domain, rsc->da, dma, rsc->len,
>>> -				rsc->flags);
>>> +		ret = iommu_map(rproc->domain, mem->da, dma, mem-
>>> len,
>>> +				mem->flags);
>>>  		if (ret) {
>>>  			dev_err(dev, "iommu_map failed: %d\n", ret);
>>>  			goto free_mapping;
>>> @@ -757,52 +714,102 @@ static int rproc_handle_carveout(struct rproc
>> *rproc,
>>>  		 * We can't trust the remote processor not to change the
>>>  		 * resource table, so we must maintain this info
>> independently.
>>>  		 */
>>> -		mapping->da = rsc->da;
>>> -		mapping->len = rsc->len;
>>> +		mapping->da = mem->da;
>>> +		mapping->len = mem->len;
>>>  		list_add_tail(&mapping->node, &rproc->mappings);
>>>
>>>  		dev_dbg(dev, "carveout mapped 0x%x to %pad\n",
>>> -			rsc->da, &dma);
>>> +			mem->da, &dma);
>>> +	} else {
>>> +		mem->da = (u32)dma;
>>
>> Hmm, what was the purpose of this? So, this appears to be handling the
>> missing implementation for the comment in the fw_rsc_carveout about
>> FW_RSC_ADDR_ANY.
> 
> It is needed to update da in rsc table when da is equal to FW_RSC_ADDR_ANY. No force update when da is fixed to another value.
> It is the only way to provide information to coprocessor about carveout dynamically allocated by rproc core.
> Coprocessor doesn't care about pa. Only da is valid from his pov.

Hmm, I think this is more to do specifically with the previous vring
handling that we were done. The previous alloc_vring would actually
overwrite the da value with the dma value. Unfortunately the vring
resource structure does not have any pa field, so the da was changed.
This was changed sometime back by Sjur in commit c0d631570ad54
("remoteproc: set vring addresses in resource table"), and we had to
modify our code on the RTOS side. The fw_rsc_carveout actually has a pa
field that we actually filled in and used by remote-side for any da to
pa translations (like for DMA operations on remote-side).

This goes back to some of my other comments about treating da - we are
forcing the behavior that this is a dma address (on most  platforms
dma_addr = phys_addr), when da actually represents a device address.
Question is if it is a reasonable approach/assumption for the long-term?

regards
Suman


> 
>>
>>>  	}
>>>
>>> -	/*
>>> -	 * Some remote processors might need to know the pa
>>> -	 * even though they are behind an IOMMU. E.g., OMAP4's
>>> -	 * remote M3 processor needs this so it can control
>>> -	 * on-chip hardware accelerators that are not behind
>>> -	 * the IOMMU, and therefor must know the pa.
>>> -	 *
>>> -	 * Generally we don't want to expose physical addresses
>>> -	 * if we don't have to (remote processors are generally
>>> -	 * _not_ trusted), so we might want to do this only for
>>> -	 * remote processor that _must_ have this (e.g. OMAP4's
>>> -	 * dual M3 subsystem).
>>> -	 *
>>> -	 * Non-IOMMU processors might also want to have this info.
>>> -	 * In this case, the device address and the physical address
>>> -	 * are the same.
>>> -	 */
>>> -	rsc->pa = (u32)rproc_va_to_pa(va);
>>> -
>>> -	carveout = rproc_mem_entry_init(dev, va, dma, rsc->len, rsc->da,
>>> -					rproc_release_carveout, rsc->name);
>>> -	if (!carveout)
>>> -		goto free_carv;
>>> -
>>> -	rproc_add_carveout(rproc, carveout);
>>> +	mem->dma = (u32)dma;
>>
>> We don't need the typecast, mem->dma is already of type dma_addr_t.
>> Same
>> comment above on the else part as well.
> 
> Ok for cast
>>
>>> +	mem->va = va;
>>>
>>>  	return 0;
>>>
>>> -free_carv:
>>> -	kfree(carveout);
>>>  free_mapping:
>>>  	kfree(mapping);
>>>  dma_free:
>>> -	dma_free_coherent(dev->parent, rsc->len, va, dma);
>>> +	dma_free_coherent(dev->parent, mem->len, va, dma);
>>>  	return ret;
>>>  }
>>>
>>>  /**
>>> + * rproc_release_carveout() - release acquired carveout
>>> + * @rproc: rproc handle
>>> + * @mem: the memory entry to release
>>> + *
>>> + * This function releases specified memory entry @mem allocated via
>>> + * rproc_alloc_carveout() function by @rproc.
>>> + */
>>> +static int rproc_release_carveout(struct rproc *rproc,
>>> +				  struct rproc_mem_entry *mem)
>>> +{
>>> +	struct device *dev = &rproc->dev;
>>> +
>>> +	/* clean up carveout allocations */
>>> +	dma_free_coherent(dev->parent, mem->len, mem->va, mem-
>>> dma);
>>> +	return 0;
>>> +}
>>> +
>>> +/**
>>> + * rproc_handle_carveout() - handle phys contig memory allocation
>> requests
>>> + * @rproc: rproc handle
>>> + * @rsc: the resource entry
>>> + * @avail: size of available data (for image validation)
>>> + *
>>> + * This function will handle firmware requests for allocation of physically
>>> + * contiguous memory regions.
>>> + *
>>> + * These request entries should come first in the firmware's resource
>> table,
>>> + * as other firmware entries might request placing other data objects
>> inside
>>> + * these memory regions (e.g. data/code segments, trace resource
>> entries, ...).
>>> + *
>>> + * Allocating memory this way helps utilizing the reserved physical
>> memory
>>> + * (e.g. CMA) more efficiently, and also minimizes the number of TLB
>> entries
>>> + * needed to map it (in case @rproc is using an IOMMU). Reducing the TLB
>>> + * pressure is important; it may have a substantial impact on performance.
>>> + */
>>> +static int rproc_handle_carveout(struct rproc *rproc,
>>> +				 struct fw_rsc_carveout *rsc,
>>> +				 int offset, int avail)
>>> +{
>>> +	struct rproc_mem_entry *carveout;
>>> +	struct device *dev = &rproc->dev;
>>> +
>>> +	if (sizeof(*rsc) > avail) {
>>> +		dev_err(dev, "carveout rsc is truncated\n");
>>> +		return -EINVAL;
>>> +	}
>>> +
>>> +	/* make sure reserved bytes are zeroes */
>>> +	if (rsc->reserved) {
>>> +		dev_err(dev, "carveout rsc has non zero reserved bytes\n");
>>> +		return -EINVAL;
>>> +	}
>>> +
>>> +	dev_dbg(dev, "carveout rsc: name: %s, da 0x%x, pa 0x%x, len 0x%x,
>> flags 0x%x\n",
>>> +		rsc->name, rsc->da, rsc->pa, rsc->len, rsc->flags);
>>> +
>>> +	/* Register carveout in in list */
>>> +	carveout = rproc_mem_entry_init(dev, 0, 0, rsc->len, rsc->da,
>>> +					rproc_alloc_carveout,
>>> +					rproc_release_carveout, rsc->name);
>>> +	if (!carveout) {
>>> +		dev_err(dev, "Can't allocate memory entry structure\n");
>>> +		return -ENOMEM;
>>> +	}
>>> +
>>> +	carveout->flags = rsc->flags;
>>> +	carveout->rsc_offset = offset;
>>> +	rproc_add_carveout(rproc, carveout);
>>
>> Once we get rid of rproc_add_carveout, the list addition will mostly be
>> handled in rproc_mem_entry_init itself.
> 
> Yes I have to figure out what could be the impact to call rproc_add_carveout() from rproc_mem_entry_init(). It should work...
> 
>>
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +/**
>>>   * rproc_add_carveout() - register an allocated carveout region
>>>   * @rproc: rproc handle
>>>   * @mem: memory entry to register
>>> @@ -832,6 +839,7 @@ void rproc_add_carveout(struct rproc *rproc, struct
>> rproc_mem_entry *mem)
>>>  struct rproc_mem_entry *
>>>  rproc_mem_entry_init(struct device *dev,
>>>  		     void *va, dma_addr_t dma, int len, u32 da,
>>> +		     int (*alloc)(struct rproc *, struct rproc_mem_entry *),
>>>  		     int (*release)(struct rproc *, struct rproc_mem_entry *),
>>>  		     const char *name, ...)
>>>  {
>>> @@ -846,7 +854,9 @@ struct rproc_mem_entry *
>>>  	mem->dma = dma;
>>>  	mem->da = da;
>>>  	mem->len = len;
>>> +	mem->alloc = alloc;
>>>  	mem->release = release;
>>> +	mem->rsc_offset = FW_RSC_ADDR_ANY;
>>>
>>>  	va_start(args, name);
>>>  	vsnprintf(mem->name, sizeof(mem->name), name, args);
>>> @@ -978,6 +988,63 @@ static void rproc_unprepare_subdevices(struct
>> rproc *rproc)
>>>  }
>>>
>>>  /**
>>> + * rproc_alloc_registered_carveouts() - allocate all carveouts registered
>>> + * in the list
>>> + * @rproc: the remote processor handle
>>> + *
>>> + * This function parses registered carveout list, performs allocation
>>> + * if alloc() ops registered and updates resource table information
>>> + * if rsc_offset set.
>>> + *
>>> + * Return: 0 on success
>>> + */
>>> +static int rproc_alloc_registered_carveouts(struct rproc *rproc)
>>> +{
>>> +	struct rproc_mem_entry *entry, *tmp;
>>> +	struct fw_rsc_carveout *rsc;
>>> +	struct device *dev = &rproc->dev;
>>> +	int ret;
>>> +
>>> +	list_for_each_entry_safe(entry, tmp, &rproc->carveouts, node) {
>>> +		if (entry->alloc) {
>>> +			ret = entry->alloc(rproc, entry);
>>> +			if (ret) {
>>> +				dev_err(dev, "Unable to allocate carveout
>> %s: %d\n",
>>> +					entry->name, ret);
>>> +				return -ENOMEM;
>>> +			}
>>> +		}
>>> +
>>> +		if (entry->rsc_offset != FW_RSC_ADDR_ANY) {
>>> +			/* update resource table */
>>> +			rsc = (void *)rproc->table_ptr + entry->rsc_offset;
>>> +
>>> +			/*
>>> +			 * Some remote processors might need to know the
>> pa
>>> +			 * even though they are behind an IOMMU. E.g.,
>> OMAP4's
>>> +			 * remote M3 processor needs this so it can control
>>> +			 * on-chip hardware accelerators that are not behind
>>> +			 * the IOMMU, and therefor must know the pa.
>>> +			 *
>>> +			 * Generally we don't want to expose physical
>> addresses
>>> +			 * if we don't have to (remote processors are
>> generally
>>> +			 * _not_ trusted), so we might want to do this only
>> for
>>> +			 * remote processor that _must_ have this (e.g.
>> OMAP4's
>>> +			 * dual M3 subsystem).
>>> +			 *
>>> +			 * Non-IOMMU processors might also want to have
>> this info.
>>> +			 * In this case, the device address and the physical
>> address
>>> +			 * are the same.
>>> +			 */
>>> +			if (entry->va)
>>> +				rsc->pa = (u32)rproc_va_to_pa(entry->va);
>>> +		}
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +/**
>>>   * rproc_coredump_cleanup() - clean up dump_segments list
>>>   * @rproc: the remote processor handle
>>>   */
>>> @@ -1148,6 +1215,14 @@ static int rproc_fw_boot(struct rproc *rproc,
>> const struct firmware *fw)
>>>  		goto clean_up_resources;
>>>  	}
>>>
>>> +	/* Allocate carveout resources associated to rproc */
>>> +	ret = rproc_alloc_registered_carveouts(rproc);
>>> +	if (ret) {
>>> +		dev_err(dev, "Failed to allocate associated carveouts: %d\n",
>>> +			ret);
>>> +		goto clean_up_resources;
>>> +	}
>>
>> This is causing an issue with RSC_TRACE on where the trace region on the
>> remote processor is actually backed by a DDR carveout address. The
>> allocations are now being done after processing the resources from the
>> rproc_loading_handlers, which causes the RSC_TRACE to be configured with
>> an incorrect kernel va, and accessing it through debugfs then results in
>> a kernel crash.
> 
> Ok I understand the issue. Mask by the double mapping you pointing in my driver.
> I need to correct that.
> Thanks for pointing it.
> 
> Regards,
> Loic
> 
>>
>> regards
>> Suman
>>
>>> +
>>>  	ret = rproc_start(rproc, fw);
>>>  	if (ret)
>>>  		goto clean_up_resources;
>>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
>>> index 55f30fc..ea95b04 100644
>>> --- a/include/linux/remoteproc.h
>>> +++ b/include/linux/remoteproc.h
>>> @@ -317,6 +317,9 @@ struct fw_rsc_vdev {
>>>   * @priv: associated data
>>>   * @name: associated memory region name (optional)
>>>   * @node: list node
>>> + * @rsc_offset: offset in resource table
>>> + * @flags: iommu protection flags
>>> + * @alloc: specific memory allocator function
>>>   */
>>>  struct rproc_mem_entry {
>>>  	void *va;
>>> @@ -326,6 +329,9 @@ struct rproc_mem_entry {
>>>  	void *priv;
>>>  	char name[32];
>>>  	struct list_head node;
>>> +	u32 rsc_offset;
>>> +	u32 flags;
>>> +	int (*alloc)(struct rproc *rproc, struct rproc_mem_entry *mem);
>>>  	int (*release)(struct rproc *rproc, struct rproc_mem_entry *mem);
>>>  };
>>>
>>> @@ -563,6 +569,7 @@ struct rproc *rproc_alloc(struct device *dev, const
>> char *name,
>>>  struct rproc_mem_entry *
>>>  rproc_mem_entry_init(struct device *dev,
>>>  		     void *va, dma_addr_t dma, int len, u32 da,
>>> +		     int (*alloc)(struct rproc *, struct rproc_mem_entry *),
>>>  		     int (*release)(struct rproc *, struct rproc_mem_entry *),
>>>  		     const char *name, ...);
>>>
>>>
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 10/17] remoteproc: add helper function to check carveout device address
  2018-10-24 15:24     ` Loic PALLARDY
@ 2018-10-25 22:50       ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-25 22:50 UTC (permalink / raw)
  To: Loic PALLARDY, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

Hi Loic,

On 10/24/18 10:24 AM, Loic PALLARDY wrote:
> Hi Suman,
> 
>> -----Original Message-----
>> From: Suman Anna <s-anna@ti.com>
>> Sent: mercredi 24 octobre 2018 00:14
>> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
>> ohad@wizery.com
>> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
>> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
>> benjamin.gaignard@linaro.org
>> Subject: Re: [PATCH v4 10/17] remoteproc: add helper function to check
>> carveout device address
>>
>> Hi Loic,
>>
>> On 7/27/18 8:14 AM, Loic Pallardy wrote:
>>> This patch introduces a function to verify that a specified carveout
>>> is fitting request device address and associated length
>>>
>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>> ---
>>>  drivers/remoteproc/remoteproc_core.c | 47
>> ++++++++++++++++++++++++++++++++++++
>>>  1 file changed, 47 insertions(+)
>>>
>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>> b/drivers/remoteproc/remoteproc_core.c
>>> index 1e0fe3e..5dd5edf 100644
>>> --- a/drivers/remoteproc/remoteproc_core.c
>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>> @@ -259,6 +259,53 @@ struct rproc_mem_entry *
>>>  	return mem;
>>>  }
>>>
>>> +/**
>>> + * rproc_check_carveout_da() - Check specified carveout da configuration
>>> + * @rproc: handle of a remote processor
>>> + * @mem: pointer on carveout to check
>>> + * @da: area device address
>>> + * @len: associated area size
>>> + *
>>> + * This function is a helper function to verify requested device area
>> (couple
>>> + * da, len) is part of specified carevout.
>>
>> %s/carevout/carveout/
> OK
>>
>>> + *
>>> + * Return: 0 if carveout matchs request else -ENOMEM
>>
>> %s/matchs/matches/
> OK
>>
>>> + */
>>> +int rproc_check_carveout_da(struct rproc *rproc, struct
>> rproc_mem_entry *mem,
>>
>> static int since this seems to be only a local function.
> OK
>>
>>> +			    u32 da, u32 len)
>>> +{
>>> +	struct device *dev = &rproc->dev;
>>> +	int delta = 0;
>>> +
>>> +	/* Check requested resource length */
>>> +	if (len > mem->len) {
>>> +		dev_err(dev, "Registered carveout doesn't fit len
>> request\n");
>>> +		return -ENOMEM;
>>
>> ENOMEM not typically used for these kind of errors, you were probably
>> inclined to used this since it is dealing with memory.
> 
> -EINVAL will be better
>>
>>> +	}
>>> +
>>
>> Both the below codepaths are exercised only when da is not
>> FW_RSC_ADDR_ANY, and you are returning 0 otherwise (which is the case of
>> matches as per your description above). Is that what you really want -
>> should it be an error
> 
> Yes when da is equal to FW_RSC_ADDR_ANY we should check the length too

Can you update the comments in the function description accordingly as
well, the current code silently returns 0 if da = FW_RSC_ADDR_ANY.

> 
>>
>>> +	if (da != FW_RSC_ADDR_ANY && mem->da == FW_RSC_ADDR_ANY)
>> {
>>> +		/* Update existing carveout da */
>>> +		mem->da = da;
>>
>> Where would you need to update this?

> In that case, we have 2 carveouts with the same name.
> One has some fixed requests. The other one has none.
> The goal here is to align both on the one which has the strongest requirements.
> I think length is missing.

It almost looks like there is a need for range overlap checks on all the
carveouts after all of them are registered, and error out if any overlap
irrespective of the name schema.

regards
Suman

> 
> Regards,
> Loic
> 
>>
>> regards
>> Suman
>>
>>> +	} else if (da != FW_RSC_ADDR_ANY && mem->da !=
>> FW_RSC_ADDR_ANY) {
>>> +		delta = da - mem->da;
>>> +
>>> +		/* Check requested resource belongs to registered carveout
>> */
>>> +		if (delta < 0) {
>>> +			dev_err(dev,
>>> +				"Registered carveout doesn't fit da
>> request\n");
>>> +			return -ENOMEM;
>>> +		}
>>> +
>>> +		if (delta + len > mem->len) {
>>> +			dev_err(dev,
>>> +				"Registered carveout doesn't fit len
>> request\n");
>>> +			return -ENOMEM;
>>> +		}
>>> +	}
>>> +
>>> +	return 0;
>>
>>
>>> +}
>>> +
>>>  int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
>>>  {
>>>  	struct rproc *rproc = rvdev->rproc;
>>>
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function
  2018-10-24 12:58         ` Loic PALLARDY
@ 2018-10-25 22:50           ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-25 22:50 UTC (permalink / raw)
  To: Loic PALLARDY, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

On 10/24/18 7:58 AM, Loic PALLARDY wrote:
> 
> 
>> -----Original Message-----
>> From: Suman Anna <s-anna@ti.com>
>> Sent: mercredi 24 octobre 2018 05:19
>> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
>> ohad@wizery.com
>> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
>> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
>> benjamin.gaignard@linaro.org
>> Subject: Re: [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function
>>
>> On 10/23/18 2:51 PM, Loic PALLARDY wrote:
>>> Hi Suman,
>>>
>>>> -----Original Message-----
>>>> From: Suman Anna <s-anna@ti.com>
>>>> Sent: mardi 23 octobre 2018 18:51
>>>> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
>>>> ohad@wizery.com
>>>> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
>>>> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
>>>> benjamin.gaignard@linaro.org
>>>> Subject: Re: [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function
>>>>
>>>> Hi Loic, Bjorn,
>>>>
>>>> On 7/27/18 8:14 AM, Loic Pallardy wrote:
>>>>> This new function translates CPU virtual address in
>>>>> CPU physical one according to virtual address location.
>>>>>
>>>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>>>> Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org>
>>>>> ---
>>>>>  drivers/remoteproc/remoteproc_core.c | 18 +++++++++++++++++-
>>>>>  1 file changed, 17 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>>>> b/drivers/remoteproc/remoteproc_core.c
>>>>> index 437fabf..8e5fe1e 100644
>>>>> --- a/drivers/remoteproc/remoteproc_core.c
>>>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>>>> @@ -140,6 +140,22 @@ static void rproc_disable_iommu(struct rproc
>>>> *rproc)
>>>>>  	iommu_domain_free(domain);
>>>>>  }
>>>>>
>>>>> +static phys_addr_t rproc_va_to_pa(void *cpu_addr)
>>>>> +{
>>>>> +	/*
>>>>> +	 * Return physical address according to virtual address location
>>>>> +	 * - in vmalloc: if region ioremapped or defined as
>>>> dma_alloc_coherent
>>>>> +	 * - in kernel: if region allocated in generic dma memory pool
>>>>> +	 */
>>>>> +	if (is_vmalloc_addr(cpu_addr)) {
>>>>> +		return page_to_phys(vmalloc_to_page(cpu_addr)) +
>>>>> +				    offset_in_page(cpu_addr);
>>>>> +	}
>>>>> +
>>>>> +	WARN_ON(!virt_addr_valid(cpu_addr));
>>>>> +	return virt_to_phys(cpu_addr);
>>>>> +}
>>>>> +
>>>>>  /**
>>>>>   * rproc_da_to_va() - lookup the kernel virtual address for a remoteproc
>>>> address
>>>>>   * @rproc: handle of a remote processor
>>>>> @@ -711,7 +727,7 @@ static int rproc_handle_carveout(struct rproc
>>>> *rproc,
>>>>>  	 * In this case, the device address and the physical address
>>>>>  	 * are the same.
>>>>>  	 */
>>>>> -	rsc->pa = dma;
>>>>> +	rsc->pa = (u32)rproc_va_to_pa(va);
>>>>
>>>> While I agree with the direction here, we ought to add a check here
>>>> warning users if some address bits are getting lost as a result of the
>>>> typecast. Granted the issue may have been present previously with
>>>> dma_addr_t as well, but most platforms were using 32-bit dma addresses,
>>>> so this was kinda masked. There are ARMv7 platforms with LPAE enabled
>>>> allowing physical addresses > 32-bits.
>>>>
>>>> In anycase, we definitely have a need for a v2 for the fw_rsc_carveout
>>>> structure to deal with 64-bit addresses.
>>>>
>>>
>>> Agree with you.
>>> Assumption for this series was to keep resource table as it is. Resource
>> table improvement is planned in a second step.
>>
>> Perhaps, we should add a WARN_ON for the time being until we enhance
>> the
>> resource table for 64-bit platforms/addresses.
> 
> OK I will propose a patch to add WARN_ON on cast applied on resource table field

Yes, thanks.

regards
Suman

> Regards,
> Loic
>>
>> regards
>> Suman
>>
>>> Regards,
>>> Loic
>>>
>>>> regards
>>>> Suman
>>>>
>>>>>
>>>>>  	carveout->va = va;
>>>>>  	carveout->len = rsc->len;
>>>>>
>>>
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 01/17] remoteproc: configure IOMMU only if device address requested
  2018-10-24 12:56         ` Loic PALLARDY
@ 2018-10-26  0:46           ` Suman Anna
  0 siblings, 0 replies; 75+ messages in thread
From: Suman Anna @ 2018-10-26  0:46 UTC (permalink / raw)
  To: Loic PALLARDY, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

Hi Loic,

> >> -----Original Message-----
>> From: Suman Anna <s-anna@ti.com>
>> Sent: mercredi 24 octobre 2018 05:47
>> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
>> ohad@wizery.com
>> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
>> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
>> benjamin.gaignard@linaro.org
>> Subject: Re: [PATCH v4 01/17] remoteproc: configure IOMMU only if device
>> address requested
>>
>> On 10/23/18 2:40 PM, Loic PALLARDY wrote:
>>> Hi Suman,
>>>
>>>> -----Original Message-----
>>>> From: Suman Anna <s-anna@ti.com>
>>>> Sent: mardi 23 octobre 2018 19:26
>>>> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
>>>> ohad@wizery.com
>>>> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
>>>> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
>>>> benjamin.gaignard@linaro.org
>>>> Subject: Re: [PATCH v4 01/17] remoteproc: configure IOMMU only if
>> device
>>>> address requested
>>>>
>>>> Hi Loic,
>>>>
>>>> On 7/27/18 8:14 AM, Loic Pallardy wrote:
>>>>> If there is no IOMMU associate to remote processor device,
>>>>> remoteproc_core won't be able to satisfy device address requested
>>>>> in firmware resource table.
>>>>> Return an error as configuration won't be coherent.
>>>>>
>>>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>>>
>>>> This patch is breaking my Davinci platforms. It is not really required
>>>> that you _should_ have IOMMUs when a valid DA is mentioned. Please
>> see
>>>> the existing description (paras 4 and 5) on the fw_rsc_carveout
>>>> kerneldoc in remoteproc.h file.
>>>
>>> Thanks for pointing this comment. Indeed sMMU is not mandatory, and at
>> first sight I agree we should remove the restriction introduced by the patch.
>>> Driver porting on the series should be done before adding this.
>>>>
>>>> We do have platforms where we have some internal sub-modules within
>> the
>>>> remote processor sub-system that provides some linear
>>>> address-translation (most common case with 32-bit processors supporting
>>>> 64-bit addresses). Also, we have some upcoming SoCs where we have an
>>>> MMU
>>>> but is not programmable by Linux.
>>>>
>>>> There is one comment there, but I don't think this is actually handled
>>>> in the current remoteproc core.
>>>> "If @da is set to
>>>>  * FW_RSC_ADDR_ANY, then the host will dynamically allocate it, and
>> then
>>>>  * overwrite @da with the dynamically allocated address."
>>>>
>>> I don't remember it was implemented like described.
>>
>> Yes, it was missing, and one of your patches seem to add this behavior
>> now. That said, I really don't think the remoteproc core can dictate the
>>  da. Even if the individual remoteproc driver were to furnish this, how
>> would you get such data without forcing a fixed behavior for all
>> possible firmwares (not desirable). We should get rid of this comment,
>> and any code that seems to do this.
> 
> Agree that if some rules are fixed by firmware, rproc core can't overwrite them.
> It was the goal of the patch as without sMMU if you don't have a match between rproc core carveout allocation and firmware rsc request, rproc core can't answer the request.

Question is what are we going to use to check that - da or pa? With
fixed memory support, I would think that this would have to be based on
pa if specified. Given that the meaning of da can change from firmware
to firmware (in the case of some translators), but you can only reserve
a fixed reserved memory at kernel bootup time.

> In the rest of the series, da in rsc table is updated only in some specific cases (i.e. da is equal to FW_RSC_ADDR_ANY). If there is a force update I agree we should not allow it.
>>
>>>
>>> I have remarks about the comment:
>>> "* We will always use @da to negotiate the device addresses, even if it
>>>  * isn't using an iommu. In that case, though, it will obviously contain
>>>  * physical addresses."
>>>
>>> When there is no sMMU, we can't consider that da contains a physical
>> address because coprocessor can have its own memory map just because it
>> is a 32bit processor accessing only a part of the memory and the main is 64bit
>> one. The 2 processors won't see the internal memory at the same base
>> address for example.
>>
>> Agreed, believe it was valid when it was written (32-bit platforms
>> supporting 32-bit addresses). I think this is akin to an IPA
>> (Intermediate Physical Address).
>>
>>> So what should we do when carveout allocated by host is not fitting with
>> resource table request?
>>> - put a warning and overwrite da address in the resource table?
>>
>> Hmm, why da? This goes to my earlier comment about how you are able to
>> decide the da. Atleast your current ST driver seems to be assigning the
>> same value as the physical bus address for da, which would prompt why
>> you would still need a carveout entry in the resource table if it is
>> truly one-to-one.
>>
>> Eg, I have an upcoming usecase with R5Fs on newer TI SoCs where we
>> actually have a sub-module called Region Address Translator (RAT) which
>> can only be programmed by the R5F for translating the 32-bit CPU
>> addresses to larger physical address space, and yet I need the da and pa
>> to be able to do loading. I cannot dictate the da since that is what the
>> firmware images are linked against. So, have to rely on the firmware
>> providing this data for me.
>>
>>> - stop rproc probe as no match detected?
>>
>> I think that is the safest approach.
> 
> So we agree on the patch if a WARN is added and comment updated.
> Are we aligned ?

Did you mixup the comments from patch 2 here? This patch is still not ok
because of the implicit assumptions - we can have a valid da specified
even with no MMUs (because of some psuedo-MMU like sub-modules that can
provide the translation on the remote-processor side).

> 
>>
>>>
>>> Later in the series, carveout allocation is changed. Resource table carveout
>> are either linked with an existing carveout registered by driver or added to
>> carveout list for allocations.
>>> In the case you described, TI driver should first register the specific
>> carveout regions thank to the helper.
>>
>> The current series should still continue to work without having to
>> enforce new name assignments (unless needed and being defined to use
>> the
>> new features being added).
>>

> OK but if we implement above solution, if da is different from "dma" address (from dma_alloc_coherent) and no sMMU registered, rproc start procedure will failed.

That would be the case only when you have no translators of any kind in
between the CPU and the bus right within the remote processor subsystem
right and really da == dma (or pa)? In patch 2, you modified the rsc->pa
from dma to rproc_va_to_pa() to actually reflect the pa, yet here using
dma as the actual device address.

regards
Suman

> 
> Regards,
> Loic
>>>
>>> Regards,
>>> Loic
>>>
>>>> regards
>>>> Suman
>>>>
>>>>> ---
>>>>>  drivers/remoteproc/remoteproc_core.c | 10 +++++++++-
>>>>>  1 file changed, 9 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>>>> b/drivers/remoteproc/remoteproc_core.c
>>>>> index 4cd1a8e..437fabf 100644
>>>>> --- a/drivers/remoteproc/remoteproc_core.c
>>>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>>>> @@ -657,7 +657,15 @@ static int rproc_handle_carveout(struct rproc
>>>> *rproc,
>>>>>  	 * to use the iommu-based DMA API: we expect 'dma' to contain the
>>>>>  	 * physical address in this case.
>>>>>  	 */
>>>>> -	if (rproc->domain) {
>>>>> +
>>>>> +	if (rsc->da != FW_RSC_ADDR_ANY && !rproc->domain) {
>>
>> Maybe this should really be a reverse check when you have MMUs,
>> if (rproc->domain && rsc->da == FW_RSC_ADDR_ANY)
>>
>> regards
>> Suman
>>
>>>>> +		dev_err(dev->parent,
>>>>> +			"Bad carveout rsc configuration\n");
>>>>> +		ret = -ENOMEM;
>>>>> +		goto dma_free;
>>>>> +	}
>>>>> +
>>>>> +	if (rsc->da != FW_RSC_ADDR_ANY && rproc->domain) {
>>>>>  		mapping = kzalloc(sizeof(*mapping), GFP_KERNEL);
>>>>>  		if (!mapping) {
>>>>>  			ret = -ENOMEM;
>>>>>
>>>
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-10-24 15:14     ` Loic PALLARDY
@ 2018-10-29 20:17       ` Suman Anna
  2018-12-04 17:56         ` Wendy Liang
  0 siblings, 1 reply; 75+ messages in thread
From: Suman Anna @ 2018-10-29 20:17 UTC (permalink / raw)
  To: Loic PALLARDY, bjorn.andersson, ohad
  Cc: linux-remoteproc, linux-kernel, Arnaud POULIQUEN, benjamin.gaignard

Hi Loic,

On 10/24/18 10:14 AM, Loic PALLARDY wrote:
> Hi Suman,
> 
>> -----Original Message-----
>> From: Suman Anna <s-anna@ti.com>
>> Sent: mercredi 24 octobre 2018 02:14
>> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
>> ohad@wizery.com
>> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
>> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
>> benjamin.gaignard@linaro.org
>> Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on
>> centralized carveout allocator
>>
>> On 7/27/18 8:14 AM, Loic Pallardy wrote:
>>> Current version of rproc_alloc_vring function supports only dynamic vring
>>> allocation.
>>>
>>> This patch allows to allocate vrings based on memory region declatation.
>>> Vrings are now manage like memory carveouts, to communize memory
>> management
>>> code in rproc_alloc_registered_carveouts().
>>>
>>> Allocated buffer is retrieved in rp_find_vq() thanks to
>>> rproc_find_carveout_by_name() functions for.
>>>
>>> This patch sets vrings names to vdev"x"vring"y" with x vdev index in
>>> resource table and y vring index in vdev. This will be updated when
>>> name will be associated to vdev in firmware resource table.
>>>
>>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
>>> ---
>>>  drivers/remoteproc/remoteproc_core.c     | 61 +++++++++++++++++------
>> ---------
>>>  drivers/remoteproc/remoteproc_internal.h |  2 ++
>>>  drivers/remoteproc/remoteproc_virtio.c   | 14 +++++++-
>>>  include/linux/remoteproc.h               |  6 ++--
>>>  4 files changed, 51 insertions(+), 32 deletions(-)
>>>
>>> diff --git a/drivers/remoteproc/remoteproc_core.c
>> b/drivers/remoteproc/remoteproc_core.c
>>> index c543d04..4edc6f0 100644
>>> --- a/drivers/remoteproc/remoteproc_core.c
>>> +++ b/drivers/remoteproc/remoteproc_core.c
>>> @@ -53,6 +53,11 @@ typedef int (*rproc_handle_resources_t)(struct
>> rproc *rproc,
>>>  typedef int (*rproc_handle_resource_t)(struct rproc *rproc,
>>>  				 void *, int offset, int avail);
>>>
>>> +static int rproc_alloc_carveout(struct rproc *rproc,
>>> +				struct rproc_mem_entry *mem);
>>> +static int rproc_release_carveout(struct rproc *rproc,
>>> +				  struct rproc_mem_entry *mem);
>>> +
>>>  /* Unique indices for remoteproc devices */
>>>  static DEFINE_IDA(rproc_dev_index);
>>>
>>> @@ -312,21 +317,33 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev,
>> int i)
>>>  	struct device *dev = &rproc->dev;
>>>  	struct rproc_vring *rvring = &rvdev->vring[i];
>>>  	struct fw_rsc_vdev *rsc;
>>> -	dma_addr_t dma;
>>> -	void *va;
>>>  	int ret, size, notifyid;
>>> +	struct rproc_mem_entry *mem;
>>>
>>>  	/* actual size of vring (in bytes) */
>>>  	size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
>>>
>>> -	/*
>>> -	 * Allocate non-cacheable memory for the vring. In the future
>>> -	 * this call will also configure the IOMMU for us
>>> -	 */
>>> -	va = dma_alloc_coherent(dev->parent, size, &dma, GFP_KERNEL);
>>> -	if (!va) {
>>> -		dev_err(dev->parent, "dma_alloc_coherent failed\n");
>>> -		return -EINVAL;
>>> +	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
>>> +
>>> +	/* Search for pre-registered carveout */
>>> +	mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d",
>> rvdev->index,
>>> +					  i);
>>> +	if (mem) {
>>> +		if (rproc_check_carveout_da(rproc, mem, rsc->vring[i].da,
>> size))
>>> +			return -ENOMEM;
>>> +	} else {
>>> +		/* Register carveout in in list */
>>> +		mem = rproc_mem_entry_init(dev, 0, 0, size, rsc-
>>> vring[i].da,
>>> +					   rproc_alloc_carveout,
>>> +					   rproc_release_carveout,
>>> +					   "vdev%dvring%d",
>>> +					   rvdev->index, i);
>>> +		if (!mem) {
>>> +			dev_err(dev, "Can't allocate memory entry
>> structure\n");
>>> +			return -ENOMEM;
>>> +		}
>>> +
>>> +		rproc_add_carveout(rproc, mem);
>>>  	}
>>>
>>>  	/*
>>> @@ -337,7 +354,6 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
>>>  	ret = idr_alloc(&rproc->notifyids, rvring, 0, 0, GFP_KERNEL);
>>>  	if (ret < 0) {
>>>  		dev_err(dev, "idr_alloc failed: %d\n", ret);
>>> -		dma_free_coherent(dev->parent, size, va, dma);
>>>  		return ret;
>>>  	}
>>>  	notifyid = ret;
>>> @@ -346,21 +362,9 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int
>> i)
>>>  	if (notifyid > rproc->max_notifyid)
>>>  		rproc->max_notifyid = notifyid;
>>>
>>> -	dev_dbg(dev, "vring%d: va %pK dma %pad size 0x%x idr %d\n",
>>> -		i, va, &dma, size, notifyid);
>>> -
>>> -	rvring->va = va;
>>> -	rvring->dma = dma;
>>>  	rvring->notifyid = notifyid;
>>>
>>> -	/*
>>> -	 * Let the rproc know the notifyid and da of this vring.
>>> -	 * Not all platforms use dma_alloc_coherent to automatically
>>> -	 * set up the iommu. In this case the device address (da) will
>>> -	 * hold the physical address and not the device address.
>>> -	 */
>>> -	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
>>> -	rsc->vring[i].da = dma;
>>> +	/* Let the rproc know the notifyid of this vring.*/
>>>  	rsc->vring[i].notifyid = notifyid;
>>>  	return 0;
>>>  }
>>> @@ -392,12 +396,10 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev,
>> int i)
>>>
>>>  void rproc_free_vring(struct rproc_vring *rvring)
>>>  {
>>> -	int size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
>>>  	struct rproc *rproc = rvring->rvdev->rproc;
>>>  	int idx = rvring->rvdev->vring - rvring;
>>>  	struct fw_rsc_vdev *rsc;
>>>
>>> -	dma_free_coherent(rproc->dev.parent, size, rvring->va, rvring-
>>> dma);
>>>  	idr_remove(&rproc->notifyids, rvring->notifyid);
>>
>> Also, I am not sure if FW_RSC_ADDR_ANY semantics were enforced
>> previously on the vring da. It was simply overwritten irrespective of
>> the value. Now, I am running again into the "bad carveout rsc
>> configuration value" due to the iommu_domain_check if
>> !FW_RSC_ADDR_ANY.
> 

Just realized that I missed responding on this thread last week.

> When are you running into the "bad carveout rsc configuration value" ? 
> This patch is creating one carveout per vring to rely on generic carveout allocator.
> Then carveout is retrieved from carveout  list and vring resource table information updated.
> 
> If the da of the carveout was fixed in the resource table, it is normal you have this error.

Yes, and if the vring da value is FW_RSC_ADDR_ANY, then I don't run into
this particular error. It's just that the semantics of vring da is open
previously, and my above comment being 0 was being used as a reset value
as well.

> To solve that ST driver is registering one fixed carveout per vring (with the right name today)

Yeah, we still expect to allocate these dynamically, so there won't be
any registration needed.

> It is the same discussion as with Patch 1. 

Right, except that we have a da from RSC_CARVEOUT and a da from vring,
and the previous code had some slight differences between the two. The
vring da semantics were never set before (value was always being
overwritten, also it didn't have a pa field), whereas the remoteproc.h
documentation did mention about FW_RSC_ADDR_ANY (without any backing
implementation previously) for the RSC_CARVEOUT da, with the entry also
having a field for pa.

If we consider we can't change fixed coprocessor address requests, rproc
core should stop its execution.
> It is the responsibility of platform driver to register the right memory regions.
> That's what we discussed with you and Bill in OpenAMP forum.
> TI usecase was to have the same DSP firmware with the same resource table being able to run on any DSP.

Yeah, it only covers one of the usecases/platforms (Keystone 2 DSP
platforms). And this only worked on these platforms so far because we
only were using internal memories - so there were no RSC_CARVEOUT
entries with valid da. Our Davinci DSP is a single instance and we do
have a RSC_CARVEOUT there, which fails due to the same Patch 1 logic
here as well.

> In that case each DSP platform driver has to provide the right memory region configuration with the correct pa to da.
> 
>>
>> FWIW, the rproc_free_vring was actually using the value 0 when resetting.
>  
> It is no more needed as the carveout list is cleared at each stop and recreated at each start.
> Moreover resource table (and firmware) is reloaded at each coprocessor start.

Yes, agreed. This is about the semantics of vring da from before (no
enforcement to strict enforcement of FW_RSC_ADDR_ANY). The overwriting
on da field on vrings with the dma address is actually a mistake, which
we are trying to proliferate more now. It all comes down to the fact of
treating da as dma address when it is not going to be the case on all
remoteprocs.

regards
Suman

> 
> Regards,
> Loic
>>
>> regards
>> Suman
>>
>>>
>>>  	/* reset resource entry info */
>>> @@ -484,6 +486,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
>> struct fw_rsc_vdev *rsc,
>>>
>>>  	rvdev->id = rsc->id;
>>>  	rvdev->rproc = rproc;
>>> +	rvdev->index = rproc->nb_vdev++;
>>>
>>>  	/* parse the vrings */
>>>  	for (i = 0; i < rsc->num_of_vrings; i++) {
>>> @@ -528,9 +531,6 @@ void rproc_vdev_release(struct kref *ref)
>>>
>>>  	for (id = 0; id < ARRAY_SIZE(rvdev->vring); id++) {
>>>  		rvring = &rvdev->vring[id];
>>> -		if (!rvring->va)
>>> -			continue;
>>> -
>>>  		rproc_free_vring(rvring);
>>>  	}
>>>
>>> @@ -1322,6 +1322,9 @@ static int rproc_fw_boot(struct rproc *rproc,
>> const struct firmware *fw)
>>>  	/* reset max_notifyid */
>>>  	rproc->max_notifyid = -1;
>>>
>>> +	/* reset handled vdev */
>>> +	rproc->nb_vdev = 0;
>>> +
>>>  	/* handle fw resources which are required to boot rproc */
>>>  	ret = rproc_handle_resources(rproc, rproc_loading_handlers);
>>>  	if (ret) {
>>> diff --git a/drivers/remoteproc/remoteproc_internal.h
>> b/drivers/remoteproc/remoteproc_internal.h
>>> index 7570beb..f6cad24 100644
>>> --- a/drivers/remoteproc/remoteproc_internal.h
>>> +++ b/drivers/remoteproc/remoteproc_internal.h
>>> @@ -60,6 +60,8 @@ struct dentry *rproc_create_trace_file(const char
>> *name, struct rproc *rproc,
>>>  int rproc_elf_load_rsc_table(struct rproc *rproc, const struct firmware
>> *fw);
>>>  struct resource_table *rproc_elf_find_loaded_rsc_table(struct rproc
>> *rproc,
>>>  						       const struct firmware
>> *fw);
>>> +struct rproc_mem_entry *
>>> +rproc_find_carveout_by_name(struct rproc *rproc, const char *name,
>> ...);
>>>
>>>  static inline
>>>  int rproc_fw_sanity_check(struct rproc *rproc, const struct firmware *fw)
>>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
>> b/drivers/remoteproc/remoteproc_virtio.c
>>> index bbecd44..de21f62 100644
>>> --- a/drivers/remoteproc/remoteproc_virtio.c
>>> +++ b/drivers/remoteproc/remoteproc_virtio.c
>>> @@ -76,7 +76,9 @@ static struct virtqueue *rp_find_vq(struct
>> virtio_device *vdev,
>>>  	struct rproc_vdev *rvdev = vdev_to_rvdev(vdev);
>>>  	struct rproc *rproc = vdev_to_rproc(vdev);
>>>  	struct device *dev = &rproc->dev;
>>> +	struct rproc_mem_entry *mem;
>>>  	struct rproc_vring *rvring;
>>> +	struct fw_rsc_vdev *rsc;
>>>  	struct virtqueue *vq;
>>>  	void *addr;
>>>  	int len, size;
>>> @@ -88,8 +90,14 @@ static struct virtqueue *rp_find_vq(struct
>> virtio_device *vdev,
>>>  	if (!name)
>>>  		return NULL;
>>>
>>> +	/* Search allocated memory region by name */
>>> +	mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d",
>> rvdev->index,
>>> +					  id);
>>> +	if (!mem || !mem->va)
>>> +		return ERR_PTR(-ENOMEM);
>>> +
>>>  	rvring = &rvdev->vring[id];
>>> -	addr = rvring->va;
>>> +	addr = mem->va;
>>>  	len = rvring->len;
>>>
>>>  	/* zero vring */
>>> @@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct
>> virtio_device *vdev,
>>>  	rvring->vq = vq;
>>>  	vq->priv = rvring;
>>>
>>> +	/* Update vring in resource table */
>>> +	rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
>>> +	rsc->vring[id].da = mem->da;
>>> +
>>>  	return vq;
>>>  }
>>>
>>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
>>> index 4cdd0c6..6b3a234 100644
>>> --- a/include/linux/remoteproc.h
>>> +++ b/include/linux/remoteproc.h
>>> @@ -453,6 +453,7 @@ struct rproc_dump_segment {
>>>   * @table_sz: size of @cached_table
>>>   * @has_iommu: flag to indicate if remote processor is behind an MMU
>>>   * @dump_segments: list of segments in the firmware
>>> + * @nb_vdev: number of vdev currently handled by rproc
>>>   */
>>>  struct rproc {
>>>  	struct list_head node;
>>> @@ -485,6 +486,7 @@ struct rproc {
>>>  	bool has_iommu;
>>>  	bool auto_boot;
>>>  	struct list_head dump_segments;
>>> +	int nb_vdev;
>>>  };
>>>
>>>  /**
>>> @@ -512,7 +514,6 @@ struct rproc_subdev {
>>>  /**
>>>   * struct rproc_vring - remoteproc vring state
>>>   * @va:	virtual address
>>> - * @dma: dma address
>>>   * @len: length, in bytes
>>>   * @da: device address
>>>   * @align: vring alignment
>>> @@ -522,7 +523,6 @@ struct rproc_subdev {
>>>   */
>>>  struct rproc_vring {
>>>  	void *va;
>>> -	dma_addr_t dma;
>>>  	int len;
>>>  	u32 da;
>>>  	u32 align;
>>> @@ -541,6 +541,7 @@ struct rproc_vring {
>>>   * @vdev: the virio device
>>>   * @vring: the vrings for this vdev
>>>   * @rsc_offset: offset of the vdev's resource entry
>>> + * @index: vdev position versus other vdev declared in resource table
>>>   */
>>>  struct rproc_vdev {
>>>  	struct kref refcount;
>>> @@ -553,6 +554,7 @@ struct rproc_vdev {
>>>  	struct virtio_device vdev;
>>>  	struct rproc_vring vring[RVDEV_NUM_VRINGS];
>>>  	u32 rsc_offset;
>>> +	u32 index;
>>>  };
>>>
>>>  struct rproc *rproc_get_by_phandle(phandle phandle);
>>>
> 


^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-10-29 20:17       ` Suman Anna
@ 2018-12-04 17:56         ` Wendy Liang
  2018-12-04 18:04           ` Loic PALLARDY
  0 siblings, 1 reply; 75+ messages in thread
From: Wendy Liang @ 2018-12-04 17:56 UTC (permalink / raw)
  To: Suman Anna
  Cc: Loic Pallardy, Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, arnaud pouliquen, Benjamin Gaignard

On Mon, Oct 29, 2018 at 1:19 PM Suman Anna <s-anna@ti.com> wrote:
>
> Hi Loic,
>
> On 10/24/18 10:14 AM, Loic PALLARDY wrote:
> > Hi Suman,
> >
> >> -----Original Message-----
> >> From: Suman Anna <s-anna@ti.com>
> >> Sent: mercredi 24 octobre 2018 02:14
> >> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> >> ohad@wizery.com
> >> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> >> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> >> benjamin.gaignard@linaro.org
> >> Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on
> >> centralized carveout allocator
> >>
> >> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> >>> Current version of rproc_alloc_vring function supports only dynamic vring
> >>> allocation.
> >>>
> >>> This patch allows to allocate vrings based on memory region declatation.
> >>> Vrings are now manage like memory carveouts, to communize memory
> >> management
> >>> code in rproc_alloc_registered_carveouts().
> >>>
> >>> Allocated buffer is retrieved in rp_find_vq() thanks to
> >>> rproc_find_carveout_by_name() functions for.
> >>>
> >>> This patch sets vrings names to vdev"x"vring"y" with x vdev index in
> >>> resource table and y vring index in vdev. This will be updated when
> >>> name will be associated to vdev in firmware resource table.
> >>>
> >>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> >>> ---
> >>>  drivers/remoteproc/remoteproc_core.c     | 61 +++++++++++++++++------
> >> ---------
> >>>  drivers/remoteproc/remoteproc_internal.h |  2 ++
> >>>  drivers/remoteproc/remoteproc_virtio.c   | 14 +++++++-
> >>>  include/linux/remoteproc.h               |  6 ++--
> >>>  4 files changed, 51 insertions(+), 32 deletions(-)
> >>>
> >>> diff --git a/drivers/remoteproc/remoteproc_core.c
> >> b/drivers/remoteproc/remoteproc_core.c
> >>> index c543d04..4edc6f0 100644
> >>> --- a/drivers/remoteproc/remoteproc_core.c
> >>> +++ b/drivers/remoteproc/remoteproc_core.c
> >>> @@ -53,6 +53,11 @@ typedef int (*rproc_handle_resources_t)(struct
> >> rproc *rproc,
> >>>  typedef int (*rproc_handle_resource_t)(struct rproc *rproc,
> >>>                              void *, int offset, int avail);
> >>>
> >>> +static int rproc_alloc_carveout(struct rproc *rproc,
> >>> +                           struct rproc_mem_entry *mem);
> >>> +static int rproc_release_carveout(struct rproc *rproc,
> >>> +                             struct rproc_mem_entry *mem);
> >>> +
> >>>  /* Unique indices for remoteproc devices */
> >>>  static DEFINE_IDA(rproc_dev_index);
> >>>
> >>> @@ -312,21 +317,33 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev,
> >> int i)
> >>>     struct device *dev = &rproc->dev;
> >>>     struct rproc_vring *rvring = &rvdev->vring[i];
> >>>     struct fw_rsc_vdev *rsc;
> >>> -   dma_addr_t dma;
> >>> -   void *va;
> >>>     int ret, size, notifyid;
> >>> +   struct rproc_mem_entry *mem;
> >>>
> >>>     /* actual size of vring (in bytes) */
> >>>     size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
> >>>
> >>> -   /*
> >>> -    * Allocate non-cacheable memory for the vring. In the future
> >>> -    * this call will also configure the IOMMU for us
> >>> -    */
> >>> -   va = dma_alloc_coherent(dev->parent, size, &dma, GFP_KERNEL);
> >>> -   if (!va) {
> >>> -           dev_err(dev->parent, "dma_alloc_coherent failed\n");
> >>> -           return -EINVAL;
> >>> +   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> >>> +
> >>> +   /* Search for pre-registered carveout */
> >>> +   mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d",
> >> rvdev->index,
> >>> +                                     i);
> >>> +   if (mem) {
> >>> +           if (rproc_check_carveout_da(rproc, mem, rsc->vring[i].da,
> >> size))
> >>> +                   return -ENOMEM;
> >>> +   } else {
> >>> +           /* Register carveout in in list */
> >>> +           mem = rproc_mem_entry_init(dev, 0, 0, size, rsc-
> >>> vring[i].da,
> >>> +                                      rproc_alloc_carveout,
> >>> +                                      rproc_release_carveout,
> >>> +                                      "vdev%dvring%d",
> >>> +                                      rvdev->index, i);
> >>> +           if (!mem) {
> >>> +                   dev_err(dev, "Can't allocate memory entry
> >> structure\n");
> >>> +                   return -ENOMEM;
> >>> +           }
> >>> +
> >>> +           rproc_add_carveout(rproc, mem);
> >>>     }
> >>>
> >>>     /*
> >>> @@ -337,7 +354,6 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int i)
> >>>     ret = idr_alloc(&rproc->notifyids, rvring, 0, 0, GFP_KERNEL);
> >>>     if (ret < 0) {
> >>>             dev_err(dev, "idr_alloc failed: %d\n", ret);
> >>> -           dma_free_coherent(dev->parent, size, va, dma);
> >>>             return ret;
> >>>     }
> >>>     notifyid = ret;
> >>> @@ -346,21 +362,9 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev, int
> >> i)
> >>>     if (notifyid > rproc->max_notifyid)
> >>>             rproc->max_notifyid = notifyid;
> >>>
> >>> -   dev_dbg(dev, "vring%d: va %pK dma %pad size 0x%x idr %d\n",
> >>> -           i, va, &dma, size, notifyid);
> >>> -
> >>> -   rvring->va = va;
> >>> -   rvring->dma = dma;
> >>>     rvring->notifyid = notifyid;
> >>>
> >>> -   /*
> >>> -    * Let the rproc know the notifyid and da of this vring.
> >>> -    * Not all platforms use dma_alloc_coherent to automatically
> >>> -    * set up the iommu. In this case the device address (da) will
> >>> -    * hold the physical address and not the device address.
> >>> -    */
> >>> -   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> >>> -   rsc->vring[i].da = dma;
> >>> +   /* Let the rproc know the notifyid of this vring.*/
> >>>     rsc->vring[i].notifyid = notifyid;
> >>>     return 0;
> >>>  }
> >>> @@ -392,12 +396,10 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev,
> >> int i)
> >>>
> >>>  void rproc_free_vring(struct rproc_vring *rvring)
> >>>  {
> >>> -   int size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
> >>>     struct rproc *rproc = rvring->rvdev->rproc;
> >>>     int idx = rvring->rvdev->vring - rvring;
> >>>     struct fw_rsc_vdev *rsc;
> >>>
> >>> -   dma_free_coherent(rproc->dev.parent, size, rvring->va, rvring-
> >>> dma);
> >>>     idr_remove(&rproc->notifyids, rvring->notifyid);
> >>
> >> Also, I am not sure if FW_RSC_ADDR_ANY semantics were enforced
> >> previously on the vring da. It was simply overwritten irrespective of
> >> the value. Now, I am running again into the "bad carveout rsc
> >> configuration value" due to the iommu_domain_check if
> >> !FW_RSC_ADDR_ANY.
> >
>
> Just realized that I missed responding on this thread last week.
>
> > When are you running into the "bad carveout rsc configuration value" ?
> > This patch is creating one carveout per vring to rely on generic carveout allocator.
> > Then carveout is retrieved from carveout  list and vring resource table information updated.
> >
> > If the da of the carveout was fixed in the resource table, it is normal you have this error.
>
> Yes, and if the vring da value is FW_RSC_ADDR_ANY, then I don't run into
> this particular error. It's just that the semantics of vring da is open
> previously, and my above comment being 0 was being used as a reset value
> as well.
>
> > To solve that ST driver is registering one fixed carveout per vring (with the right name today)
>
> Yeah, we still expect to allocate these dynamically, so there won't be
> any registration needed.
>
> > It is the same discussion as with Patch 1.
>
> Right, except that we have a da from RSC_CARVEOUT and a da from vring,
> and the previous code had some slight differences between the two. The
> vring da semantics were never set before (value was always being
> overwritten, also it didn't have a pa field), whereas the remoteproc.h
> documentation did mention about FW_RSC_ADDR_ANY (without any backing
> implementation previously) for the RSC_CARVEOUT da, with the entry also
> having a field for pa.
>
> If we consider we can't change fixed coprocessor address requests, rproc
> core should stop its execution.
> > It is the responsibility of platform driver to register the right memory regions.
> > That's what we discussed with you and Bill in OpenAMP forum.
> > TI usecase was to have the same DSP firmware with the same resource table being able to run on any DSP.
>
> Yeah, it only covers one of the usecases/platforms (Keystone 2 DSP
> platforms). And this only worked on these platforms so far because we
> only were using internal memories - so there were no RSC_CARVEOUT
> entries with valid da. Our Davinci DSP is a single instance and we do
> have a RSC_CARVEOUT there, which fails due to the same Patch 1 logic
> here as well.
>
> > In that case each DSP platform driver has to provide the right memory region configuration with the correct pa to da.
> >
> >>
> >> FWIW, the rproc_free_vring was actually using the value 0 when resetting.
> >
> > It is no more needed as the carveout list is cleared at each stop and recreated at each start.
> > Moreover resource table (and firmware) is reloaded at each coprocessor start.
>
> Yes, agreed. This is about the semantics of vring da from before (no
> enforcement to strict enforcement of FW_RSC_ADDR_ANY). The overwriting
> on da field on vrings with the dma address is actually a mistake, which
> we are trying to proliferate more now. It all comes down to the fact of
> treating da as dma address when it is not going to be the case on all
> remoteprocs.
[Wendy] Are we assuming that the vring da is always predefined.
But in the Linux kernel side, without IOMMU case, we use dma_alloc_coherent() to
allocate for vring address. In this case, there is no gurantee the
allocated vrings
address matches the predefined value.
Or we assume that only vdev devices of the remoteproc can use
dma_alloc_coherent()
but not subdevices?
Or we can still store the rsc table da pointer. And instead of always
set da with dma address
we can pass the dma address to remoteproc and let remoteproc to do the
pa to da conversion if
required.

Thanks,
Wendy
>
> regards
> Suman
>
> >
> > Regards,
> > Loic
> >>
> >> regards
> >> Suman
> >>
> >>>
> >>>     /* reset resource entry info */
> >>> @@ -484,6 +486,7 @@ static int rproc_handle_vdev(struct rproc *rproc,
> >> struct fw_rsc_vdev *rsc,
> >>>
> >>>     rvdev->id = rsc->id;
> >>>     rvdev->rproc = rproc;
> >>> +   rvdev->index = rproc->nb_vdev++;
> >>>
> >>>     /* parse the vrings */
> >>>     for (i = 0; i < rsc->num_of_vrings; i++) {
> >>> @@ -528,9 +531,6 @@ void rproc_vdev_release(struct kref *ref)
> >>>
> >>>     for (id = 0; id < ARRAY_SIZE(rvdev->vring); id++) {
> >>>             rvring = &rvdev->vring[id];
> >>> -           if (!rvring->va)
> >>> -                   continue;
> >>> -
> >>>             rproc_free_vring(rvring);
> >>>     }
> >>>
> >>> @@ -1322,6 +1322,9 @@ static int rproc_fw_boot(struct rproc *rproc,
> >> const struct firmware *fw)
> >>>     /* reset max_notifyid */
> >>>     rproc->max_notifyid = -1;
> >>>
> >>> +   /* reset handled vdev */
> >>> +   rproc->nb_vdev = 0;
> >>> +
> >>>     /* handle fw resources which are required to boot rproc */
> >>>     ret = rproc_handle_resources(rproc, rproc_loading_handlers);
> >>>     if (ret) {
> >>> diff --git a/drivers/remoteproc/remoteproc_internal.h
> >> b/drivers/remoteproc/remoteproc_internal.h
> >>> index 7570beb..f6cad24 100644
> >>> --- a/drivers/remoteproc/remoteproc_internal.h
> >>> +++ b/drivers/remoteproc/remoteproc_internal.h
> >>> @@ -60,6 +60,8 @@ struct dentry *rproc_create_trace_file(const char
> >> *name, struct rproc *rproc,
> >>>  int rproc_elf_load_rsc_table(struct rproc *rproc, const struct firmware
> >> *fw);
> >>>  struct resource_table *rproc_elf_find_loaded_rsc_table(struct rproc
> >> *rproc,
> >>>                                                    const struct firmware
> >> *fw);
> >>> +struct rproc_mem_entry *
> >>> +rproc_find_carveout_by_name(struct rproc *rproc, const char *name,
> >> ...);
> >>>
> >>>  static inline
> >>>  int rproc_fw_sanity_check(struct rproc *rproc, const struct firmware *fw)
> >>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
> >> b/drivers/remoteproc/remoteproc_virtio.c
> >>> index bbecd44..de21f62 100644
> >>> --- a/drivers/remoteproc/remoteproc_virtio.c
> >>> +++ b/drivers/remoteproc/remoteproc_virtio.c
> >>> @@ -76,7 +76,9 @@ static struct virtqueue *rp_find_vq(struct
> >> virtio_device *vdev,
> >>>     struct rproc_vdev *rvdev = vdev_to_rvdev(vdev);
> >>>     struct rproc *rproc = vdev_to_rproc(vdev);
> >>>     struct device *dev = &rproc->dev;
> >>> +   struct rproc_mem_entry *mem;
> >>>     struct rproc_vring *rvring;
> >>> +   struct fw_rsc_vdev *rsc;
> >>>     struct virtqueue *vq;
> >>>     void *addr;
> >>>     int len, size;
> >>> @@ -88,8 +90,14 @@ static struct virtqueue *rp_find_vq(struct
> >> virtio_device *vdev,
> >>>     if (!name)
> >>>             return NULL;
> >>>
> >>> +   /* Search allocated memory region by name */
> >>> +   mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d",
> >> rvdev->index,
> >>> +                                     id);
> >>> +   if (!mem || !mem->va)
> >>> +           return ERR_PTR(-ENOMEM);
> >>> +
> >>>     rvring = &rvdev->vring[id];
> >>> -   addr = rvring->va;
> >>> +   addr = mem->va;
> >>>     len = rvring->len;
> >>>
> >>>     /* zero vring */
> >>> @@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct
> >> virtio_device *vdev,
> >>>     rvring->vq = vq;
> >>>     vq->priv = rvring;
> >>>
> >>> +   /* Update vring in resource table */
> >>> +   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> >>> +   rsc->vring[id].da = mem->da;
> >>> +
> >>>     return vq;
> >>>  }
> >>>
> >>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> >>> index 4cdd0c6..6b3a234 100644
> >>> --- a/include/linux/remoteproc.h
> >>> +++ b/include/linux/remoteproc.h
> >>> @@ -453,6 +453,7 @@ struct rproc_dump_segment {
> >>>   * @table_sz: size of @cached_table
> >>>   * @has_iommu: flag to indicate if remote processor is behind an MMU
> >>>   * @dump_segments: list of segments in the firmware
> >>> + * @nb_vdev: number of vdev currently handled by rproc
> >>>   */
> >>>  struct rproc {
> >>>     struct list_head node;
> >>> @@ -485,6 +486,7 @@ struct rproc {
> >>>     bool has_iommu;
> >>>     bool auto_boot;
> >>>     struct list_head dump_segments;
> >>> +   int nb_vdev;
> >>>  };
> >>>
> >>>  /**
> >>> @@ -512,7 +514,6 @@ struct rproc_subdev {
> >>>  /**
> >>>   * struct rproc_vring - remoteproc vring state
> >>>   * @va:    virtual address
> >>> - * @dma: dma address
> >>>   * @len: length, in bytes
> >>>   * @da: device address
> >>>   * @align: vring alignment
> >>> @@ -522,7 +523,6 @@ struct rproc_subdev {
> >>>   */
> >>>  struct rproc_vring {
> >>>     void *va;
> >>> -   dma_addr_t dma;
> >>>     int len;
> >>>     u32 da;
> >>>     u32 align;
> >>> @@ -541,6 +541,7 @@ struct rproc_vring {
> >>>   * @vdev: the virio device
> >>>   * @vring: the vrings for this vdev
> >>>   * @rsc_offset: offset of the vdev's resource entry
> >>> + * @index: vdev position versus other vdev declared in resource table
> >>>   */
> >>>  struct rproc_vdev {
> >>>     struct kref refcount;
> >>> @@ -553,6 +554,7 @@ struct rproc_vdev {
> >>>     struct virtio_device vdev;
> >>>     struct rproc_vring vring[RVDEV_NUM_VRINGS];
> >>>     u32 rsc_offset;
> >>> +   u32 index;
> >>>  };
> >>>
> >>>  struct rproc *rproc_get_by_phandle(phandle phandle);
> >>>
> >
>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-12-04 17:56         ` Wendy Liang
@ 2018-12-04 18:04           ` Loic PALLARDY
  2018-12-04 18:58             ` Wendy Liang
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-12-04 18:04 UTC (permalink / raw)
  To: Wendy Liang, Suman Anna
  Cc: Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, Arnaud POULIQUEN, Benjamin Gaignard



> -----Original Message-----
> From: Wendy Liang <sunnyliangjy@gmail.com>
> Sent: mardi 4 décembre 2018 18:57
> To: Suman Anna <s-anna@ti.com>
> Cc: Loic PALLARDY <loic.pallardy@st.com>; Bjorn Andersson
> <bjorn.andersson@linaro.org>; Ohad Ben-Cohen <ohad@wizery.com>;
> linux-remoteproc@vger.kernel.org; Linux Kernel Mailing List <linux-
> kernel@vger.kernel.org>; Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> Benjamin Gaignard <benjamin.gaignard@linaro.org>
> Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on
> centralized carveout allocator
> 
> On Mon, Oct 29, 2018 at 1:19 PM Suman Anna <s-anna@ti.com> wrote:
> >
> > Hi Loic,
> >
> > On 10/24/18 10:14 AM, Loic PALLARDY wrote:
> > > Hi Suman,
> > >
> > >> -----Original Message-----
> > >> From: Suman Anna <s-anna@ti.com>
> > >> Sent: mercredi 24 octobre 2018 02:14
> > >> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> > >> ohad@wizery.com
> > >> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> > >> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> > >> benjamin.gaignard@linaro.org
> > >> Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to
> rely on
> > >> centralized carveout allocator
> > >>
> > >> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> > >>> Current version of rproc_alloc_vring function supports only dynamic
> vring
> > >>> allocation.
> > >>>
> > >>> This patch allows to allocate vrings based on memory region
> declatation.
> > >>> Vrings are now manage like memory carveouts, to communize
> memory
> > >> management
> > >>> code in rproc_alloc_registered_carveouts().
> > >>>
> > >>> Allocated buffer is retrieved in rp_find_vq() thanks to
> > >>> rproc_find_carveout_by_name() functions for.
> > >>>
> > >>> This patch sets vrings names to vdev"x"vring"y" with x vdev index in
> > >>> resource table and y vring index in vdev. This will be updated when
> > >>> name will be associated to vdev in firmware resource table.
> > >>>
> > >>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > >>> ---
> > >>>  drivers/remoteproc/remoteproc_core.c     | 61 +++++++++++++++++-
> -----
> > >> ---------
> > >>>  drivers/remoteproc/remoteproc_internal.h |  2 ++
> > >>>  drivers/remoteproc/remoteproc_virtio.c   | 14 +++++++-
> > >>>  include/linux/remoteproc.h               |  6 ++--
> > >>>  4 files changed, 51 insertions(+), 32 deletions(-)
> > >>>
> > >>> diff --git a/drivers/remoteproc/remoteproc_core.c
> > >> b/drivers/remoteproc/remoteproc_core.c
> > >>> index c543d04..4edc6f0 100644
> > >>> --- a/drivers/remoteproc/remoteproc_core.c
> > >>> +++ b/drivers/remoteproc/remoteproc_core.c
> > >>> @@ -53,6 +53,11 @@ typedef int (*rproc_handle_resources_t)(struct
> > >> rproc *rproc,
> > >>>  typedef int (*rproc_handle_resource_t)(struct rproc *rproc,
> > >>>                              void *, int offset, int avail);
> > >>>
> > >>> +static int rproc_alloc_carveout(struct rproc *rproc,
> > >>> +                           struct rproc_mem_entry *mem);
> > >>> +static int rproc_release_carveout(struct rproc *rproc,
> > >>> +                             struct rproc_mem_entry *mem);
> > >>> +
> > >>>  /* Unique indices for remoteproc devices */
> > >>>  static DEFINE_IDA(rproc_dev_index);
> > >>>
> > >>> @@ -312,21 +317,33 @@ int rproc_alloc_vring(struct rproc_vdev
> *rvdev,
> > >> int i)
> > >>>     struct device *dev = &rproc->dev;
> > >>>     struct rproc_vring *rvring = &rvdev->vring[i];
> > >>>     struct fw_rsc_vdev *rsc;
> > >>> -   dma_addr_t dma;
> > >>> -   void *va;
> > >>>     int ret, size, notifyid;
> > >>> +   struct rproc_mem_entry *mem;
> > >>>
> > >>>     /* actual size of vring (in bytes) */
> > >>>     size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
> > >>>
> > >>> -   /*
> > >>> -    * Allocate non-cacheable memory for the vring. In the future
> > >>> -    * this call will also configure the IOMMU for us
> > >>> -    */
> > >>> -   va = dma_alloc_coherent(dev->parent, size, &dma, GFP_KERNEL);
> > >>> -   if (!va) {
> > >>> -           dev_err(dev->parent, "dma_alloc_coherent failed\n");
> > >>> -           return -EINVAL;
> > >>> +   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > >>> +
> > >>> +   /* Search for pre-registered carveout */
> > >>> +   mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d",
> > >> rvdev->index,
> > >>> +                                     i);
> > >>> +   if (mem) {
> > >>> +           if (rproc_check_carveout_da(rproc, mem, rsc->vring[i].da,
> > >> size))
> > >>> +                   return -ENOMEM;
> > >>> +   } else {
> > >>> +           /* Register carveout in in list */
> > >>> +           mem = rproc_mem_entry_init(dev, 0, 0, size, rsc-
> > >>> vring[i].da,
> > >>> +                                      rproc_alloc_carveout,
> > >>> +                                      rproc_release_carveout,
> > >>> +                                      "vdev%dvring%d",
> > >>> +                                      rvdev->index, i);
> > >>> +           if (!mem) {
> > >>> +                   dev_err(dev, "Can't allocate memory entry
> > >> structure\n");
> > >>> +                   return -ENOMEM;
> > >>> +           }
> > >>> +
> > >>> +           rproc_add_carveout(rproc, mem);
> > >>>     }
> > >>>
> > >>>     /*
> > >>> @@ -337,7 +354,6 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev,
> int i)
> > >>>     ret = idr_alloc(&rproc->notifyids, rvring, 0, 0, GFP_KERNEL);
> > >>>     if (ret < 0) {
> > >>>             dev_err(dev, "idr_alloc failed: %d\n", ret);
> > >>> -           dma_free_coherent(dev->parent, size, va, dma);
> > >>>             return ret;
> > >>>     }
> > >>>     notifyid = ret;
> > >>> @@ -346,21 +362,9 @@ int rproc_alloc_vring(struct rproc_vdev
> *rvdev, int
> > >> i)
> > >>>     if (notifyid > rproc->max_notifyid)
> > >>>             rproc->max_notifyid = notifyid;
> > >>>
> > >>> -   dev_dbg(dev, "vring%d: va %pK dma %pad size 0x%x idr %d\n",
> > >>> -           i, va, &dma, size, notifyid);
> > >>> -
> > >>> -   rvring->va = va;
> > >>> -   rvring->dma = dma;
> > >>>     rvring->notifyid = notifyid;
> > >>>
> > >>> -   /*
> > >>> -    * Let the rproc know the notifyid and da of this vring.
> > >>> -    * Not all platforms use dma_alloc_coherent to automatically
> > >>> -    * set up the iommu. In this case the device address (da) will
> > >>> -    * hold the physical address and not the device address.
> > >>> -    */
> > >>> -   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > >>> -   rsc->vring[i].da = dma;
> > >>> +   /* Let the rproc know the notifyid of this vring.*/
> > >>>     rsc->vring[i].notifyid = notifyid;
> > >>>     return 0;
> > >>>  }
> > >>> @@ -392,12 +396,10 @@ int rproc_alloc_vring(struct rproc_vdev
> *rvdev,
> > >> int i)
> > >>>
> > >>>  void rproc_free_vring(struct rproc_vring *rvring)
> > >>>  {
> > >>> -   int size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
> > >>>     struct rproc *rproc = rvring->rvdev->rproc;
> > >>>     int idx = rvring->rvdev->vring - rvring;
> > >>>     struct fw_rsc_vdev *rsc;
> > >>>
> > >>> -   dma_free_coherent(rproc->dev.parent, size, rvring->va, rvring-
> > >>> dma);
> > >>>     idr_remove(&rproc->notifyids, rvring->notifyid);
> > >>
> > >> Also, I am not sure if FW_RSC_ADDR_ANY semantics were enforced
> > >> previously on the vring da. It was simply overwritten irrespective of
> > >> the value. Now, I am running again into the "bad carveout rsc
> > >> configuration value" due to the iommu_domain_check if
> > >> !FW_RSC_ADDR_ANY.
> > >
> >
> > Just realized that I missed responding on this thread last week.
> >
> > > When are you running into the "bad carveout rsc configuration value" ?
> > > This patch is creating one carveout per vring to rely on generic carveout
> allocator.
> > > Then carveout is retrieved from carveout  list and vring resource table
> information updated.
> > >
> > > If the da of the carveout was fixed in the resource table, it is normal you
> have this error.
> >
> > Yes, and if the vring da value is FW_RSC_ADDR_ANY, then I don't run into
> > this particular error. It's just that the semantics of vring da is open
> > previously, and my above comment being 0 was being used as a reset value
> > as well.
> >
> > > To solve that ST driver is registering one fixed carveout per vring (with the
> right name today)
> >
> > Yeah, we still expect to allocate these dynamically, so there won't be
> > any registration needed.
> >
> > > It is the same discussion as with Patch 1.
> >
> > Right, except that we have a da from RSC_CARVEOUT and a da from vring,
> > and the previous code had some slight differences between the two. The
> > vring da semantics were never set before (value was always being
> > overwritten, also it didn't have a pa field), whereas the remoteproc.h
> > documentation did mention about FW_RSC_ADDR_ANY (without any
> backing
> > implementation previously) for the RSC_CARVEOUT da, with the entry also
> > having a field for pa.
> >
> > If we consider we can't change fixed coprocessor address requests, rproc
> > core should stop its execution.
> > > It is the responsibility of platform driver to register the right memory
> regions.
> > > That's what we discussed with you and Bill in OpenAMP forum.
> > > TI usecase was to have the same DSP firmware with the same resource
> table being able to run on any DSP.
> >
> > Yeah, it only covers one of the usecases/platforms (Keystone 2 DSP
> > platforms). And this only worked on these platforms so far because we
> > only were using internal memories - so there were no RSC_CARVEOUT
> > entries with valid da. Our Davinci DSP is a single instance and we do
> > have a RSC_CARVEOUT there, which fails due to the same Patch 1 logic
> > here as well.
> >
> > > In that case each DSP platform driver has to provide the right memory
> region configuration with the correct pa to da.
> > >
> > >>
> > >> FWIW, the rproc_free_vring was actually using the value 0 when
> resetting.
> > >
> > > It is no more needed as the carveout list is cleared at each stop and
> recreated at each start.
> > > Moreover resource table (and firmware) is reloaded at each coprocessor
> start.
> >
> > Yes, agreed. This is about the semantics of vring da from before (no
> > enforcement to strict enforcement of FW_RSC_ADDR_ANY). The
> overwriting
> > on da field on vrings with the dma address is actually a mistake, which
> > we are trying to proliferate more now. It all comes down to the fact of
> > treating da as dma address when it is not going to be the case on all
> > remoteprocs.
> [Wendy] Are we assuming that the vring da is always predefined.
> But in the Linux kernel side, without IOMMU case, we use
> dma_alloc_coherent() to
> allocate for vring address. In this case, there is no gurantee the
> allocated vrings
> address matches the predefined value.
> Or we assume that only vdev devices of the remoteproc can use
> dma_alloc_coherent()
> but not subdevices?
> Or we can still store the rsc table da pointer. And instead of always
> set da with dma address
> we can pass the dma address to remoteproc and let remoteproc to do the
> pa to da conversion if
> required.

Hi Wendy,
I pushed a correction patch which check dma address with requested da from resource table.
If no match, a warning is display, but the sequence is not stopped as requested by Suman.
I have a next patch which is adding a rproc_pa_to_da() platform dependent function to concert a pa in da understandable by rproc firmware.
The goal is to answer Suman request, that some rproc can configure an offset in HW to access part of the DRR or an memory.
On another side, I'm looking how to do the same with the carveout registration functions...

Regards,
Loic


> 
> Thanks,
> Wendy
> >
> > regards
> > Suman
> >
> > >
> > > Regards,
> > > Loic
> > >>
> > >> regards
> > >> Suman
> > >>
> > >>>
> > >>>     /* reset resource entry info */
> > >>> @@ -484,6 +486,7 @@ static int rproc_handle_vdev(struct rproc
> *rproc,
> > >> struct fw_rsc_vdev *rsc,
> > >>>
> > >>>     rvdev->id = rsc->id;
> > >>>     rvdev->rproc = rproc;
> > >>> +   rvdev->index = rproc->nb_vdev++;
> > >>>
> > >>>     /* parse the vrings */
> > >>>     for (i = 0; i < rsc->num_of_vrings; i++) {
> > >>> @@ -528,9 +531,6 @@ void rproc_vdev_release(struct kref *ref)
> > >>>
> > >>>     for (id = 0; id < ARRAY_SIZE(rvdev->vring); id++) {
> > >>>             rvring = &rvdev->vring[id];
> > >>> -           if (!rvring->va)
> > >>> -                   continue;
> > >>> -
> > >>>             rproc_free_vring(rvring);
> > >>>     }
> > >>>
> > >>> @@ -1322,6 +1322,9 @@ static int rproc_fw_boot(struct rproc *rproc,
> > >> const struct firmware *fw)
> > >>>     /* reset max_notifyid */
> > >>>     rproc->max_notifyid = -1;
> > >>>
> > >>> +   /* reset handled vdev */
> > >>> +   rproc->nb_vdev = 0;
> > >>> +
> > >>>     /* handle fw resources which are required to boot rproc */
> > >>>     ret = rproc_handle_resources(rproc, rproc_loading_handlers);
> > >>>     if (ret) {
> > >>> diff --git a/drivers/remoteproc/remoteproc_internal.h
> > >> b/drivers/remoteproc/remoteproc_internal.h
> > >>> index 7570beb..f6cad24 100644
> > >>> --- a/drivers/remoteproc/remoteproc_internal.h
> > >>> +++ b/drivers/remoteproc/remoteproc_internal.h
> > >>> @@ -60,6 +60,8 @@ struct dentry *rproc_create_trace_file(const char
> > >> *name, struct rproc *rproc,
> > >>>  int rproc_elf_load_rsc_table(struct rproc *rproc, const struct firmware
> > >> *fw);
> > >>>  struct resource_table *rproc_elf_find_loaded_rsc_table(struct rproc
> > >> *rproc,
> > >>>                                                    const struct firmware
> > >> *fw);
> > >>> +struct rproc_mem_entry *
> > >>> +rproc_find_carveout_by_name(struct rproc *rproc, const char
> *name,
> > >> ...);
> > >>>
> > >>>  static inline
> > >>>  int rproc_fw_sanity_check(struct rproc *rproc, const struct firmware
> *fw)
> > >>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
> > >> b/drivers/remoteproc/remoteproc_virtio.c
> > >>> index bbecd44..de21f62 100644
> > >>> --- a/drivers/remoteproc/remoteproc_virtio.c
> > >>> +++ b/drivers/remoteproc/remoteproc_virtio.c
> > >>> @@ -76,7 +76,9 @@ static struct virtqueue *rp_find_vq(struct
> > >> virtio_device *vdev,
> > >>>     struct rproc_vdev *rvdev = vdev_to_rvdev(vdev);
> > >>>     struct rproc *rproc = vdev_to_rproc(vdev);
> > >>>     struct device *dev = &rproc->dev;
> > >>> +   struct rproc_mem_entry *mem;
> > >>>     struct rproc_vring *rvring;
> > >>> +   struct fw_rsc_vdev *rsc;
> > >>>     struct virtqueue *vq;
> > >>>     void *addr;
> > >>>     int len, size;
> > >>> @@ -88,8 +90,14 @@ static struct virtqueue *rp_find_vq(struct
> > >> virtio_device *vdev,
> > >>>     if (!name)
> > >>>             return NULL;
> > >>>
> > >>> +   /* Search allocated memory region by name */
> > >>> +   mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d",
> > >> rvdev->index,
> > >>> +                                     id);
> > >>> +   if (!mem || !mem->va)
> > >>> +           return ERR_PTR(-ENOMEM);
> > >>> +
> > >>>     rvring = &rvdev->vring[id];
> > >>> -   addr = rvring->va;
> > >>> +   addr = mem->va;
> > >>>     len = rvring->len;
> > >>>
> > >>>     /* zero vring */
> > >>> @@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct
> > >> virtio_device *vdev,
> > >>>     rvring->vq = vq;
> > >>>     vq->priv = rvring;
> > >>>
> > >>> +   /* Update vring in resource table */
> > >>> +   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > >>> +   rsc->vring[id].da = mem->da;
> > >>> +
> > >>>     return vq;
> > >>>  }
> > >>>
> > >>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> > >>> index 4cdd0c6..6b3a234 100644
> > >>> --- a/include/linux/remoteproc.h
> > >>> +++ b/include/linux/remoteproc.h
> > >>> @@ -453,6 +453,7 @@ struct rproc_dump_segment {
> > >>>   * @table_sz: size of @cached_table
> > >>>   * @has_iommu: flag to indicate if remote processor is behind an
> MMU
> > >>>   * @dump_segments: list of segments in the firmware
> > >>> + * @nb_vdev: number of vdev currently handled by rproc
> > >>>   */
> > >>>  struct rproc {
> > >>>     struct list_head node;
> > >>> @@ -485,6 +486,7 @@ struct rproc {
> > >>>     bool has_iommu;
> > >>>     bool auto_boot;
> > >>>     struct list_head dump_segments;
> > >>> +   int nb_vdev;
> > >>>  };
> > >>>
> > >>>  /**
> > >>> @@ -512,7 +514,6 @@ struct rproc_subdev {
> > >>>  /**
> > >>>   * struct rproc_vring - remoteproc vring state
> > >>>   * @va:    virtual address
> > >>> - * @dma: dma address
> > >>>   * @len: length, in bytes
> > >>>   * @da: device address
> > >>>   * @align: vring alignment
> > >>> @@ -522,7 +523,6 @@ struct rproc_subdev {
> > >>>   */
> > >>>  struct rproc_vring {
> > >>>     void *va;
> > >>> -   dma_addr_t dma;
> > >>>     int len;
> > >>>     u32 da;
> > >>>     u32 align;
> > >>> @@ -541,6 +541,7 @@ struct rproc_vring {
> > >>>   * @vdev: the virio device
> > >>>   * @vring: the vrings for this vdev
> > >>>   * @rsc_offset: offset of the vdev's resource entry
> > >>> + * @index: vdev position versus other vdev declared in resource table
> > >>>   */
> > >>>  struct rproc_vdev {
> > >>>     struct kref refcount;
> > >>> @@ -553,6 +554,7 @@ struct rproc_vdev {
> > >>>     struct virtio_device vdev;
> > >>>     struct rproc_vring vring[RVDEV_NUM_VRINGS];
> > >>>     u32 rsc_offset;
> > >>> +   u32 index;
> > >>>  };
> > >>>
> > >>>  struct rproc *rproc_get_by_phandle(phandle phandle);
> > >>>
> > >
> >

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-12-04 18:04           ` Loic PALLARDY
@ 2018-12-04 18:58             ` Wendy Liang
  2018-12-04 19:57               ` Loic PALLARDY
  0 siblings, 1 reply; 75+ messages in thread
From: Wendy Liang @ 2018-12-04 18:58 UTC (permalink / raw)
  To: Loic Pallardy
  Cc: Suman Anna, Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, arnaud pouliquen, Benjamin Gaignard

On Tue, Dec 4, 2018 at 10:04 AM Loic PALLARDY <loic.pallardy@st.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Wendy Liang <sunnyliangjy@gmail.com>
> > Sent: mardi 4 décembre 2018 18:57
> > To: Suman Anna <s-anna@ti.com>
> > Cc: Loic PALLARDY <loic.pallardy@st.com>; Bjorn Andersson
> > <bjorn.andersson@linaro.org>; Ohad Ben-Cohen <ohad@wizery.com>;
> > linux-remoteproc@vger.kernel.org; Linux Kernel Mailing List <linux-
> > kernel@vger.kernel.org>; Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> > Benjamin Gaignard <benjamin.gaignard@linaro.org>
> > Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on
> > centralized carveout allocator
> >
> > On Mon, Oct 29, 2018 at 1:19 PM Suman Anna <s-anna@ti.com> wrote:
> > >
> > > Hi Loic,
> > >
> > > On 10/24/18 10:14 AM, Loic PALLARDY wrote:
> > > > Hi Suman,
> > > >
> > > >> -----Original Message-----
> > > >> From: Suman Anna <s-anna@ti.com>
> > > >> Sent: mercredi 24 octobre 2018 02:14
> > > >> To: Loic PALLARDY <loic.pallardy@st.com>; bjorn.andersson@linaro.org;
> > > >> ohad@wizery.com
> > > >> Cc: linux-remoteproc@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > >> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> > > >> benjamin.gaignard@linaro.org
> > > >> Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to
> > rely on
> > > >> centralized carveout allocator
> > > >>
> > > >> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> > > >>> Current version of rproc_alloc_vring function supports only dynamic
> > vring
> > > >>> allocation.
> > > >>>
> > > >>> This patch allows to allocate vrings based on memory region
> > declatation.
> > > >>> Vrings are now manage like memory carveouts, to communize
> > memory
> > > >> management
> > > >>> code in rproc_alloc_registered_carveouts().
> > > >>>
> > > >>> Allocated buffer is retrieved in rp_find_vq() thanks to
> > > >>> rproc_find_carveout_by_name() functions for.
> > > >>>
> > > >>> This patch sets vrings names to vdev"x"vring"y" with x vdev index in
> > > >>> resource table and y vring index in vdev. This will be updated when
> > > >>> name will be associated to vdev in firmware resource table.
> > > >>>
> > > >>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > > >>> ---
> > > >>>  drivers/remoteproc/remoteproc_core.c     | 61 +++++++++++++++++-
> > -----
> > > >> ---------
> > > >>>  drivers/remoteproc/remoteproc_internal.h |  2 ++
> > > >>>  drivers/remoteproc/remoteproc_virtio.c   | 14 +++++++-
> > > >>>  include/linux/remoteproc.h               |  6 ++--
> > > >>>  4 files changed, 51 insertions(+), 32 deletions(-)
> > > >>>
> > > >>> diff --git a/drivers/remoteproc/remoteproc_core.c
> > > >> b/drivers/remoteproc/remoteproc_core.c
> > > >>> index c543d04..4edc6f0 100644
> > > >>> --- a/drivers/remoteproc/remoteproc_core.c
> > > >>> +++ b/drivers/remoteproc/remoteproc_core.c
> > > >>> @@ -53,6 +53,11 @@ typedef int (*rproc_handle_resources_t)(struct
> > > >> rproc *rproc,
> > > >>>  typedef int (*rproc_handle_resource_t)(struct rproc *rproc,
> > > >>>                              void *, int offset, int avail);
> > > >>>
> > > >>> +static int rproc_alloc_carveout(struct rproc *rproc,
> > > >>> +                           struct rproc_mem_entry *mem);
> > > >>> +static int rproc_release_carveout(struct rproc *rproc,
> > > >>> +                             struct rproc_mem_entry *mem);
> > > >>> +
> > > >>>  /* Unique indices for remoteproc devices */
> > > >>>  static DEFINE_IDA(rproc_dev_index);
> > > >>>
> > > >>> @@ -312,21 +317,33 @@ int rproc_alloc_vring(struct rproc_vdev
> > *rvdev,
> > > >> int i)
> > > >>>     struct device *dev = &rproc->dev;
> > > >>>     struct rproc_vring *rvring = &rvdev->vring[i];
> > > >>>     struct fw_rsc_vdev *rsc;
> > > >>> -   dma_addr_t dma;
> > > >>> -   void *va;
> > > >>>     int ret, size, notifyid;
> > > >>> +   struct rproc_mem_entry *mem;
> > > >>>
> > > >>>     /* actual size of vring (in bytes) */
> > > >>>     size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
> > > >>>
> > > >>> -   /*
> > > >>> -    * Allocate non-cacheable memory for the vring. In the future
> > > >>> -    * this call will also configure the IOMMU for us
> > > >>> -    */
> > > >>> -   va = dma_alloc_coherent(dev->parent, size, &dma, GFP_KERNEL);
> > > >>> -   if (!va) {
> > > >>> -           dev_err(dev->parent, "dma_alloc_coherent failed\n");
> > > >>> -           return -EINVAL;
> > > >>> +   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > > >>> +
> > > >>> +   /* Search for pre-registered carveout */
> > > >>> +   mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d",
> > > >> rvdev->index,
> > > >>> +                                     i);
> > > >>> +   if (mem) {
> > > >>> +           if (rproc_check_carveout_da(rproc, mem, rsc->vring[i].da,
> > > >> size))
> > > >>> +                   return -ENOMEM;
> > > >>> +   } else {
> > > >>> +           /* Register carveout in in list */
> > > >>> +           mem = rproc_mem_entry_init(dev, 0, 0, size, rsc-
> > > >>> vring[i].da,
> > > >>> +                                      rproc_alloc_carveout,
> > > >>> +                                      rproc_release_carveout,
> > > >>> +                                      "vdev%dvring%d",
> > > >>> +                                      rvdev->index, i);
> > > >>> +           if (!mem) {
> > > >>> +                   dev_err(dev, "Can't allocate memory entry
> > > >> structure\n");
> > > >>> +                   return -ENOMEM;
> > > >>> +           }
> > > >>> +
> > > >>> +           rproc_add_carveout(rproc, mem);
> > > >>>     }
> > > >>>
> > > >>>     /*
> > > >>> @@ -337,7 +354,6 @@ int rproc_alloc_vring(struct rproc_vdev *rvdev,
> > int i)
> > > >>>     ret = idr_alloc(&rproc->notifyids, rvring, 0, 0, GFP_KERNEL);
> > > >>>     if (ret < 0) {
> > > >>>             dev_err(dev, "idr_alloc failed: %d\n", ret);
> > > >>> -           dma_free_coherent(dev->parent, size, va, dma);
> > > >>>             return ret;
> > > >>>     }
> > > >>>     notifyid = ret;
> > > >>> @@ -346,21 +362,9 @@ int rproc_alloc_vring(struct rproc_vdev
> > *rvdev, int
> > > >> i)
> > > >>>     if (notifyid > rproc->max_notifyid)
> > > >>>             rproc->max_notifyid = notifyid;
> > > >>>
> > > >>> -   dev_dbg(dev, "vring%d: va %pK dma %pad size 0x%x idr %d\n",
> > > >>> -           i, va, &dma, size, notifyid);
> > > >>> -
> > > >>> -   rvring->va = va;
> > > >>> -   rvring->dma = dma;
> > > >>>     rvring->notifyid = notifyid;
> > > >>>
> > > >>> -   /*
> > > >>> -    * Let the rproc know the notifyid and da of this vring.
> > > >>> -    * Not all platforms use dma_alloc_coherent to automatically
> > > >>> -    * set up the iommu. In this case the device address (da) will
> > > >>> -    * hold the physical address and not the device address.
> > > >>> -    */
> > > >>> -   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > > >>> -   rsc->vring[i].da = dma;
> > > >>> +   /* Let the rproc know the notifyid of this vring.*/
> > > >>>     rsc->vring[i].notifyid = notifyid;
> > > >>>     return 0;
> > > >>>  }
> > > >>> @@ -392,12 +396,10 @@ int rproc_alloc_vring(struct rproc_vdev
> > *rvdev,
> > > >> int i)
> > > >>>
> > > >>>  void rproc_free_vring(struct rproc_vring *rvring)
> > > >>>  {
> > > >>> -   int size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
> > > >>>     struct rproc *rproc = rvring->rvdev->rproc;
> > > >>>     int idx = rvring->rvdev->vring - rvring;
> > > >>>     struct fw_rsc_vdev *rsc;
> > > >>>
> > > >>> -   dma_free_coherent(rproc->dev.parent, size, rvring->va, rvring-
> > > >>> dma);
> > > >>>     idr_remove(&rproc->notifyids, rvring->notifyid);
> > > >>
> > > >> Also, I am not sure if FW_RSC_ADDR_ANY semantics were enforced
> > > >> previously on the vring da. It was simply overwritten irrespective of
> > > >> the value. Now, I am running again into the "bad carveout rsc
> > > >> configuration value" due to the iommu_domain_check if
> > > >> !FW_RSC_ADDR_ANY.
> > > >
> > >
> > > Just realized that I missed responding on this thread last week.
> > >
> > > > When are you running into the "bad carveout rsc configuration value" ?
> > > > This patch is creating one carveout per vring to rely on generic carveout
> > allocator.
> > > > Then carveout is retrieved from carveout  list and vring resource table
> > information updated.
> > > >
> > > > If the da of the carveout was fixed in the resource table, it is normal you
> > have this error.
> > >
> > > Yes, and if the vring da value is FW_RSC_ADDR_ANY, then I don't run into
> > > this particular error. It's just that the semantics of vring da is open
> > > previously, and my above comment being 0 was being used as a reset value
> > > as well.
> > >
> > > > To solve that ST driver is registering one fixed carveout per vring (with the
> > right name today)
> > >
> > > Yeah, we still expect to allocate these dynamically, so there won't be
> > > any registration needed.
> > >
> > > > It is the same discussion as with Patch 1.
> > >
> > > Right, except that we have a da from RSC_CARVEOUT and a da from vring,
> > > and the previous code had some slight differences between the two. The
> > > vring da semantics were never set before (value was always being
> > > overwritten, also it didn't have a pa field), whereas the remoteproc.h
> > > documentation did mention about FW_RSC_ADDR_ANY (without any
> > backing
> > > implementation previously) for the RSC_CARVEOUT da, with the entry also
> > > having a field for pa.
> > >
> > > If we consider we can't change fixed coprocessor address requests, rproc
> > > core should stop its execution.
> > > > It is the responsibility of platform driver to register the right memory
> > regions.
> > > > That's what we discussed with you and Bill in OpenAMP forum.
> > > > TI usecase was to have the same DSP firmware with the same resource
> > table being able to run on any DSP.
> > >
> > > Yeah, it only covers one of the usecases/platforms (Keystone 2 DSP
> > > platforms). And this only worked on these platforms so far because we
> > > only were using internal memories - so there were no RSC_CARVEOUT
> > > entries with valid da. Our Davinci DSP is a single instance and we do
> > > have a RSC_CARVEOUT there, which fails due to the same Patch 1 logic
> > > here as well.
> > >
> > > > In that case each DSP platform driver has to provide the right memory
> > region configuration with the correct pa to da.
> > > >
> > > >>
> > > >> FWIW, the rproc_free_vring was actually using the value 0 when
> > resetting.
> > > >
> > > > It is no more needed as the carveout list is cleared at each stop and
> > recreated at each start.
> > > > Moreover resource table (and firmware) is reloaded at each coprocessor
> > start.
> > >
> > > Yes, agreed. This is about the semantics of vring da from before (no
> > > enforcement to strict enforcement of FW_RSC_ADDR_ANY). The
> > overwriting
> > > on da field on vrings with the dma address is actually a mistake, which
> > > we are trying to proliferate more now. It all comes down to the fact of
> > > treating da as dma address when it is not going to be the case on all
> > > remoteprocs.
> > [Wendy] Are we assuming that the vring da is always predefined.
> > But in the Linux kernel side, without IOMMU case, we use
> > dma_alloc_coherent() to
> > allocate for vring address. In this case, there is no gurantee the
> > allocated vrings
> > address matches the predefined value.
> > Or we assume that only vdev devices of the remoteproc can use
> > dma_alloc_coherent()
> > but not subdevices?
> > Or we can still store the rsc table da pointer. And instead of always
> > set da with dma address
> > we can pass the dma address to remoteproc and let remoteproc to do the
> > pa to da conversion if
> > required.
>
> Hi Wendy,
> I pushed a correction patch which check dma address with requested da from resource table.
> If no match, a warning is display, but the sequence is not stopped as requested by Suman.
> I have a next patch which is adding a rproc_pa_to_da() platform dependent function to concert a pa in da understandable by rproc firmware.
> The goal is to answer Suman request, that some rproc can configure an offset in HW to access part of the DRR or an memory.
> On another side, I'm looking how to do the same with the carveout registration functions...
HI Loic,

I agreed that carveout registration functions should allow ANY_ADDRESS
for no IOMMU case.
In my case, the openamp remoteproc virtio assumes that when the app is
running, it assumes the da are all set by master, but the latest
upstream delayed the vring da setting
when it tries to create the vq, it looks, it looks like the remote
should not even start to checking vdev configs until the master set
the status to DRIVER_OK.

Best Regards,
Wendy
>
> Regards,
> Loic
>
>
> >
> > Thanks,
> > Wendy
> > >
> > > regards
> > > Suman
> > >
> > > >
> > > > Regards,
> > > > Loic
> > > >>
> > > >> regards
> > > >> Suman
> > > >>
> > > >>>
> > > >>>     /* reset resource entry info */
> > > >>> @@ -484,6 +486,7 @@ static int rproc_handle_vdev(struct rproc
> > *rproc,
> > > >> struct fw_rsc_vdev *rsc,
> > > >>>
> > > >>>     rvdev->id = rsc->id;
> > > >>>     rvdev->rproc = rproc;
> > > >>> +   rvdev->index = rproc->nb_vdev++;
> > > >>>
> > > >>>     /* parse the vrings */
> > > >>>     for (i = 0; i < rsc->num_of_vrings; i++) {
> > > >>> @@ -528,9 +531,6 @@ void rproc_vdev_release(struct kref *ref)
> > > >>>
> > > >>>     for (id = 0; id < ARRAY_SIZE(rvdev->vring); id++) {
> > > >>>             rvring = &rvdev->vring[id];
> > > >>> -           if (!rvring->va)
> > > >>> -                   continue;
> > > >>> -
> > > >>>             rproc_free_vring(rvring);
> > > >>>     }
> > > >>>
> > > >>> @@ -1322,6 +1322,9 @@ static int rproc_fw_boot(struct rproc *rproc,
> > > >> const struct firmware *fw)
> > > >>>     /* reset max_notifyid */
> > > >>>     rproc->max_notifyid = -1;
> > > >>>
> > > >>> +   /* reset handled vdev */
> > > >>> +   rproc->nb_vdev = 0;
> > > >>> +
> > > >>>     /* handle fw resources which are required to boot rproc */
> > > >>>     ret = rproc_handle_resources(rproc, rproc_loading_handlers);
> > > >>>     if (ret) {
> > > >>> diff --git a/drivers/remoteproc/remoteproc_internal.h
> > > >> b/drivers/remoteproc/remoteproc_internal.h
> > > >>> index 7570beb..f6cad24 100644
> > > >>> --- a/drivers/remoteproc/remoteproc_internal.h
> > > >>> +++ b/drivers/remoteproc/remoteproc_internal.h
> > > >>> @@ -60,6 +60,8 @@ struct dentry *rproc_create_trace_file(const char
> > > >> *name, struct rproc *rproc,
> > > >>>  int rproc_elf_load_rsc_table(struct rproc *rproc, const struct firmware
> > > >> *fw);
> > > >>>  struct resource_table *rproc_elf_find_loaded_rsc_table(struct rproc
> > > >> *rproc,
> > > >>>                                                    const struct firmware
> > > >> *fw);
> > > >>> +struct rproc_mem_entry *
> > > >>> +rproc_find_carveout_by_name(struct rproc *rproc, const char
> > *name,
> > > >> ...);
> > > >>>
> > > >>>  static inline
> > > >>>  int rproc_fw_sanity_check(struct rproc *rproc, const struct firmware
> > *fw)
> > > >>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
> > > >> b/drivers/remoteproc/remoteproc_virtio.c
> > > >>> index bbecd44..de21f62 100644
> > > >>> --- a/drivers/remoteproc/remoteproc_virtio.c
> > > >>> +++ b/drivers/remoteproc/remoteproc_virtio.c
> > > >>> @@ -76,7 +76,9 @@ static struct virtqueue *rp_find_vq(struct
> > > >> virtio_device *vdev,
> > > >>>     struct rproc_vdev *rvdev = vdev_to_rvdev(vdev);
> > > >>>     struct rproc *rproc = vdev_to_rproc(vdev);
> > > >>>     struct device *dev = &rproc->dev;
> > > >>> +   struct rproc_mem_entry *mem;
> > > >>>     struct rproc_vring *rvring;
> > > >>> +   struct fw_rsc_vdev *rsc;
> > > >>>     struct virtqueue *vq;
> > > >>>     void *addr;
> > > >>>     int len, size;
> > > >>> @@ -88,8 +90,14 @@ static struct virtqueue *rp_find_vq(struct
> > > >> virtio_device *vdev,
> > > >>>     if (!name)
> > > >>>             return NULL;
> > > >>>
> > > >>> +   /* Search allocated memory region by name */
> > > >>> +   mem = rproc_find_carveout_by_name(rproc, "vdev%dvring%d",
> > > >> rvdev->index,
> > > >>> +                                     id);
> > > >>> +   if (!mem || !mem->va)
> > > >>> +           return ERR_PTR(-ENOMEM);
> > > >>> +
> > > >>>     rvring = &rvdev->vring[id];
> > > >>> -   addr = rvring->va;
> > > >>> +   addr = mem->va;
> > > >>>     len = rvring->len;
> > > >>>
> > > >>>     /* zero vring */
> > > >>> @@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct
> > > >> virtio_device *vdev,
> > > >>>     rvring->vq = vq;
> > > >>>     vq->priv = rvring;
> > > >>>
> > > >>> +   /* Update vring in resource table */
> > > >>> +   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > > >>> +   rsc->vring[id].da = mem->da;
> > > >>> +
> > > >>>     return vq;
> > > >>>  }
> > > >>>
> > > >>> diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h
> > > >>> index 4cdd0c6..6b3a234 100644
> > > >>> --- a/include/linux/remoteproc.h
> > > >>> +++ b/include/linux/remoteproc.h
> > > >>> @@ -453,6 +453,7 @@ struct rproc_dump_segment {
> > > >>>   * @table_sz: size of @cached_table
> > > >>>   * @has_iommu: flag to indicate if remote processor is behind an
> > MMU
> > > >>>   * @dump_segments: list of segments in the firmware
> > > >>> + * @nb_vdev: number of vdev currently handled by rproc
> > > >>>   */
> > > >>>  struct rproc {
> > > >>>     struct list_head node;
> > > >>> @@ -485,6 +486,7 @@ struct rproc {
> > > >>>     bool has_iommu;
> > > >>>     bool auto_boot;
> > > >>>     struct list_head dump_segments;
> > > >>> +   int nb_vdev;
> > > >>>  };
> > > >>>
> > > >>>  /**
> > > >>> @@ -512,7 +514,6 @@ struct rproc_subdev {
> > > >>>  /**
> > > >>>   * struct rproc_vring - remoteproc vring state
> > > >>>   * @va:    virtual address
> > > >>> - * @dma: dma address
> > > >>>   * @len: length, in bytes
> > > >>>   * @da: device address
> > > >>>   * @align: vring alignment
> > > >>> @@ -522,7 +523,6 @@ struct rproc_subdev {
> > > >>>   */
> > > >>>  struct rproc_vring {
> > > >>>     void *va;
> > > >>> -   dma_addr_t dma;
> > > >>>     int len;
> > > >>>     u32 da;
> > > >>>     u32 align;
> > > >>> @@ -541,6 +541,7 @@ struct rproc_vring {
> > > >>>   * @vdev: the virio device
> > > >>>   * @vring: the vrings for this vdev
> > > >>>   * @rsc_offset: offset of the vdev's resource entry
> > > >>> + * @index: vdev position versus other vdev declared in resource table
> > > >>>   */
> > > >>>  struct rproc_vdev {
> > > >>>     struct kref refcount;
> > > >>> @@ -553,6 +554,7 @@ struct rproc_vdev {
> > > >>>     struct virtio_device vdev;
> > > >>>     struct rproc_vring vring[RVDEV_NUM_VRINGS];
> > > >>>     u32 rsc_offset;
> > > >>> +   u32 index;
> > > >>>  };
> > > >>>
> > > >>>  struct rproc *rproc_get_by_phandle(phandle phandle);
> > > >>>
> > > >
> > >

^ permalink raw reply	[flat|nested] 75+ messages in thread

* RE: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-12-04 18:58             ` Wendy Liang
@ 2018-12-04 19:57               ` Loic PALLARDY
  2018-12-04 21:24                 ` Wendy Liang
  0 siblings, 1 reply; 75+ messages in thread
From: Loic PALLARDY @ 2018-12-04 19:57 UTC (permalink / raw)
  To: Wendy Liang
  Cc: Suman Anna, Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, Arnaud POULIQUEN, Benjamin Gaignard



> -----Original Message-----
> From: Wendy Liang <sunnyliangjy@gmail.com>
> Sent: mardi 4 décembre 2018 19:58
> To: Loic PALLARDY <loic.pallardy@st.com>
> Cc: Suman Anna <s-anna@ti.com>; Bjorn Andersson
> <bjorn.andersson@linaro.org>; Ohad Ben-Cohen <ohad@wizery.com>;
> linux-remoteproc@vger.kernel.org; Linux Kernel Mailing List <linux-
> kernel@vger.kernel.org>; Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> Benjamin Gaignard <benjamin.gaignard@linaro.org>
> Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on
> centralized carveout allocator
> 
> On Tue, Dec 4, 2018 at 10:04 AM Loic PALLARDY <loic.pallardy@st.com>
> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Wendy Liang <sunnyliangjy@gmail.com>
> > > Sent: mardi 4 décembre 2018 18:57
> > > To: Suman Anna <s-anna@ti.com>
> > > Cc: Loic PALLARDY <loic.pallardy@st.com>; Bjorn Andersson
> > > <bjorn.andersson@linaro.org>; Ohad Ben-Cohen <ohad@wizery.com>;
> > > linux-remoteproc@vger.kernel.org; Linux Kernel Mailing List <linux-
> > > kernel@vger.kernel.org>; Arnaud POULIQUEN
> <arnaud.pouliquen@st.com>;
> > > Benjamin Gaignard <benjamin.gaignard@linaro.org>
> > > Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely
> on
> > > centralized carveout allocator
> > >
> > > On Mon, Oct 29, 2018 at 1:19 PM Suman Anna <s-anna@ti.com> wrote:
> > > >
> > > > Hi Loic,
> > > >
> > > > On 10/24/18 10:14 AM, Loic PALLARDY wrote:
> > > > > Hi Suman,
> > > > >
> > > > >> -----Original Message-----
> > > > >> From: Suman Anna <s-anna@ti.com>
> > > > >> Sent: mercredi 24 octobre 2018 02:14
> > > > >> To: Loic PALLARDY <loic.pallardy@st.com>;
> bjorn.andersson@linaro.org;
> > > > >> ohad@wizery.com
> > > > >> Cc: linux-remoteproc@vger.kernel.org; linux-
> kernel@vger.kernel.org;
> > > > >> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> > > > >> benjamin.gaignard@linaro.org
> > > > >> Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to
> > > rely on
> > > > >> centralized carveout allocator
> > > > >>
> > > > >> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> > > > >>> Current version of rproc_alloc_vring function supports only
> dynamic
> > > vring
> > > > >>> allocation.
> > > > >>>
> > > > >>> This patch allows to allocate vrings based on memory region
> > > declatation.
> > > > >>> Vrings are now manage like memory carveouts, to communize
> > > memory
> > > > >> management
> > > > >>> code in rproc_alloc_registered_carveouts().
> > > > >>>
> > > > >>> Allocated buffer is retrieved in rp_find_vq() thanks to
> > > > >>> rproc_find_carveout_by_name() functions for.
> > > > >>>
> > > > >>> This patch sets vrings names to vdev"x"vring"y" with x vdev index
> in
> > > > >>> resource table and y vring index in vdev. This will be updated when
> > > > >>> name will be associated to vdev in firmware resource table.
> > > > >>>
> > > > >>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > > > >>> ---
> > > > >>>  drivers/remoteproc/remoteproc_core.c     | 61
> +++++++++++++++++-
> > > -----
> > > > >> ---------
> > > > >>>  drivers/remoteproc/remoteproc_internal.h |  2 ++
> > > > >>>  drivers/remoteproc/remoteproc_virtio.c   | 14 +++++++-
> > > > >>>  include/linux/remoteproc.h               |  6 ++--
> > > > >>>  4 files changed, 51 insertions(+), 32 deletions(-)
> > > > >>>
> > > > >>> diff --git a/drivers/remoteproc/remoteproc_core.c
> > > > >> b/drivers/remoteproc/remoteproc_core.c
> > > > >>> index c543d04..4edc6f0 100644
> > > > >>> --- a/drivers/remoteproc/remoteproc_core.c
> > > > >>> +++ b/drivers/remoteproc/remoteproc_core.c
> > > > >>> @@ -53,6 +53,11 @@ typedef int
> (*rproc_handle_resources_t)(struct
> > > > >> rproc *rproc,
> > > > >>>  typedef int (*rproc_handle_resource_t)(struct rproc *rproc,
> > > > >>>                              void *, int offset, int avail);
> > > > >>>
> > > > >>> +static int rproc_alloc_carveout(struct rproc *rproc,
> > > > >>> +                           struct rproc_mem_entry *mem);
> > > > >>> +static int rproc_release_carveout(struct rproc *rproc,
> > > > >>> +                             struct rproc_mem_entry *mem);
> > > > >>> +
> > > > >>>  /* Unique indices for remoteproc devices */
> > > > >>>  static DEFINE_IDA(rproc_dev_index);
> > > > >>>
> > > > >>> @@ -312,21 +317,33 @@ int rproc_alloc_vring(struct rproc_vdev
> > > *rvdev,
> > > > >> int i)
> > > > >>>     struct device *dev = &rproc->dev;
> > > > >>>     struct rproc_vring *rvring = &rvdev->vring[i];
> > > > >>>     struct fw_rsc_vdev *rsc;
> > > > >>> -   dma_addr_t dma;
> > > > >>> -   void *va;
> > > > >>>     int ret, size, notifyid;
> > > > >>> +   struct rproc_mem_entry *mem;
> > > > >>>
> > > > >>>     /* actual size of vring (in bytes) */
> > > > >>>     size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
> > > > >>>
> > > > >>> -   /*
> > > > >>> -    * Allocate non-cacheable memory for the vring. In the future
> > > > >>> -    * this call will also configure the IOMMU for us
> > > > >>> -    */
> > > > >>> -   va = dma_alloc_coherent(dev->parent, size, &dma,
> GFP_KERNEL);
> > > > >>> -   if (!va) {
> > > > >>> -           dev_err(dev->parent, "dma_alloc_coherent failed\n");
> > > > >>> -           return -EINVAL;
> > > > >>> +   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > > > >>> +
> > > > >>> +   /* Search for pre-registered carveout */
> > > > >>> +   mem = rproc_find_carveout_by_name(rproc,
> "vdev%dvring%d",
> > > > >> rvdev->index,
> > > > >>> +                                     i);
> > > > >>> +   if (mem) {
> > > > >>> +           if (rproc_check_carveout_da(rproc, mem, rsc->vring[i].da,
> > > > >> size))
> > > > >>> +                   return -ENOMEM;
> > > > >>> +   } else {
> > > > >>> +           /* Register carveout in in list */
> > > > >>> +           mem = rproc_mem_entry_init(dev, 0, 0, size, rsc-
> > > > >>> vring[i].da,
> > > > >>> +                                      rproc_alloc_carveout,
> > > > >>> +                                      rproc_release_carveout,
> > > > >>> +                                      "vdev%dvring%d",
> > > > >>> +                                      rvdev->index, i);
> > > > >>> +           if (!mem) {
> > > > >>> +                   dev_err(dev, "Can't allocate memory entry
> > > > >> structure\n");
> > > > >>> +                   return -ENOMEM;
> > > > >>> +           }
> > > > >>> +
> > > > >>> +           rproc_add_carveout(rproc, mem);
> > > > >>>     }
> > > > >>>
> > > > >>>     /*
> > > > >>> @@ -337,7 +354,6 @@ int rproc_alloc_vring(struct rproc_vdev
> *rvdev,
> > > int i)
> > > > >>>     ret = idr_alloc(&rproc->notifyids, rvring, 0, 0, GFP_KERNEL);
> > > > >>>     if (ret < 0) {
> > > > >>>             dev_err(dev, "idr_alloc failed: %d\n", ret);
> > > > >>> -           dma_free_coherent(dev->parent, size, va, dma);
> > > > >>>             return ret;
> > > > >>>     }
> > > > >>>     notifyid = ret;
> > > > >>> @@ -346,21 +362,9 @@ int rproc_alloc_vring(struct rproc_vdev
> > > *rvdev, int
> > > > >> i)
> > > > >>>     if (notifyid > rproc->max_notifyid)
> > > > >>>             rproc->max_notifyid = notifyid;
> > > > >>>
> > > > >>> -   dev_dbg(dev, "vring%d: va %pK dma %pad size 0x%x idr %d\n",
> > > > >>> -           i, va, &dma, size, notifyid);
> > > > >>> -
> > > > >>> -   rvring->va = va;
> > > > >>> -   rvring->dma = dma;
> > > > >>>     rvring->notifyid = notifyid;
> > > > >>>
> > > > >>> -   /*
> > > > >>> -    * Let the rproc know the notifyid and da of this vring.
> > > > >>> -    * Not all platforms use dma_alloc_coherent to automatically
> > > > >>> -    * set up the iommu. In this case the device address (da) will
> > > > >>> -    * hold the physical address and not the device address.
> > > > >>> -    */
> > > > >>> -   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > > > >>> -   rsc->vring[i].da = dma;
> > > > >>> +   /* Let the rproc know the notifyid of this vring.*/
> > > > >>>     rsc->vring[i].notifyid = notifyid;
> > > > >>>     return 0;
> > > > >>>  }
> > > > >>> @@ -392,12 +396,10 @@ int rproc_alloc_vring(struct rproc_vdev
> > > *rvdev,
> > > > >> int i)
> > > > >>>
> > > > >>>  void rproc_free_vring(struct rproc_vring *rvring)
> > > > >>>  {
> > > > >>> -   int size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
> > > > >>>     struct rproc *rproc = rvring->rvdev->rproc;
> > > > >>>     int idx = rvring->rvdev->vring - rvring;
> > > > >>>     struct fw_rsc_vdev *rsc;
> > > > >>>
> > > > >>> -   dma_free_coherent(rproc->dev.parent, size, rvring->va, rvring-
> > > > >>> dma);
> > > > >>>     idr_remove(&rproc->notifyids, rvring->notifyid);
> > > > >>
> > > > >> Also, I am not sure if FW_RSC_ADDR_ANY semantics were enforced
> > > > >> previously on the vring da. It was simply overwritten irrespective of
> > > > >> the value. Now, I am running again into the "bad carveout rsc
> > > > >> configuration value" due to the iommu_domain_check if
> > > > >> !FW_RSC_ADDR_ANY.
> > > > >
> > > >
> > > > Just realized that I missed responding on this thread last week.
> > > >
> > > > > When are you running into the "bad carveout rsc configuration value"
> ?
> > > > > This patch is creating one carveout per vring to rely on generic
> carveout
> > > allocator.
> > > > > Then carveout is retrieved from carveout  list and vring resource table
> > > information updated.
> > > > >
> > > > > If the da of the carveout was fixed in the resource table, it is normal
> you
> > > have this error.
> > > >
> > > > Yes, and if the vring da value is FW_RSC_ADDR_ANY, then I don't run
> into
> > > > this particular error. It's just that the semantics of vring da is open
> > > > previously, and my above comment being 0 was being used as a reset
> value
> > > > as well.
> > > >
> > > > > To solve that ST driver is registering one fixed carveout per vring (with
> the
> > > right name today)
> > > >
> > > > Yeah, we still expect to allocate these dynamically, so there won't be
> > > > any registration needed.
> > > >
> > > > > It is the same discussion as with Patch 1.
> > > >
> > > > Right, except that we have a da from RSC_CARVEOUT and a da from
> vring,
> > > > and the previous code had some slight differences between the two.
> The
> > > > vring da semantics were never set before (value was always being
> > > > overwritten, also it didn't have a pa field), whereas the remoteproc.h
> > > > documentation did mention about FW_RSC_ADDR_ANY (without any
> > > backing
> > > > implementation previously) for the RSC_CARVEOUT da, with the entry
> also
> > > > having a field for pa.
> > > >
> > > > If we consider we can't change fixed coprocessor address requests,
> rproc
> > > > core should stop its execution.
> > > > > It is the responsibility of platform driver to register the right memory
> > > regions.
> > > > > That's what we discussed with you and Bill in OpenAMP forum.
> > > > > TI usecase was to have the same DSP firmware with the same
> resource
> > > table being able to run on any DSP.
> > > >
> > > > Yeah, it only covers one of the usecases/platforms (Keystone 2 DSP
> > > > platforms). And this only worked on these platforms so far because we
> > > > only were using internal memories - so there were no RSC_CARVEOUT
> > > > entries with valid da. Our Davinci DSP is a single instance and we do
> > > > have a RSC_CARVEOUT there, which fails due to the same Patch 1 logic
> > > > here as well.
> > > >
> > > > > In that case each DSP platform driver has to provide the right memory
> > > region configuration with the correct pa to da.
> > > > >
> > > > >>
> > > > >> FWIW, the rproc_free_vring was actually using the value 0 when
> > > resetting.
> > > > >
> > > > > It is no more needed as the carveout list is cleared at each stop and
> > > recreated at each start.
> > > > > Moreover resource table (and firmware) is reloaded at each
> coprocessor
> > > start.
> > > >
> > > > Yes, agreed. This is about the semantics of vring da from before (no
> > > > enforcement to strict enforcement of FW_RSC_ADDR_ANY). The
> > > overwriting
> > > > on da field on vrings with the dma address is actually a mistake, which
> > > > we are trying to proliferate more now. It all comes down to the fact of
> > > > treating da as dma address when it is not going to be the case on all
> > > > remoteprocs.
> > > [Wendy] Are we assuming that the vring da is always predefined.
> > > But in the Linux kernel side, without IOMMU case, we use
> > > dma_alloc_coherent() to
> > > allocate for vring address. In this case, there is no gurantee the
> > > allocated vrings
> > > address matches the predefined value.
> > > Or we assume that only vdev devices of the remoteproc can use
> > > dma_alloc_coherent()
> > > but not subdevices?
> > > Or we can still store the rsc table da pointer. And instead of always
> > > set da with dma address
> > > we can pass the dma address to remoteproc and let remoteproc to do
> the
> > > pa to da conversion if
> > > required.
> >
> > Hi Wendy,
> > I pushed a correction patch which check dma address with requested da
> from resource table.
> > If no match, a warning is display, but the sequence is not stopped as
> requested by Suman.
> > I have a next patch which is adding a rproc_pa_to_da() platform
> dependent function to concert a pa in da understandable by rproc firmware.
> > The goal is to answer Suman request, that some rproc can configure an
> offset in HW to access part of the DRR or an memory.
> > On another side, I'm looking how to do the same with the carveout
> registration functions...
> HI Loic,
> 
> I agreed that carveout registration functions should allow ANY_ADDRESS
> for no IOMMU case.
> In my case, the openamp remoteproc virtio assumes that when the app is
> running, it assumes the da are all set by master, but the latest
> upstream delayed the vring da setting
> when it tries to create the vq, it looks, it looks like the remote
> should not even start to checking vdev configs until the master set
> the status to DRIVER_OK.

Yes that's an old discussion I had with Bjorn, when he introduced sub dev. Rpmsg buffer are allocated after rproc start.
But we agree on the fact master and slave initialization can be fixed and slave should use synchronization flags like
virtio kick or vdev status before accessing configuration. New implementation is respecting same rule.
About this issue, I know Arnaud has done a development on OpenAMP side to wait vdev status OK before reading vring da.

Regards,
Loic
> 
> Best Regards,
> Wendy
> >
> > Regards,
> > Loic
> >
> >
> > >
> > > Thanks,
> > > Wendy
> > > >
> > > > regards
> > > > Suman
> > > >
> > > > >
> > > > > Regards,
> > > > > Loic
> > > > >>
> > > > >> regards
> > > > >> Suman
> > > > >>
> > > > >>>
> > > > >>>     /* reset resource entry info */
> > > > >>> @@ -484,6 +486,7 @@ static int rproc_handle_vdev(struct rproc
> > > *rproc,
> > > > >> struct fw_rsc_vdev *rsc,
> > > > >>>
> > > > >>>     rvdev->id = rsc->id;
> > > > >>>     rvdev->rproc = rproc;
> > > > >>> +   rvdev->index = rproc->nb_vdev++;
> > > > >>>
> > > > >>>     /* parse the vrings */
> > > > >>>     for (i = 0; i < rsc->num_of_vrings; i++) {
> > > > >>> @@ -528,9 +531,6 @@ void rproc_vdev_release(struct kref *ref)
> > > > >>>
> > > > >>>     for (id = 0; id < ARRAY_SIZE(rvdev->vring); id++) {
> > > > >>>             rvring = &rvdev->vring[id];
> > > > >>> -           if (!rvring->va)
> > > > >>> -                   continue;
> > > > >>> -
> > > > >>>             rproc_free_vring(rvring);
> > > > >>>     }
> > > > >>>
> > > > >>> @@ -1322,6 +1322,9 @@ static int rproc_fw_boot(struct rproc
> *rproc,
> > > > >> const struct firmware *fw)
> > > > >>>     /* reset max_notifyid */
> > > > >>>     rproc->max_notifyid = -1;
> > > > >>>
> > > > >>> +   /* reset handled vdev */
> > > > >>> +   rproc->nb_vdev = 0;
> > > > >>> +
> > > > >>>     /* handle fw resources which are required to boot rproc */
> > > > >>>     ret = rproc_handle_resources(rproc, rproc_loading_handlers);
> > > > >>>     if (ret) {
> > > > >>> diff --git a/drivers/remoteproc/remoteproc_internal.h
> > > > >> b/drivers/remoteproc/remoteproc_internal.h
> > > > >>> index 7570beb..f6cad24 100644
> > > > >>> --- a/drivers/remoteproc/remoteproc_internal.h
> > > > >>> +++ b/drivers/remoteproc/remoteproc_internal.h
> > > > >>> @@ -60,6 +60,8 @@ struct dentry *rproc_create_trace_file(const
> char
> > > > >> *name, struct rproc *rproc,
> > > > >>>  int rproc_elf_load_rsc_table(struct rproc *rproc, const struct
> firmware
> > > > >> *fw);
> > > > >>>  struct resource_table *rproc_elf_find_loaded_rsc_table(struct
> rproc
> > > > >> *rproc,
> > > > >>>                                                    const struct firmware
> > > > >> *fw);
> > > > >>> +struct rproc_mem_entry *
> > > > >>> +rproc_find_carveout_by_name(struct rproc *rproc, const char
> > > *name,
> > > > >> ...);
> > > > >>>
> > > > >>>  static inline
> > > > >>>  int rproc_fw_sanity_check(struct rproc *rproc, const struct
> firmware
> > > *fw)
> > > > >>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
> > > > >> b/drivers/remoteproc/remoteproc_virtio.c
> > > > >>> index bbecd44..de21f62 100644
> > > > >>> --- a/drivers/remoteproc/remoteproc_virtio.c
> > > > >>> +++ b/drivers/remoteproc/remoteproc_virtio.c
> > > > >>> @@ -76,7 +76,9 @@ static struct virtqueue *rp_find_vq(struct
> > > > >> virtio_device *vdev,
> > > > >>>     struct rproc_vdev *rvdev = vdev_to_rvdev(vdev);
> > > > >>>     struct rproc *rproc = vdev_to_rproc(vdev);
> > > > >>>     struct device *dev = &rproc->dev;
> > > > >>> +   struct rproc_mem_entry *mem;
> > > > >>>     struct rproc_vring *rvring;
> > > > >>> +   struct fw_rsc_vdev *rsc;
> > > > >>>     struct virtqueue *vq;
> > > > >>>     void *addr;
> > > > >>>     int len, size;
> > > > >>> @@ -88,8 +90,14 @@ static struct virtqueue *rp_find_vq(struct
> > > > >> virtio_device *vdev,
> > > > >>>     if (!name)
> > > > >>>             return NULL;
> > > > >>>
> > > > >>> +   /* Search allocated memory region by name */
> > > > >>> +   mem = rproc_find_carveout_by_name(rproc,
> "vdev%dvring%d",
> > > > >> rvdev->index,
> > > > >>> +                                     id);
> > > > >>> +   if (!mem || !mem->va)
> > > > >>> +           return ERR_PTR(-ENOMEM);
> > > > >>> +
> > > > >>>     rvring = &rvdev->vring[id];
> > > > >>> -   addr = rvring->va;
> > > > >>> +   addr = mem->va;
> > > > >>>     len = rvring->len;
> > > > >>>
> > > > >>>     /* zero vring */
> > > > >>> @@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct
> > > > >> virtio_device *vdev,
> > > > >>>     rvring->vq = vq;
> > > > >>>     vq->priv = rvring;
> > > > >>>
> > > > >>> +   /* Update vring in resource table */
> > > > >>> +   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > > > >>> +   rsc->vring[id].da = mem->da;
> > > > >>> +
> > > > >>>     return vq;
> > > > >>>  }
> > > > >>>
> > > > >>> diff --git a/include/linux/remoteproc.h
> b/include/linux/remoteproc.h
> > > > >>> index 4cdd0c6..6b3a234 100644
> > > > >>> --- a/include/linux/remoteproc.h
> > > > >>> +++ b/include/linux/remoteproc.h
> > > > >>> @@ -453,6 +453,7 @@ struct rproc_dump_segment {
> > > > >>>   * @table_sz: size of @cached_table
> > > > >>>   * @has_iommu: flag to indicate if remote processor is behind an
> > > MMU
> > > > >>>   * @dump_segments: list of segments in the firmware
> > > > >>> + * @nb_vdev: number of vdev currently handled by rproc
> > > > >>>   */
> > > > >>>  struct rproc {
> > > > >>>     struct list_head node;
> > > > >>> @@ -485,6 +486,7 @@ struct rproc {
> > > > >>>     bool has_iommu;
> > > > >>>     bool auto_boot;
> > > > >>>     struct list_head dump_segments;
> > > > >>> +   int nb_vdev;
> > > > >>>  };
> > > > >>>
> > > > >>>  /**
> > > > >>> @@ -512,7 +514,6 @@ struct rproc_subdev {
> > > > >>>  /**
> > > > >>>   * struct rproc_vring - remoteproc vring state
> > > > >>>   * @va:    virtual address
> > > > >>> - * @dma: dma address
> > > > >>>   * @len: length, in bytes
> > > > >>>   * @da: device address
> > > > >>>   * @align: vring alignment
> > > > >>> @@ -522,7 +523,6 @@ struct rproc_subdev {
> > > > >>>   */
> > > > >>>  struct rproc_vring {
> > > > >>>     void *va;
> > > > >>> -   dma_addr_t dma;
> > > > >>>     int len;
> > > > >>>     u32 da;
> > > > >>>     u32 align;
> > > > >>> @@ -541,6 +541,7 @@ struct rproc_vring {
> > > > >>>   * @vdev: the virio device
> > > > >>>   * @vring: the vrings for this vdev
> > > > >>>   * @rsc_offset: offset of the vdev's resource entry
> > > > >>> + * @index: vdev position versus other vdev declared in resource
> table
> > > > >>>   */
> > > > >>>  struct rproc_vdev {
> > > > >>>     struct kref refcount;
> > > > >>> @@ -553,6 +554,7 @@ struct rproc_vdev {
> > > > >>>     struct virtio_device vdev;
> > > > >>>     struct rproc_vring vring[RVDEV_NUM_VRINGS];
> > > > >>>     u32 rsc_offset;
> > > > >>> +   u32 index;
> > > > >>>  };
> > > > >>>
> > > > >>>  struct rproc *rproc_get_by_phandle(phandle phandle);
> > > > >>>
> > > > >
> > > >

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator
  2018-12-04 19:57               ` Loic PALLARDY
@ 2018-12-04 21:24                 ` Wendy Liang
  0 siblings, 0 replies; 75+ messages in thread
From: Wendy Liang @ 2018-12-04 21:24 UTC (permalink / raw)
  To: Loic Pallardy
  Cc: Suman Anna, Bjorn Andersson, Ohad Ben-Cohen, linux-remoteproc,
	Linux Kernel Mailing List, arnaud pouliquen, Benjamin Gaignard

On Tue, Dec 4, 2018 at 11:57 AM Loic PALLARDY <loic.pallardy@st.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Wendy Liang <sunnyliangjy@gmail.com>
> > Sent: mardi 4 décembre 2018 19:58
> > To: Loic PALLARDY <loic.pallardy@st.com>
> > Cc: Suman Anna <s-anna@ti.com>; Bjorn Andersson
> > <bjorn.andersson@linaro.org>; Ohad Ben-Cohen <ohad@wizery.com>;
> > linux-remoteproc@vger.kernel.org; Linux Kernel Mailing List <linux-
> > kernel@vger.kernel.org>; Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> > Benjamin Gaignard <benjamin.gaignard@linaro.org>
> > Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely on
> > centralized carveout allocator
> >
> > On Tue, Dec 4, 2018 at 10:04 AM Loic PALLARDY <loic.pallardy@st.com>
> > wrote:
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Wendy Liang <sunnyliangjy@gmail.com>
> > > > Sent: mardi 4 décembre 2018 18:57
> > > > To: Suman Anna <s-anna@ti.com>
> > > > Cc: Loic PALLARDY <loic.pallardy@st.com>; Bjorn Andersson
> > > > <bjorn.andersson@linaro.org>; Ohad Ben-Cohen <ohad@wizery.com>;
> > > > linux-remoteproc@vger.kernel.org; Linux Kernel Mailing List <linux-
> > > > kernel@vger.kernel.org>; Arnaud POULIQUEN
> > <arnaud.pouliquen@st.com>;
> > > > Benjamin Gaignard <benjamin.gaignard@linaro.org>
> > > > Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to rely
> > on
> > > > centralized carveout allocator
> > > >
> > > > On Mon, Oct 29, 2018 at 1:19 PM Suman Anna <s-anna@ti.com> wrote:
> > > > >
> > > > > Hi Loic,
> > > > >
> > > > > On 10/24/18 10:14 AM, Loic PALLARDY wrote:
> > > > > > Hi Suman,
> > > > > >
> > > > > >> -----Original Message-----
> > > > > >> From: Suman Anna <s-anna@ti.com>
> > > > > >> Sent: mercredi 24 octobre 2018 02:14
> > > > > >> To: Loic PALLARDY <loic.pallardy@st.com>;
> > bjorn.andersson@linaro.org;
> > > > > >> ohad@wizery.com
> > > > > >> Cc: linux-remoteproc@vger.kernel.org; linux-
> > kernel@vger.kernel.org;
> > > > > >> Arnaud POULIQUEN <arnaud.pouliquen@st.com>;
> > > > > >> benjamin.gaignard@linaro.org
> > > > > >> Subject: Re: [PATCH v4 12/17] remoteproc: modify vring allocation to
> > > > rely on
> > > > > >> centralized carveout allocator
> > > > > >>
> > > > > >> On 7/27/18 8:14 AM, Loic Pallardy wrote:
> > > > > >>> Current version of rproc_alloc_vring function supports only
> > dynamic
> > > > vring
> > > > > >>> allocation.
> > > > > >>>
> > > > > >>> This patch allows to allocate vrings based on memory region
> > > > declatation.
> > > > > >>> Vrings are now manage like memory carveouts, to communize
> > > > memory
> > > > > >> management
> > > > > >>> code in rproc_alloc_registered_carveouts().
> > > > > >>>
> > > > > >>> Allocated buffer is retrieved in rp_find_vq() thanks to
> > > > > >>> rproc_find_carveout_by_name() functions for.
> > > > > >>>
> > > > > >>> This patch sets vrings names to vdev"x"vring"y" with x vdev index
> > in
> > > > > >>> resource table and y vring index in vdev. This will be updated when
> > > > > >>> name will be associated to vdev in firmware resource table.
> > > > > >>>
> > > > > >>> Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
> > > > > >>> ---
> > > > > >>>  drivers/remoteproc/remoteproc_core.c     | 61
> > +++++++++++++++++-
> > > > -----
> > > > > >> ---------
> > > > > >>>  drivers/remoteproc/remoteproc_internal.h |  2 ++
> > > > > >>>  drivers/remoteproc/remoteproc_virtio.c   | 14 +++++++-
> > > > > >>>  include/linux/remoteproc.h               |  6 ++--
> > > > > >>>  4 files changed, 51 insertions(+), 32 deletions(-)
> > > > > >>>
> > > > > >>> diff --git a/drivers/remoteproc/remoteproc_core.c
> > > > > >> b/drivers/remoteproc/remoteproc_core.c
> > > > > >>> index c543d04..4edc6f0 100644
> > > > > >>> --- a/drivers/remoteproc/remoteproc_core.c
> > > > > >>> +++ b/drivers/remoteproc/remoteproc_core.c
> > > > > >>> @@ -53,6 +53,11 @@ typedef int
> > (*rproc_handle_resources_t)(struct
> > > > > >> rproc *rproc,
> > > > > >>>  typedef int (*rproc_handle_resource_t)(struct rproc *rproc,
> > > > > >>>                              void *, int offset, int avail);
> > > > > >>>
> > > > > >>> +static int rproc_alloc_carveout(struct rproc *rproc,
> > > > > >>> +                           struct rproc_mem_entry *mem);
> > > > > >>> +static int rproc_release_carveout(struct rproc *rproc,
> > > > > >>> +                             struct rproc_mem_entry *mem);
> > > > > >>> +
> > > > > >>>  /* Unique indices for remoteproc devices */
> > > > > >>>  static DEFINE_IDA(rproc_dev_index);
> > > > > >>>
> > > > > >>> @@ -312,21 +317,33 @@ int rproc_alloc_vring(struct rproc_vdev
> > > > *rvdev,
> > > > > >> int i)
> > > > > >>>     struct device *dev = &rproc->dev;
> > > > > >>>     struct rproc_vring *rvring = &rvdev->vring[i];
> > > > > >>>     struct fw_rsc_vdev *rsc;
> > > > > >>> -   dma_addr_t dma;
> > > > > >>> -   void *va;
> > > > > >>>     int ret, size, notifyid;
> > > > > >>> +   struct rproc_mem_entry *mem;
> > > > > >>>
> > > > > >>>     /* actual size of vring (in bytes) */
> > > > > >>>     size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
> > > > > >>>
> > > > > >>> -   /*
> > > > > >>> -    * Allocate non-cacheable memory for the vring. In the future
> > > > > >>> -    * this call will also configure the IOMMU for us
> > > > > >>> -    */
> > > > > >>> -   va = dma_alloc_coherent(dev->parent, size, &dma,
> > GFP_KERNEL);
> > > > > >>> -   if (!va) {
> > > > > >>> -           dev_err(dev->parent, "dma_alloc_coherent failed\n");
> > > > > >>> -           return -EINVAL;
> > > > > >>> +   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > > > > >>> +
> > > > > >>> +   /* Search for pre-registered carveout */
> > > > > >>> +   mem = rproc_find_carveout_by_name(rproc,
> > "vdev%dvring%d",
> > > > > >> rvdev->index,
> > > > > >>> +                                     i);
> > > > > >>> +   if (mem) {
> > > > > >>> +           if (rproc_check_carveout_da(rproc, mem, rsc->vring[i].da,
> > > > > >> size))
> > > > > >>> +                   return -ENOMEM;
> > > > > >>> +   } else {
> > > > > >>> +           /* Register carveout in in list */
> > > > > >>> +           mem = rproc_mem_entry_init(dev, 0, 0, size, rsc-
> > > > > >>> vring[i].da,
> > > > > >>> +                                      rproc_alloc_carveout,
> > > > > >>> +                                      rproc_release_carveout,
> > > > > >>> +                                      "vdev%dvring%d",
> > > > > >>> +                                      rvdev->index, i);
> > > > > >>> +           if (!mem) {
> > > > > >>> +                   dev_err(dev, "Can't allocate memory entry
> > > > > >> structure\n");
> > > > > >>> +                   return -ENOMEM;
> > > > > >>> +           }
> > > > > >>> +
> > > > > >>> +           rproc_add_carveout(rproc, mem);
> > > > > >>>     }
> > > > > >>>
> > > > > >>>     /*
> > > > > >>> @@ -337,7 +354,6 @@ int rproc_alloc_vring(struct rproc_vdev
> > *rvdev,
> > > > int i)
> > > > > >>>     ret = idr_alloc(&rproc->notifyids, rvring, 0, 0, GFP_KERNEL);
> > > > > >>>     if (ret < 0) {
> > > > > >>>             dev_err(dev, "idr_alloc failed: %d\n", ret);
> > > > > >>> -           dma_free_coherent(dev->parent, size, va, dma);
> > > > > >>>             return ret;
> > > > > >>>     }
> > > > > >>>     notifyid = ret;
> > > > > >>> @@ -346,21 +362,9 @@ int rproc_alloc_vring(struct rproc_vdev
> > > > *rvdev, int
> > > > > >> i)
> > > > > >>>     if (notifyid > rproc->max_notifyid)
> > > > > >>>             rproc->max_notifyid = notifyid;
> > > > > >>>
> > > > > >>> -   dev_dbg(dev, "vring%d: va %pK dma %pad size 0x%x idr %d\n",
> > > > > >>> -           i, va, &dma, size, notifyid);
> > > > > >>> -
> > > > > >>> -   rvring->va = va;
> > > > > >>> -   rvring->dma = dma;
> > > > > >>>     rvring->notifyid = notifyid;
> > > > > >>>
> > > > > >>> -   /*
> > > > > >>> -    * Let the rproc know the notifyid and da of this vring.
> > > > > >>> -    * Not all platforms use dma_alloc_coherent to automatically
> > > > > >>> -    * set up the iommu. In this case the device address (da) will
> > > > > >>> -    * hold the physical address and not the device address.
> > > > > >>> -    */
> > > > > >>> -   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > > > > >>> -   rsc->vring[i].da = dma;
> > > > > >>> +   /* Let the rproc know the notifyid of this vring.*/
> > > > > >>>     rsc->vring[i].notifyid = notifyid;
> > > > > >>>     return 0;
> > > > > >>>  }
> > > > > >>> @@ -392,12 +396,10 @@ int rproc_alloc_vring(struct rproc_vdev
> > > > *rvdev,
> > > > > >> int i)
> > > > > >>>
> > > > > >>>  void rproc_free_vring(struct rproc_vring *rvring)
> > > > > >>>  {
> > > > > >>> -   int size = PAGE_ALIGN(vring_size(rvring->len, rvring->align));
> > > > > >>>     struct rproc *rproc = rvring->rvdev->rproc;
> > > > > >>>     int idx = rvring->rvdev->vring - rvring;
> > > > > >>>     struct fw_rsc_vdev *rsc;
> > > > > >>>
> > > > > >>> -   dma_free_coherent(rproc->dev.parent, size, rvring->va, rvring-
> > > > > >>> dma);
> > > > > >>>     idr_remove(&rproc->notifyids, rvring->notifyid);
> > > > > >>
> > > > > >> Also, I am not sure if FW_RSC_ADDR_ANY semantics were enforced
> > > > > >> previously on the vring da. It was simply overwritten irrespective of
> > > > > >> the value. Now, I am running again into the "bad carveout rsc
> > > > > >> configuration value" due to the iommu_domain_check if
> > > > > >> !FW_RSC_ADDR_ANY.
> > > > > >
> > > > >
> > > > > Just realized that I missed responding on this thread last week.
> > > > >
> > > > > > When are you running into the "bad carveout rsc configuration value"
> > ?
> > > > > > This patch is creating one carveout per vring to rely on generic
> > carveout
> > > > allocator.
> > > > > > Then carveout is retrieved from carveout  list and vring resource table
> > > > information updated.
> > > > > >
> > > > > > If the da of the carveout was fixed in the resource table, it is normal
> > you
> > > > have this error.
> > > > >
> > > > > Yes, and if the vring da value is FW_RSC_ADDR_ANY, then I don't run
> > into
> > > > > this particular error. It's just that the semantics of vring da is open
> > > > > previously, and my above comment being 0 was being used as a reset
> > value
> > > > > as well.
> > > > >
> > > > > > To solve that ST driver is registering one fixed carveout per vring (with
> > the
> > > > right name today)
> > > > >
> > > > > Yeah, we still expect to allocate these dynamically, so there won't be
> > > > > any registration needed.
> > > > >
> > > > > > It is the same discussion as with Patch 1.
> > > > >
> > > > > Right, except that we have a da from RSC_CARVEOUT and a da from
> > vring,
> > > > > and the previous code had some slight differences between the two.
> > The
> > > > > vring da semantics were never set before (value was always being
> > > > > overwritten, also it didn't have a pa field), whereas the remoteproc.h
> > > > > documentation did mention about FW_RSC_ADDR_ANY (without any
> > > > backing
> > > > > implementation previously) for the RSC_CARVEOUT da, with the entry
> > also
> > > > > having a field for pa.
> > > > >
> > > > > If we consider we can't change fixed coprocessor address requests,
> > rproc
> > > > > core should stop its execution.
> > > > > > It is the responsibility of platform driver to register the right memory
> > > > regions.
> > > > > > That's what we discussed with you and Bill in OpenAMP forum.
> > > > > > TI usecase was to have the same DSP firmware with the same
> > resource
> > > > table being able to run on any DSP.
> > > > >
> > > > > Yeah, it only covers one of the usecases/platforms (Keystone 2 DSP
> > > > > platforms). And this only worked on these platforms so far because we
> > > > > only were using internal memories - so there were no RSC_CARVEOUT
> > > > > entries with valid da. Our Davinci DSP is a single instance and we do
> > > > > have a RSC_CARVEOUT there, which fails due to the same Patch 1 logic
> > > > > here as well.
> > > > >
> > > > > > In that case each DSP platform driver has to provide the right memory
> > > > region configuration with the correct pa to da.
> > > > > >
> > > > > >>
> > > > > >> FWIW, the rproc_free_vring was actually using the value 0 when
> > > > resetting.
> > > > > >
> > > > > > It is no more needed as the carveout list is cleared at each stop and
> > > > recreated at each start.
> > > > > > Moreover resource table (and firmware) is reloaded at each
> > coprocessor
> > > > start.
> > > > >
> > > > > Yes, agreed. This is about the semantics of vring da from before (no
> > > > > enforcement to strict enforcement of FW_RSC_ADDR_ANY). The
> > > > overwriting
> > > > > on da field on vrings with the dma address is actually a mistake, which
> > > > > we are trying to proliferate more now. It all comes down to the fact of
> > > > > treating da as dma address when it is not going to be the case on all
> > > > > remoteprocs.
> > > > [Wendy] Are we assuming that the vring da is always predefined.
> > > > But in the Linux kernel side, without IOMMU case, we use
> > > > dma_alloc_coherent() to
> > > > allocate for vring address. In this case, there is no gurantee the
> > > > allocated vrings
> > > > address matches the predefined value.
> > > > Or we assume that only vdev devices of the remoteproc can use
> > > > dma_alloc_coherent()
> > > > but not subdevices?
> > > > Or we can still store the rsc table da pointer. And instead of always
> > > > set da with dma address
> > > > we can pass the dma address to remoteproc and let remoteproc to do
> > the
> > > > pa to da conversion if
> > > > required.
> > >
> > > Hi Wendy,
> > > I pushed a correction patch which check dma address with requested da
> > from resource table.
> > > If no match, a warning is display, but the sequence is not stopped as
> > requested by Suman.
> > > I have a next patch which is adding a rproc_pa_to_da() platform
> > dependent function to concert a pa in da understandable by rproc firmware.
> > > The goal is to answer Suman request, that some rproc can configure an
> > offset in HW to access part of the DRR or an memory.
> > > On another side, I'm looking how to do the same with the carveout
> > registration functions...
> > HI Loic,
> >
> > I agreed that carveout registration functions should allow ANY_ADDRESS
> > for no IOMMU case.
> > In my case, the openamp remoteproc virtio assumes that when the app is
> > running, it assumes the da are all set by master, but the latest
> > upstream delayed the vring da setting
> > when it tries to create the vq, it looks, it looks like the remote
> > should not even start to checking vdev configs until the master set
> > the status to DRIVER_OK.
>
> Yes that's an old discussion I had with Bjorn, when he introduced sub dev. Rpmsg buffer are allocated after rproc start.
> But we agree on the fact master and slave initialization can be fixed and slave should use synchronization flags like
> virtio kick or vdev status before accessing configuration. New implementation is respecting same rule.
> About this issue, I know Arnaud has done a development on OpenAMP side to wait vdev status OK before reading vring da.
Maybe that's virtio without remoteproc implementation, the remoteproc
virtio implementation in OpenAMP assumes
the vring da is already set by master when the application starts.
I am changing the remoteproc virtio backend in OpenAMP not to get the
vring da until status is OK.

Best Regards,
Wendy

>
> Regards,
> Loic
> >
> > Best Regards,
> > Wendy
> > >
> > > Regards,
> > > Loic
> > >
> > >
> > > >
> > > > Thanks,
> > > > Wendy
> > > > >
> > > > > regards
> > > > > Suman
> > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Loic
> > > > > >>
> > > > > >> regards
> > > > > >> Suman
> > > > > >>
> > > > > >>>
> > > > > >>>     /* reset resource entry info */
> > > > > >>> @@ -484,6 +486,7 @@ static int rproc_handle_vdev(struct rproc
> > > > *rproc,
> > > > > >> struct fw_rsc_vdev *rsc,
> > > > > >>>
> > > > > >>>     rvdev->id = rsc->id;
> > > > > >>>     rvdev->rproc = rproc;
> > > > > >>> +   rvdev->index = rproc->nb_vdev++;
> > > > > >>>
> > > > > >>>     /* parse the vrings */
> > > > > >>>     for (i = 0; i < rsc->num_of_vrings; i++) {
> > > > > >>> @@ -528,9 +531,6 @@ void rproc_vdev_release(struct kref *ref)
> > > > > >>>
> > > > > >>>     for (id = 0; id < ARRAY_SIZE(rvdev->vring); id++) {
> > > > > >>>             rvring = &rvdev->vring[id];
> > > > > >>> -           if (!rvring->va)
> > > > > >>> -                   continue;
> > > > > >>> -
> > > > > >>>             rproc_free_vring(rvring);
> > > > > >>>     }
> > > > > >>>
> > > > > >>> @@ -1322,6 +1322,9 @@ static int rproc_fw_boot(struct rproc
> > *rproc,
> > > > > >> const struct firmware *fw)
> > > > > >>>     /* reset max_notifyid */
> > > > > >>>     rproc->max_notifyid = -1;
> > > > > >>>
> > > > > >>> +   /* reset handled vdev */
> > > > > >>> +   rproc->nb_vdev = 0;
> > > > > >>> +
> > > > > >>>     /* handle fw resources which are required to boot rproc */
> > > > > >>>     ret = rproc_handle_resources(rproc, rproc_loading_handlers);
> > > > > >>>     if (ret) {
> > > > > >>> diff --git a/drivers/remoteproc/remoteproc_internal.h
> > > > > >> b/drivers/remoteproc/remoteproc_internal.h
> > > > > >>> index 7570beb..f6cad24 100644
> > > > > >>> --- a/drivers/remoteproc/remoteproc_internal.h
> > > > > >>> +++ b/drivers/remoteproc/remoteproc_internal.h
> > > > > >>> @@ -60,6 +60,8 @@ struct dentry *rproc_create_trace_file(const
> > char
> > > > > >> *name, struct rproc *rproc,
> > > > > >>>  int rproc_elf_load_rsc_table(struct rproc *rproc, const struct
> > firmware
> > > > > >> *fw);
> > > > > >>>  struct resource_table *rproc_elf_find_loaded_rsc_table(struct
> > rproc
> > > > > >> *rproc,
> > > > > >>>                                                    const struct firmware
> > > > > >> *fw);
> > > > > >>> +struct rproc_mem_entry *
> > > > > >>> +rproc_find_carveout_by_name(struct rproc *rproc, const char
> > > > *name,
> > > > > >> ...);
> > > > > >>>
> > > > > >>>  static inline
> > > > > >>>  int rproc_fw_sanity_check(struct rproc *rproc, const struct
> > firmware
> > > > *fw)
> > > > > >>> diff --git a/drivers/remoteproc/remoteproc_virtio.c
> > > > > >> b/drivers/remoteproc/remoteproc_virtio.c
> > > > > >>> index bbecd44..de21f62 100644
> > > > > >>> --- a/drivers/remoteproc/remoteproc_virtio.c
> > > > > >>> +++ b/drivers/remoteproc/remoteproc_virtio.c
> > > > > >>> @@ -76,7 +76,9 @@ static struct virtqueue *rp_find_vq(struct
> > > > > >> virtio_device *vdev,
> > > > > >>>     struct rproc_vdev *rvdev = vdev_to_rvdev(vdev);
> > > > > >>>     struct rproc *rproc = vdev_to_rproc(vdev);
> > > > > >>>     struct device *dev = &rproc->dev;
> > > > > >>> +   struct rproc_mem_entry *mem;
> > > > > >>>     struct rproc_vring *rvring;
> > > > > >>> +   struct fw_rsc_vdev *rsc;
> > > > > >>>     struct virtqueue *vq;
> > > > > >>>     void *addr;
> > > > > >>>     int len, size;
> > > > > >>> @@ -88,8 +90,14 @@ static struct virtqueue *rp_find_vq(struct
> > > > > >> virtio_device *vdev,
> > > > > >>>     if (!name)
> > > > > >>>             return NULL;
> > > > > >>>
> > > > > >>> +   /* Search allocated memory region by name */
> > > > > >>> +   mem = rproc_find_carveout_by_name(rproc,
> > "vdev%dvring%d",
> > > > > >> rvdev->index,
> > > > > >>> +                                     id);
> > > > > >>> +   if (!mem || !mem->va)
> > > > > >>> +           return ERR_PTR(-ENOMEM);
> > > > > >>> +
> > > > > >>>     rvring = &rvdev->vring[id];
> > > > > >>> -   addr = rvring->va;
> > > > > >>> +   addr = mem->va;
> > > > > >>>     len = rvring->len;
> > > > > >>>
> > > > > >>>     /* zero vring */
> > > > > >>> @@ -114,6 +122,10 @@ static struct virtqueue *rp_find_vq(struct
> > > > > >> virtio_device *vdev,
> > > > > >>>     rvring->vq = vq;
> > > > > >>>     vq->priv = rvring;
> > > > > >>>
> > > > > >>> +   /* Update vring in resource table */
> > > > > >>> +   rsc = (void *)rproc->table_ptr + rvdev->rsc_offset;
> > > > > >>> +   rsc->vring[id].da = mem->da;
> > > > > >>> +
> > > > > >>>     return vq;
> > > > > >>>  }
> > > > > >>>
> > > > > >>> diff --git a/include/linux/remoteproc.h
> > b/include/linux/remoteproc.h
> > > > > >>> index 4cdd0c6..6b3a234 100644
> > > > > >>> --- a/include/linux/remoteproc.h
> > > > > >>> +++ b/include/linux/remoteproc.h
> > > > > >>> @@ -453,6 +453,7 @@ struct rproc_dump_segment {
> > > > > >>>   * @table_sz: size of @cached_table
> > > > > >>>   * @has_iommu: flag to indicate if remote processor is behind an
> > > > MMU
> > > > > >>>   * @dump_segments: list of segments in the firmware
> > > > > >>> + * @nb_vdev: number of vdev currently handled by rproc
> > > > > >>>   */
> > > > > >>>  struct rproc {
> > > > > >>>     struct list_head node;
> > > > > >>> @@ -485,6 +486,7 @@ struct rproc {
> > > > > >>>     bool has_iommu;
> > > > > >>>     bool auto_boot;
> > > > > >>>     struct list_head dump_segments;
> > > > > >>> +   int nb_vdev;
> > > > > >>>  };
> > > > > >>>
> > > > > >>>  /**
> > > > > >>> @@ -512,7 +514,6 @@ struct rproc_subdev {
> > > > > >>>  /**
> > > > > >>>   * struct rproc_vring - remoteproc vring state
> > > > > >>>   * @va:    virtual address
> > > > > >>> - * @dma: dma address
> > > > > >>>   * @len: length, in bytes
> > > > > >>>   * @da: device address
> > > > > >>>   * @align: vring alignment
> > > > > >>> @@ -522,7 +523,6 @@ struct rproc_subdev {
> > > > > >>>   */
> > > > > >>>  struct rproc_vring {
> > > > > >>>     void *va;
> > > > > >>> -   dma_addr_t dma;
> > > > > >>>     int len;
> > > > > >>>     u32 da;
> > > > > >>>     u32 align;
> > > > > >>> @@ -541,6 +541,7 @@ struct rproc_vring {
> > > > > >>>   * @vdev: the virio device
> > > > > >>>   * @vring: the vrings for this vdev
> > > > > >>>   * @rsc_offset: offset of the vdev's resource entry
> > > > > >>> + * @index: vdev position versus other vdev declared in resource
> > table
> > > > > >>>   */
> > > > > >>>  struct rproc_vdev {
> > > > > >>>     struct kref refcount;
> > > > > >>> @@ -553,6 +554,7 @@ struct rproc_vdev {
> > > > > >>>     struct virtio_device vdev;
> > > > > >>>     struct rproc_vring vring[RVDEV_NUM_VRINGS];
> > > > > >>>     u32 rsc_offset;
> > > > > >>> +   u32 index;
> > > > > >>>  };
> > > > > >>>
> > > > > >>>  struct rproc *rproc_get_by_phandle(phandle phandle);
> > > > > >>>
> > > > > >
> > > > >

^ permalink raw reply	[flat|nested] 75+ messages in thread

end of thread, other threads:[~2018-12-04 21:24 UTC | newest]

Thread overview: 75+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-27 13:14 [PATCH v4 00/17] remoteproc: add fixed memory region support Loic Pallardy
2018-07-27 13:14 ` [PATCH v4 01/17] remoteproc: configure IOMMU only if device address requested Loic Pallardy
2018-10-23 17:25   ` Suman Anna
2018-10-23 19:40     ` Loic PALLARDY
2018-10-24  3:46       ` Suman Anna
2018-10-24 12:56         ` Loic PALLARDY
2018-10-26  0:46           ` Suman Anna
2018-07-27 13:14 ` [PATCH v4 02/17] remoteproc: add rproc_va_to_pa function Loic Pallardy
2018-10-23 16:50   ` Suman Anna
2018-10-23 19:51     ` Loic PALLARDY
2018-10-24  3:19       ` Suman Anna
2018-10-24 12:58         ` Loic PALLARDY
2018-10-25 22:50           ` Suman Anna
2018-07-27 13:14 ` [PATCH v4 03/17] remoteproc: add release ops in rproc_mem_entry struct Loic Pallardy
2018-10-23 16:53   ` Suman Anna
2018-10-23 20:48     ` Suman Anna
2018-07-27 13:14 ` [PATCH v4 04/17] remoteproc: add name " Loic Pallardy
2018-10-23 17:06   ` Suman Anna
2018-07-27 13:14 ` [PATCH v4 05/17] remoteproc: add helper function to allocate and init " Loic Pallardy
2018-10-23 19:24   ` Suman Anna
2018-07-27 13:14 ` [PATCH v4 06/17] remoteproc: introduce rproc_add_carveout function Loic Pallardy
2018-10-23 17:05   ` Suman Anna
2018-10-23 19:48     ` Loic PALLARDY
2018-07-27 13:14 ` [PATCH v4 07/17] remoteproc: introduce rproc_find_carveout_by_name function Loic Pallardy
2018-10-23 19:28   ` Suman Anna
2018-07-27 13:14 ` [PATCH v4 08/17] remoteproc: add alloc ops in rproc_mem_entry struct Loic Pallardy
2018-10-23 21:20   ` Suman Anna
2018-10-24 16:00     ` Loic PALLARDY
2018-10-25 22:37       ` Suman Anna
2018-07-27 13:14 ` [PATCH v4 09/17] remoteproc: add helper function to allocate rproc_mem_entry from reserved memory Loic Pallardy
2018-07-27 13:14 ` [PATCH v4 10/17] remoteproc: add helper function to check carveout device address Loic Pallardy
2018-10-23 22:14   ` Suman Anna
2018-10-24 15:24     ` Loic PALLARDY
2018-10-25 22:50       ` Suman Anna
2018-07-27 13:14 ` [PATCH v4 11/17] remoteproc: modify rproc_handle_carveout to support pre-registered region Loic Pallardy
2018-07-27 13:14 ` [PATCH v4 12/17] remoteproc: modify vring allocation to rely on centralized carveout allocator Loic Pallardy
2018-10-10  5:32   ` Bjorn Andersson
2018-10-10 18:58     ` Loic PALLARDY
2018-10-15  6:40       ` Bjorn Andersson
2018-10-23 23:24         ` Suman Anna
2018-10-24  0:14   ` Suman Anna
2018-10-24 15:14     ` Loic PALLARDY
2018-10-29 20:17       ` Suman Anna
2018-12-04 17:56         ` Wendy Liang
2018-12-04 18:04           ` Loic PALLARDY
2018-12-04 18:58             ` Wendy Liang
2018-12-04 19:57               ` Loic PALLARDY
2018-12-04 21:24                 ` Wendy Liang
2018-07-27 13:14 ` [PATCH v4 13/17] remoteproc: create vdev subdevice with specific dma memory pool Loic Pallardy
2018-09-27 17:17   ` Wendy Liang
2018-09-27 19:22     ` Loic PALLARDY
2018-09-27 20:18       ` Wendy Liang
2018-10-24  1:22         ` Suman Anna
2018-10-24  1:48           ` Suman Anna
2018-10-24 12:42             ` Loic PALLARDY
2018-10-25 22:06               ` Suman Anna
2018-10-24 12:40           ` Loic PALLARDY
2018-10-25 20:16             ` Suman Anna
2018-10-10  5:58   ` Bjorn Andersson
2018-10-10 19:17     ` Loic PALLARDY
2018-10-24  1:27       ` Suman Anna
2018-07-27 13:14 ` [PATCH v4 14/17] remoteproc: keystone: declare reserved memory region for vdev device Loic Pallardy
2018-07-27 13:14 ` [PATCH v4 15/17] remoteproc: da8xx: " Loic Pallardy
2018-10-24  2:57   ` Suman Anna
2018-10-24 13:19     ` Loic PALLARDY
2018-10-25 22:11       ` Suman Anna
2018-07-27 13:14 ` [PATCH v4 16/17] remoteproc: st: add reserved memory support Loic Pallardy
2018-10-24  3:01   ` Suman Anna
2018-10-24 12:37     ` Loic PALLARDY
2018-07-27 13:14 ` [PATCH v4 17/17] rpmsg: virtio: allocate buffer from parent Loic Pallardy
2018-09-28  7:56   ` Anup Patel
2018-09-21  6:04 ` [PATCH v4 00/17] remoteproc: add fixed memory region support Anup Patel
2018-09-26 16:00   ` Loic PALLARDY
2018-09-28  7:54     ` Anup Patel
2018-10-23 16:42 ` Suman Anna

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).