[PATCH v4 0/5] media: add v4l2_pipeline_stream_{enable,disable}

[PATCH v4 0/5] media: add v4l2_pipeline_stream_{enable,disable}

[Dafna Hirschfeld]

Hi,
This is v4 of the patchset that was sent by Helen Koike.

Media drivers need to iterate through the pipeline and call .s_stream()
callbacks in the subdevices.

Instead of repeating code, add helpers for this.

These helpers will go walk through the pipeline only visiting entities
that participates in the stream, i.e. it follows links from sink to source
(and not the opposite).
For example, in a topology like this https://bit.ly/3b2MxjI
calling v4l2_pipeline_stream_enable() from rkisp1_mainpath won't call
.s_stream(true) for rkisp1_resizer_selfpath.

stream_count variable was added in v4l2_subdevice to handle nested calls
to the helpers.
This is useful when the driver allows streaming from more then one
capture device sharing subdevices.

This patchset was tested on rkisp1 and vimc drivers.

Other cleanup might be possible (but I won't add in this patchset as I
don't have the hw to test):
	https://git.linuxtv.org/media_tree.git/tree/drivers/media/platform/qcom/camss/camss-video.c#n430
	https://git.linuxtv.org/media_tree.git/tree/drivers/media/platform/omap3isp/isp.c#n697
	https://git.linuxtv.org/media_tree.git/tree/drivers/media/platform/stm32/stm32-dcmi.c#n680
	https://git.linuxtv.org/media_tree.git/tree/drivers/media/platform/xilinx/xilinx-dma.c#n97

Testing:
--------

SEN_DEV=/dev/v4l-subdev3
ISP_DEV=/dev/v4l-subdev0
RSZ_SP_DEV=/dev/v4l-subdev2
RSZ_MP_DEV=/dev/v4l-subdev1
CAP_SP_DEV=/dev/video1
CAP_MP_DEV=/dev/video0

WIDTH=1920
HEIGHT=1080
RAW_CODE=SRGGB10_1X10
YUV_CODE=YUYV8_2X8

v4l2-ctl --set-subdev-fmt pad=0,width=$WIDTH,height=$HEIGHT,code=$RAW_CODE -d $SEN_DEV

v4l2-ctl --set-subdev-fmt pad=0,width=$WIDTH,height=$HEIGHT,code=$RAW_CODE -d $ISP_DEV
v4l2-ctl --set-subdev-selection pad=0,target=crop,top=0,left=0,width=$WIDTH,height=$HEIGHT -d $ISP_DEV

v4l2-ctl --set-subdev-selection pad=2,target=crop,top=0,left=0,width=$WIDTH,height=$HEIGHT -d $ISP_DEV
v4l2-ctl --set-subdev-fmt pad=2,width=$WIDTH,height=$HEIGHT,code=$YUV_CODE -d $ISP_DEV

v4l2-ctl --set-subdev-fmt pad=0,width=$WIDTH,height=$HEIGHT,code=$YUV_CODE -d $RSZ_SP_DEV
v4l2-ctl --set-subdev-fmt pad=1,width=$WIDTH,height=$HEIGHT,code=$YUV_CODE -d $RSZ_SP_DEV

v4l2-ctl --set-subdev-selection pad=0,target=crop,top=0,left=0,width=$WIDTH,height=$HEIGHT -d $RSZ_SP_DEV

v4l2-ctl --set-subdev-fmt pad=0,width=$WIDTH,height=$HEIGHT,code=$YUV_CODE -d $RSZ_MP_DEV
v4l2-ctl --set-subdev-fmt pad=1,width=$WIDTH,height=$HEIGHT,code=$YUV_CODE -d $RSZ_MP_DEV

v4l2-ctl --set-subdev-selection pad=0,target=crop,top=0,left=0,width=$WIDTH,height=$HEIGHT -d $RSZ_MP_DEV

v4l2-ctl -v width=$WIDTH,height=$HEIGHT,pixelformat=NV12 -d $CAP_SP_DEV
v4l2-ctl -v width=$WIDTH,height=$HEIGHT,pixelformat=NV12 -d $CAP_MP_DEV

sleep 1

v4l2-ctl --stream-mmap --stream-count=100 -d $CAP_SP_DEV --stream-to=/tmp/test_sp.raw &
v4l2-ctl --stream-mmap --stream-count=100 -d $CAP_MP_DEV --stream-to=/tmp/test_mp.raw &

wait
echo "Completed"


Changes in v4:
==============
patch 1: fix coding style issues

patch 2:
- in function v4l2_pipeline_subdevs_get, use a local media_graph to walk on the topology so a lock is not needed
and also the pipe parameter is not needed.
- adding a function v4l2_subdevs_stream_disable to avoid code duplication
- change v4l2_pipeline_stream_disable to return an error code if failed
- don't add a new field to subdevice "stream_counter" when calling s_stream, since this counter is updated only in
the helper functions, and might be confusing that it is generally not an indication of the number of calls to s_stream.
Also, except of rkisp1, and vimc, it seems that the other drivers that might use the new helpers don't use a counter.

new added - patch 3: the call to media_pipeline_start should be called before calling s_stream on the subdevices in order
to validate the links and prevents them from changing, this patch fixes it.

patch 4: (use the helpers in rkisp1). The helpers now don't have a counter for the number of calls to s_stream, so rkisp1
should check if the other capture is streaming and in that case call s_stream only for its resizer.

patch 5: - (use the helpers in vimc)
- test the return value of v4l2_pipeline_stream_disable
- the call to the helerps now doesn't need the pipe parameter.

Overview of patches in V3:
--------------------------

Patch 1/5 adds a new iterator function in media-controller to follow links from sink to
source only.

Patch 2/5 adds the helpers in v4l2-common.c,

Patch 3/5 calles media_pipeline_start before calling s_stream on the subdevices

Patch 4/5 cleanup rkisp1 driver to use the helpers.

Patch 5/5 cleanup vimc driver to use the helpers.

Changes in V3:
====================
Following up Niklas' comments in V2 https://patchwork.kernel.org/patch/11473681/#23270823

* I removed the limitation in topologies with entities with multiple enabled
links to its sink pads in the topology.
Now it enables all subdevs in the pipeline that have an enabled link going
from sink to source while walking from the video device, so it can be
also useful for rcar-vin driver.

To implement this, I added back in the series the patch from v1:
    "media: mc-entity.c: add media_graph_walk_next_stream()"

* "size" was renamed to "max_size" in function v4l2_pipeline_subdevs_get()
to reflect the maximum number of elements that can fit in the subdevs array,
with proper documentation.

* v4l2_pipeline_subdevs_get() returns a negative number for error, instead
of returning 0 and printing a warning.

* I also add if defined(CONFIG_MEDIA_CONTROLLER) around helpers to avoid
compiling errors.

Overview of patches in V3:
--------------------------

Patch 1/4 adds a new iterator function in media-controller to follow links from sink to
source only.

Patch 2/4 adds the helpers in v4l2-common.c, allowing nested calls by
adding stream_count in the subdevice struct.

Patch 3/4 cleanup rkisp1 driver to use the helpers.

Patch 4/4 cleanup vimc driver to use the helpers.

Changes in V2:
====================
The first version was calling the s_stream() callbacks from sensor to
capture.

This was generating errors in the Scarlet Chromebook, when the sensor
was being enabled before the ISP.

It make sense to enable subdevices from capture to sensor instead (which
is what most drivers do already).

This v2 drops the changes from mc-entity.c, and re-implement helpers in
v4l2-common.c

Overview of patches in V2:
--------------------------

Path 1/3 adds the helpers in v4l2-common.c, allowing nested calls by
adding stream_count in the subdevice struct.

Patch 2/3 cleanup rkisp1 driver to use the helpers.

Patch 3/3 cleanup vimc driver to use the helpers.

Dafna Hirschfeld (1):
  media: staging: rkisp1: validate links before powering and streaming

Helen Koike (4):
  media: mc-entity.c: add media_graph_walk_next_stream()
  media: v4l2-common: add helper functions to call s_stream() callbacks
  media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable}
    helpers
  media: vimc: use v4l2_pipeline_stream_{enable,disable} helpers

 drivers/media/mc/mc-entity.c                  |  34 ++++-
 .../media/test-drivers/vimc/vimc-capture.c    |  31 +++--
 .../media/test-drivers/vimc/vimc-streamer.c   |  49 +------
 drivers/media/v4l2-core/v4l2-common.c         |  99 ++++++++++++++
 drivers/staging/media/rkisp1/rkisp1-capture.c | 125 ++++++------------
 include/media/media-entity.h                  |  15 +++
 include/media/v4l2-common.h                   |  39 ++++++
 7 files changed, 253 insertions(+), 139 deletions(-)

-- 
2.17.1

[PATCH v4 1/5] media: mc-entity.c: add media_graph_walk_next_stream()

[Dafna Hirschfeld]

From: Helen Koike <helen.koike@collabora.com>

Add media_graph_walk_next_stream() function to follow links only from
sink to source (not the opposite) to allow iteration only through the
entities participating in a given stream.

This is useful to allow calling .s_stream() callback only in the
subdevices that requires to be enabled/disabled, and avoid calling this
callback when not required.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
[Dafna: fixing coding style issues]
Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
---
 drivers/media/mc/mc-entity.c | 34 +++++++++++++++++++++++++++++++---
 include/media/media-entity.h | 15 +++++++++++++++
 2 files changed, 46 insertions(+), 3 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 12b45e669bcc..555f9dd9f7f2 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -228,6 +228,11 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * Graph traversal
  */
 
+enum media_graph_walk_type {
+	MEDIA_GRAPH_WALK_CONNECTED_NODES,
+	MEDIA_GRAPH_WALK_STREAM_NODES,
+};
+
 static struct media_entity *
 media_entity_other(struct media_entity *entity, struct media_link *link)
 {
@@ -305,7 +310,8 @@ void media_graph_walk_start(struct media_graph *graph,
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_start);
 
-static void media_graph_walk_iter(struct media_graph *graph)
+static void media_graph_walk_iter(struct media_graph *graph,
+				  enum media_graph_walk_type type)
 {
 	struct media_entity *entity = stack_top(graph);
 	struct media_link *link;
@@ -326,6 +332,15 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	/* Get the entity in the other end of the link . */
 	next = media_entity_other(entity, link);
 
+	if (type == MEDIA_GRAPH_WALK_STREAM_NODES &&
+	    next == link->sink->entity) {
+		link_top(graph) = link_top(graph)->next;
+		dev_dbg(entity->graph_obj.mdev->dev,
+			"walk: skipping '%s' (outside of the stream path)\n",
+			link->sink->entity->name);
+		return;
+	}
+
 	/* Has the entity already been visited? */
 	if (media_entity_enum_test_and_set(&graph->ent_enum, next)) {
 		link_top(graph) = link_top(graph)->next;
@@ -342,7 +357,9 @@ static void media_graph_walk_iter(struct media_graph *graph)
 		next->name);
 }
 
-struct media_entity *media_graph_walk_next(struct media_graph *graph)
+static struct media_entity *
+__media_graph_walk_next(struct media_graph *graph,
+			enum media_graph_walk_type type)
 {
 	struct media_entity *entity;
 
@@ -355,7 +372,7 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 	 * found.
 	 */
 	while (link_top(graph) != &stack_top(graph)->links)
-		media_graph_walk_iter(graph);
+		media_graph_walk_iter(graph, type);
 
 	entity = stack_pop(graph);
 	dev_dbg(entity->graph_obj.mdev->dev,
@@ -363,8 +380,19 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 
 	return entity;
 }
+
+struct media_entity *media_graph_walk_next(struct media_graph *graph)
+{
+	return __media_graph_walk_next(graph, MEDIA_GRAPH_WALK_CONNECTED_NODES);
+}
 EXPORT_SYMBOL_GPL(media_graph_walk_next);
 
+struct media_entity *media_graph_walk_next_stream(struct media_graph *graph)
+{
+	return __media_graph_walk_next(graph, MEDIA_GRAPH_WALK_STREAM_NODES);
+}
+EXPORT_SYMBOL_GPL(media_graph_walk_next_stream);
+
 int media_entity_get_fwnode_pad(struct media_entity *entity,
 				struct fwnode_handle *fwnode,
 				unsigned long direction_flags)
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index cde80ad029b7..9035a36e9442 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -928,6 +928,21 @@ void media_graph_walk_start(struct media_graph *graph,
  */
 struct media_entity *media_graph_walk_next(struct media_graph *graph);
 
+/**
+ * media_graph_walk_next_stream - Get the next entity in the graph
+ * @graph: Media graph structure
+ *
+ * Perform a depth-first traversal of the given media entities graph only
+ * following links from sink to source (and not the opposite).
+ *
+ * The graph structure must have been previously initialized with a call to
+ * media_graph_walk_start().
+ *
+ * Return: returns the next entity in the graph in the stream path
+ * or %NULL if the whole stream path have been traversed.
+ */
+struct media_entity *media_graph_walk_next_stream(struct media_graph *graph);
+
 /**
  * media_pipeline_start - Mark a pipeline as streaming
  * @entity: Starting entity
-- 
2.17.1

[PATCH v4 2/5] media: v4l2-common: add helper functions to call s_stream() callbacks

[Dafna Hirschfeld]

From: Helen Koike <helen.koike@collabora.com>

Add v4l2_pipeline_stream_{enable,disable} helper functions to iterate
through the subdevices in a given stream (i.e following links from sink
to source) and call .s_stream() callback.

If .s_stream(true) fails, call .s_stream(false) in the reverse order.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
---
 drivers/media/v4l2-core/v4l2-common.c | 99 +++++++++++++++++++++++++++
 include/media/v4l2-common.h           | 39 +++++++++++
 2 files changed, 138 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-common.c b/drivers/media/v4l2-core/v4l2-common.c
index 9e8eb45a5b03..734db2bf5ca9 100644
--- a/drivers/media/v4l2-core/v4l2-common.c
+++ b/drivers/media/v4l2-core/v4l2-common.c
@@ -442,3 +442,102 @@ int v4l2_fill_pixfmt(struct v4l2_pix_format *pixfmt, u32 pixelformat,
 	return 0;
 }
 EXPORT_SYMBOL_GPL(v4l2_fill_pixfmt);
+
+#if defined(CONFIG_MEDIA_CONTROLLER)
+
+/*
+ * v4l2_pipeline_subdevs_get - Assemble a list of subdevices in a pipeline
+ * @subdevs: the array to be filled.
+ * @max_size: max number of elements that can fit in subdevs array.
+ *
+ * Walk from a video node, following links from sink to source and fill the
+ * array with subdevices in the path.
+ * The walker performs a depth-first traversal, which means that, in a topology
+ * sd1->sd2->sd3->vdev, sd1 will be the first element placed in the array.
+ *
+ * Return the number of subdevices filled in the array.
+ */
+static int v4l2_pipeline_subdevs_get(struct video_device *vdev,
+				     struct v4l2_subdev **subdevs,
+				     unsigned int max_size)
+{
+	struct media_entity *entity = &vdev->entity;
+	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_graph graph;
+	int idx = 0;
+	int ret;
+
+	ret = media_graph_walk_init(&graph, mdev);
+	if (ret)
+		return ret;
+
+	media_graph_walk_start(&graph, entity);
+
+	while ((entity = media_graph_walk_next_stream(&graph))) {
+		if (!is_media_entity_v4l2_subdev(entity))
+			continue;
+		if (WARN_ON(idx >= max_size))
+			return -EINVAL;
+		subdevs[idx++] = media_entity_to_v4l2_subdev(entity);
+	}
+
+	media_graph_walk_cleanup(&graph);
+
+	return idx;
+}
+
+static void v4l2_subdevs_stream_disable(struct v4l2_subdev **subdevs, int size)
+{
+	int i;
+
+	for (i = 0; i < size; i++) {
+		struct v4l2_subdev *sd = subdevs[i];
+
+		dev_dbg(sd->dev,
+			"disabling stream for '%s'\n", sd->entity.name);
+		v4l2_subdev_call(sd, video, s_stream, false);
+	}
+}
+
+__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev)
+{
+	struct media_device *mdev = vdev->entity.graph_obj.mdev;
+	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
+	struct v4l2_subdev *sd;
+	int i, size, ret;
+
+	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
+	if (size <= 0)
+		return size;
+
+	/* Call s_stream() in reverse order to enable sensors last */
+	for (i = size - 1; i >= 0; i--) {
+		sd = subdevs[i];
+		dev_dbg(mdev->dev,
+			"enabling stream for '%s'\n", sd->entity.name);
+		ret = v4l2_subdev_call(sd, video, s_stream, true);
+		if (ret && ret != -ENOIOCTLCMD) {
+			v4l2_subdevs_stream_disable(subdevs + i + 1,
+						    size - i - 1);
+			return ret;
+		}
+	}
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_enable);
+
+int v4l2_pipeline_stream_disable(struct video_device *vdev)
+{
+	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
+	int size;
+
+	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
+	if (size < 0)
+		return size;
+
+	v4l2_subdevs_stream_disable(subdevs, size);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_disable);
+
+#endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-common.h b/include/media/v4l2-common.h
index 150ee16ebd81..a278bcf3c5bc 100644
--- a/include/media/v4l2-common.h
+++ b/include/media/v4l2-common.h
@@ -539,4 +539,43 @@ static inline void v4l2_buffer_set_timestamp(struct v4l2_buffer *buf,
 	buf->timestamp.tv_usec = ts.tv_nsec / NSEC_PER_USEC;
 }
 
+#if defined(CONFIG_MEDIA_CONTROLLER)
+
+/**
+ * v4l2_pipeline_stream_enable - Call .s_stream(true) callbacks in the stream
+ * @vdev: Starting video device.
+ *
+ * Call .s_stream(true) callback in all the subdevices participating in the
+ * stream, i.e. following links from sink to source.
+ *
+ * .s_stream(true) is also called from sink to source, i.e. in a topology
+ * sd1->sd2->sd3->vdev, .s_stream(true) of sd3 is called first.
+ *
+ * Calls to this function can be nested, in which case the same number of
+ * v4l2_pipeline_stream_disable() calls will be required to disable streaming
+ * through subdevices in the pipeline.
+ * The  pipeline pointer must be identical for all nested calls to
+ * v4l2_pipeline_stream_enable().
+ */
+__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev);
+
+/**
+ * v4l2_pipeline_stream_disable - Call .s_stream(false) callbacks in the stream
+ * @vdev: Starting video device.
+ *
+ * Call .s_stream(false) callback in all the subdevices participating in the
+ * stream, i.e. following links from sink to source.
+ *
+ * s_stream(false) is called in a reverse order from
+ * v4l2_pipeline_stream_enable(), i.e. in a topology sd1->sd2->sd3->vdev,
+ * .s_stream(false) of sd1 is called first.
+ *
+ * If multiple calls to v4l2_pipeline_stream_enable() have been made, the same
+ * number of calls to this function are required to disable streaming through
+ * subdevices in the pipeline.
+ */
+int v4l2_pipeline_stream_disable(struct video_device *vdev);
+
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
 #endif /* V4L2_COMMON_H_ */
-- 
2.17.1

[PATCH v4 3/5] media: staging: rkisp1: validate links before powering and streaming

[Dafna Hirschfeld]

In function rkisp1_vb2_start_streaming, the call to
media_pipeline_start should be the first thing in order
to validate the links and prevents their state from being modified
before power up and streaming.

Adjust stop streaming to the same logic, call media_pipeline_stop
after we disable streaming on the entities in the topology.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
---
 drivers/staging/media/rkisp1/rkisp1-capture.c | 21 ++++++++++---------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
index f69235f82c45..008c070373f8 100644
--- a/drivers/staging/media/rkisp1/rkisp1-capture.c
+++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
@@ -898,7 +898,6 @@ static void rkisp1_vb2_stop_streaming(struct vb2_queue *queue)
 	mutex_lock(&cap->rkisp1->stream_lock);
 
 	rkisp1_stream_stop(cap);
-	media_pipeline_stop(&node->vdev.entity);
 	ret = rkisp1_pipeline_sink_walk(&node->vdev.entity, NULL,
 					rkisp1_pipeline_disable_cb);
 	if (ret)
@@ -914,6 +913,8 @@ static void rkisp1_vb2_stop_streaming(struct vb2_queue *queue)
 
 	rkisp1_dummy_buf_destroy(cap);
 
+	media_pipeline_stop(&node->vdev.entity);
+
 	mutex_unlock(&cap->rkisp1->stream_lock);
 }
 
@@ -961,9 +962,15 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 
 	mutex_lock(&cap->rkisp1->stream_lock);
 
+	ret = media_pipeline_start(entity, &cap->rkisp1->pipe);
+	if (ret) {
+		dev_err(cap->rkisp1->dev, "start pipeline failed %d\n", ret);
+		goto err_ret_buffers;
+	}
+
 	ret = rkisp1_dummy_buf_create(cap);
 	if (ret)
-		goto err_ret_buffers;
+		goto err_pipeline_stop;
 
 	ret = pm_runtime_get_sync(cap->rkisp1->dev);
 	if (ret < 0) {
@@ -984,18 +991,10 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 	if (ret)
 		goto err_stop_stream;
 
-	ret = media_pipeline_start(entity, &cap->rkisp1->pipe);
-	if (ret) {
-		dev_err(cap->rkisp1->dev, "start pipeline failed %d\n", ret);
-		goto err_pipe_disable;
-	}
-
 	mutex_unlock(&cap->rkisp1->stream_lock);
 
 	return 0;
 
-err_pipe_disable:
-	rkisp1_pipeline_sink_walk(entity, NULL, rkisp1_pipeline_disable_cb);
 err_stop_stream:
 	rkisp1_stream_stop(cap);
 	v4l2_pipeline_pm_put(entity);
@@ -1003,6 +1002,8 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 	pm_runtime_put(cap->rkisp1->dev);
 err_destroy_dummy:
 	rkisp1_dummy_buf_destroy(cap);
+err_pipeline_stop:
+	media_pipeline_stop(entity);
 err_ret_buffers:
 	rkisp1_return_all_buffers(cap, VB2_BUF_STATE_QUEUED);
 	mutex_unlock(&cap->rkisp1->stream_lock);
-- 
2.17.1

[PATCH v4 4/5] media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable} helpers

[Dafna Hirschfeld]

From: Helen Koike <helen.koike@collabora.com>

Use v4l2_pipeline_stream_{enable,disable} to call .s_stream()
subdevice callbacks through the pipeline.
Those helpers are called only if the other capture is not streaming.

If the other capture is streaming then he already did that for us
so we call s_stream only on the resizer that is connected to the
capture node.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
---
 drivers/staging/media/rkisp1/rkisp1-capture.c | 104 ++++++------------
 1 file changed, 32 insertions(+), 72 deletions(-)

diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
index 008c070373f8..d53a8549da77 100644
--- a/drivers/staging/media/rkisp1/rkisp1-capture.c
+++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
@@ -806,71 +806,6 @@ static void rkisp1_return_all_buffers(struct rkisp1_capture *cap,
 	spin_unlock_irqrestore(&cap->buf.lock, flags);
 }
 
-/*
- * rkisp1_pipeline_sink_walk - Walk through the pipeline and call cb
- * @from: entity at which to start pipeline walk
- * @until: entity at which to stop pipeline walk
- *
- * Walk the entities chain starting at the pipeline video node and stop
- * all subdevices in the chain.
- *
- * If the until argument isn't NULL, stop the pipeline walk when reaching the
- * until entity. This is used to disable a partially started pipeline due to a
- * subdev start error.
- */
-static int rkisp1_pipeline_sink_walk(struct media_entity *from,
-				     struct media_entity *until,
-				     int (*cb)(struct media_entity *from,
-					       struct media_entity *curr))
-{
-	struct media_entity *entity = from;
-	struct media_pad *pad;
-	unsigned int i;
-	int ret;
-
-	while (1) {
-		pad = NULL;
-		/* Find remote source pad */
-		for (i = 0; i < entity->num_pads; i++) {
-			struct media_pad *spad = &entity->pads[i];
-
-			if (!(spad->flags & MEDIA_PAD_FL_SINK))
-				continue;
-			pad = media_entity_remote_pad(spad);
-			if (pad && is_media_entity_v4l2_subdev(pad->entity))
-				break;
-		}
-		if (!pad || !is_media_entity_v4l2_subdev(pad->entity))
-			break;
-
-		entity = pad->entity;
-		if (entity == until)
-			break;
-
-		ret = cb(from, entity);
-		if (ret)
-			return ret;
-	}
-
-	return 0;
-}
-
-static int rkisp1_pipeline_disable_cb(struct media_entity *from,
-				      struct media_entity *curr)
-{
-	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(curr);
-
-	return v4l2_subdev_call(sd, video, s_stream, false);
-}
-
-static int rkisp1_pipeline_enable_cb(struct media_entity *from,
-				     struct media_entity *curr)
-{
-	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(curr);
-
-	return v4l2_subdev_call(sd, video, s_stream, true);
-}
-
 static void rkisp1_stream_stop(struct rkisp1_capture *cap)
 {
 	int ret;
@@ -888,6 +823,36 @@ static void rkisp1_stream_stop(struct rkisp1_capture *cap)
 	}
 }
 
+static int rkisp1_s_stream_subdev(struct rkisp1_capture *cap, int enable)
+{
+	struct rkisp1_device *rkisp1 = cap->rkisp1;
+	struct rkisp1_capture *other = &rkisp1->capture_devs[cap->id ^ 1];
+	int ret;
+
+	/*
+	 * if the other capture is already streaming then we only need to
+	 * call s_stream of our reszier
+	 */
+	if (other->is_streaming) {
+		struct v4l2_subdev *rsz_sd  = &rkisp1->resizer_devs[cap->id].sd;
+
+		ret = v4l2_subdev_call(rsz_sd, video, s_stream, enable);
+		if (ret && ret != -ENOIOCTLCMD)
+			dev_err(rkisp1->dev,
+				"stream %s resizer '%s' failed (%d)\n",
+				enable ? "on" : "off", rsz_sd->name, ret);
+	} else {
+		if (enable)
+			ret = v4l2_pipeline_stream_enable(&cap->vnode.vdev);
+		else
+			ret = v4l2_pipeline_stream_disable(&cap->vnode.vdev);
+		if (ret < 0)
+			dev_err(rkisp1->dev, "stream subdevs %s failed %d\n",
+				enable ? "on" : "off", ret);
+	}
+	return ret;
+}
+
 static void rkisp1_vb2_stop_streaming(struct vb2_queue *queue)
 {
 	struct rkisp1_capture *cap = queue->drv_priv;
@@ -898,11 +863,7 @@ static void rkisp1_vb2_stop_streaming(struct vb2_queue *queue)
 	mutex_lock(&cap->rkisp1->stream_lock);
 
 	rkisp1_stream_stop(cap);
-	ret = rkisp1_pipeline_sink_walk(&node->vdev.entity, NULL,
-					rkisp1_pipeline_disable_cb);
-	if (ret)
-		dev_err(rkisp1->dev,
-			"pipeline stream-off failed error:%d\n", ret);
+	rkisp1_s_stream_subdev(cap, 0);
 
 	rkisp1_return_all_buffers(cap, VB2_BUF_STATE_ERROR);
 
@@ -986,8 +947,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 	rkisp1_stream_start(cap);
 
 	/* start sub-devices */
-	ret = rkisp1_pipeline_sink_walk(entity, NULL,
-					rkisp1_pipeline_enable_cb);
+	ret = rkisp1_s_stream_subdev(cap, 1);
 	if (ret)
 		goto err_stop_stream;
 
-- 
2.17.1

[PATCH v4 5/5] media: vimc: use v4l2_pipeline_stream_{enable,disable} helpers

[Dafna Hirschfeld]

From: Helen Koike <helen.koike@collabora.com>

Use v4l2_pipeline_stream_{enable,disable} to call .s_stream() subdevice
callbacks through the pipeline.

Tested streaming works with:

media-ctl -d /dev/media0 -V '"Sensor A":0[fmt:SBGGR8_1X8/640x480]'
media-ctl -d /dev/media0 -V '"Debayer A":0[fmt:SBGGR8_1X8/640x480]'
media-ctl -d /dev/media0 -V '"Sensor B":0[fmt:SBGGR8_1X8/640x480]'
media-ctl -d /dev/media0 -V '"Debayer B":0[fmt:SBGGR8_1X8/640x480]'
media-ctl -d /dev/media0 -V '"Scaler":0[fmt:RGB888_1X24/640x480]'
media-ctl -d /dev/media0 -V '"Scaler":0[crop:(100,50)/400x150]'
media-ctl -d /dev/media0 -V '"Scaler":1[fmt:RGB888_1X24/1920x1440]'
v4l2-ctl -d /dev/video2 -v width=1200,height=450
v4l2-ctl -d /dev/video0 -v pixelformat=BA81
v4l2-ctl -d /dev/video1 -v pixelformat=BA81
v4l2-ctl --stream-mmap --stream-count=10 -d /dev/video2 --stream-to=/tmp/test.raw

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
---
 .../media/test-drivers/vimc/vimc-capture.c    | 31 ++++++++----
 .../media/test-drivers/vimc/vimc-streamer.c   | 49 ++-----------------
 2 files changed, 26 insertions(+), 54 deletions(-)

diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c
index c63496b17b9a..36ba664db092 100644
--- a/drivers/media/test-drivers/vimc/vimc-capture.c
+++ b/drivers/media/test-drivers/vimc/vimc-capture.c
@@ -245,21 +245,27 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	vcap->sequence = 0;
 
-	/* Start the media pipeline */
 	ret = media_pipeline_start(entity, &vcap->stream.pipe);
-	if (ret) {
-		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
-		return ret;
-	}
+	if (ret)
+		goto err_return_all_buffers;
+
+	ret = v4l2_pipeline_stream_enable(&vcap->vdev);
+	if (ret)
+		goto err_stop_media_pipe;
 
 	ret = vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 1);
-	if (ret) {
-		media_pipeline_stop(entity);
-		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
-		return ret;
-	}
+	if (ret)
+		goto err_stop_stream;
 
 	return 0;
+
+err_stop_stream:
+	v4l2_pipeline_stream_disable(&vcap->vdev);
+err_stop_media_pipe:
+	media_pipeline_stop(entity);
+err_return_all_buffers:
+	vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
+	return ret;
 }
 
 /*
@@ -269,9 +275,14 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
 static void vimc_cap_stop_streaming(struct vb2_queue *vq)
 {
 	struct vimc_cap_device *vcap = vb2_get_drv_priv(vq);
+	int ret;
 
 	vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 0);
 
+	ret = v4l2_pipeline_stream_disable(&vcap->vdev);
+	if (ret)
+		dev_err(vcap->ved.dev, "stream disable failed: %d\n", ret);
+
 	/* Stop the media pipeline */
 	media_pipeline_stop(&vcap->vdev.entity);
 
diff --git a/drivers/media/test-drivers/vimc/vimc-streamer.c b/drivers/media/test-drivers/vimc/vimc-streamer.c
index 65feb3c596db..c0085f4695c2 100644
--- a/drivers/media/test-drivers/vimc/vimc-streamer.c
+++ b/drivers/media/test-drivers/vimc/vimc-streamer.c
@@ -36,33 +36,6 @@ static struct media_entity *vimc_get_source_entity(struct media_entity *ent)
 	return NULL;
 }
 
-/**
- * vimc_streamer_pipeline_terminate - Disable stream in all ved in stream
- *
- * @stream: the pointer to the stream structure with the pipeline to be
- *	    disabled.
- *
- * Calls s_stream to disable the stream in each entity of the pipeline
- *
- */
-static void vimc_streamer_pipeline_terminate(struct vimc_stream *stream)
-{
-	struct vimc_ent_device *ved;
-	struct v4l2_subdev *sd;
-
-	while (stream->pipe_size) {
-		stream->pipe_size--;
-		ved = stream->ved_pipeline[stream->pipe_size];
-		stream->ved_pipeline[stream->pipe_size] = NULL;
-
-		if (!is_media_entity_v4l2_subdev(ved->ent))
-			continue;
-
-		sd = media_entity_to_v4l2_subdev(ved->ent);
-		v4l2_subdev_call(sd, video, s_stream, 0);
-	}
-}
-
 /**
  * vimc_streamer_pipeline_init - Initializes the stream structure
  *
@@ -82,27 +55,15 @@ static int vimc_streamer_pipeline_init(struct vimc_stream *stream,
 	struct media_entity *entity;
 	struct video_device *vdev;
 	struct v4l2_subdev *sd;
-	int ret = 0;
 
 	stream->pipe_size = 0;
 	while (stream->pipe_size < VIMC_STREAMER_PIPELINE_MAX_SIZE) {
 		if (!ved) {
-			vimc_streamer_pipeline_terminate(stream);
+			stream->pipe_size = 0;
 			return -EINVAL;
 		}
 		stream->ved_pipeline[stream->pipe_size++] = ved;
 
-		if (is_media_entity_v4l2_subdev(ved->ent)) {
-			sd = media_entity_to_v4l2_subdev(ved->ent);
-			ret = v4l2_subdev_call(sd, video, s_stream, 1);
-			if (ret && ret != -ENOIOCTLCMD) {
-				dev_err(ved->dev, "subdev_call error %s\n",
-					ved->ent->name);
-				vimc_streamer_pipeline_terminate(stream);
-				return ret;
-			}
-		}
-
 		entity = vimc_get_source_entity(ved->ent);
 		/* Check if the end of the pipeline was reached */
 		if (!entity) {
@@ -111,7 +72,7 @@ static int vimc_streamer_pipeline_init(struct vimc_stream *stream,
 				dev_err(ved->dev,
 					"first entity in the pipe '%s' is not a source\n",
 					ved->ent->name);
-				vimc_streamer_pipeline_terminate(stream);
+				stream->pipe_size = 0;
 				return -EPIPE;
 			}
 			return 0;
@@ -129,7 +90,7 @@ static int vimc_streamer_pipeline_init(struct vimc_stream *stream,
 		}
 	}
 
-	vimc_streamer_pipeline_terminate(stream);
+	stream->pipe_size = 0;
 	return -EINVAL;
 }
 
@@ -210,7 +171,7 @@ int vimc_streamer_s_stream(struct vimc_stream *stream,
 		if (IS_ERR(stream->kthread)) {
 			ret = PTR_ERR(stream->kthread);
 			dev_err(ved->dev, "kthread_run failed with %d\n", ret);
-			vimc_streamer_pipeline_terminate(stream);
+			stream->pipe_size = 0;
 			stream->kthread = NULL;
 			return ret;
 		}
@@ -231,7 +192,7 @@ int vimc_streamer_s_stream(struct vimc_stream *stream,
 
 		stream->kthread = NULL;
 
-		vimc_streamer_pipeline_terminate(stream);
+		stream->pipe_size = 0;
 	}
 
 	return 0;
-- 
2.17.1

Re: [PATCH v4 0/5] media: add v4l2_pipeline_stream_{enable,disable}

[Helen Koike]

Hi Dafna,

Thanks for working on this.

On 5/22/20 4:55 AM, Dafna Hirschfeld wrote:
> Hi,
> This is v4 of the patchset that was sent by Helen Koike.
> 
> Media drivers need to iterate through the pipeline and call .s_stream()
> callbacks in the subdevices.
> 
> Instead of repeating code, add helpers for this.
> 
> These helpers will go walk through the pipeline only visiting entities
> that participates in the stream, i.e. it follows links from sink to source
> (and not the opposite).
> For example, in a topology like this https://bit.ly/3b2MxjI
> calling v4l2_pipeline_stream_enable() from rkisp1_mainpath won't call
> .s_stream(true) for rkisp1_resizer_selfpath.
> 
> stream_count variable was added in v4l2_subdevice to handle nested calls
> to the helpers.
> This is useful when the driver allows streaming from more then one
> capture device sharing subdevices.

If I understand correctly, this isn't  true anymore right? Nested calls aren't
possible anymore since this version doesn't contain stream_count in struct v4l2_subdevice.

Documentation of v4l2_pipeline_stream_*() should also be updated.

Just to be clear, without the nested call, vimc will require to add its own
counters, patch https://patchwork.kernel.org/patch/10948833/ will be
required again to allow multi streaming.

Also, patch "media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable}"
is cleaner in the previous version (with stream_count in struct v4l2_subdevice).

All drivers that allows multi streaming will need to implement some special handling.

Adding stream_count in struct v4l2_subdevice still seems cleaner to me. I'd like to hear
what others think.

> 
> This patchset was tested on rkisp1 and vimc drivers.
> 
> Other cleanup might be possible (but I won't add in this patchset as I
> don't have the hw to test):
> 	https://git.linuxtv.org/media_tree.git/tree/drivers/media/platform/qcom/camss/camss-video.c#n430
> 	https://git.linuxtv.org/media_tree.git/tree/drivers/media/platform/omap3isp/isp.c#n697
> 	https://git.linuxtv.org/media_tree.git/tree/drivers/media/platform/stm32/stm32-dcmi.c#n680
> 	https://git.linuxtv.org/media_tree.git/tree/drivers/media/platform/xilinx/xilinx-dma.c#n97
> 
> Testing:
> --------
> 
> SEN_DEV=/dev/v4l-subdev3
> ISP_DEV=/dev/v4l-subdev0
> RSZ_SP_DEV=/dev/v4l-subdev2
> RSZ_MP_DEV=/dev/v4l-subdev1
> CAP_SP_DEV=/dev/video1
> CAP_MP_DEV=/dev/video0
> 
> WIDTH=1920
> HEIGHT=1080
> RAW_CODE=SRGGB10_1X10
> YUV_CODE=YUYV8_2X8
> 
> v4l2-ctl --set-subdev-fmt pad=0,width=$WIDTH,height=$HEIGHT,code=$RAW_CODE -d $SEN_DEV
> 
> v4l2-ctl --set-subdev-fmt pad=0,width=$WIDTH,height=$HEIGHT,code=$RAW_CODE -d $ISP_DEV
> v4l2-ctl --set-subdev-selection pad=0,target=crop,top=0,left=0,width=$WIDTH,height=$HEIGHT -d $ISP_DEV
> 
> v4l2-ctl --set-subdev-selection pad=2,target=crop,top=0,left=0,width=$WIDTH,height=$HEIGHT -d $ISP_DEV
> v4l2-ctl --set-subdev-fmt pad=2,width=$WIDTH,height=$HEIGHT,code=$YUV_CODE -d $ISP_DEV
> 
> v4l2-ctl --set-subdev-fmt pad=0,width=$WIDTH,height=$HEIGHT,code=$YUV_CODE -d $RSZ_SP_DEV
> v4l2-ctl --set-subdev-fmt pad=1,width=$WIDTH,height=$HEIGHT,code=$YUV_CODE -d $RSZ_SP_DEV
> 
> v4l2-ctl --set-subdev-selection pad=0,target=crop,top=0,left=0,width=$WIDTH,height=$HEIGHT -d $RSZ_SP_DEV
> 
> v4l2-ctl --set-subdev-fmt pad=0,width=$WIDTH,height=$HEIGHT,code=$YUV_CODE -d $RSZ_MP_DEV
> v4l2-ctl --set-subdev-fmt pad=1,width=$WIDTH,height=$HEIGHT,code=$YUV_CODE -d $RSZ_MP_DEV
> 
> v4l2-ctl --set-subdev-selection pad=0,target=crop,top=0,left=0,width=$WIDTH,height=$HEIGHT -d $RSZ_MP_DEV
> 
> v4l2-ctl -v width=$WIDTH,height=$HEIGHT,pixelformat=NV12 -d $CAP_SP_DEV
> v4l2-ctl -v width=$WIDTH,height=$HEIGHT,pixelformat=NV12 -d $CAP_MP_DEV
> 
> sleep 1
> 
> v4l2-ctl --stream-mmap --stream-count=100 -d $CAP_SP_DEV --stream-to=/tmp/test_sp.raw &
> v4l2-ctl --stream-mmap --stream-count=100 -d $CAP_MP_DEV --stream-to=/tmp/test_mp.raw &
> 
> wait
> echo "Completed"
> 
> 
> Changes in v4:
> ==============
> patch 1: fix coding style issues
> 
> patch 2:
> - in function v4l2_pipeline_subdevs_get, use a local media_graph to walk on the topology so a lock is not needed
> and also the pipe parameter is not needed.

You are assuming that media_pipeline_start() will always be called before v4l2_pipeline_stream_enable().
I think this is fine, but it should reflect in the docs.

Regards,
Helen

> - adding a function v4l2_subdevs_stream_disable to avoid code duplication
> - change v4l2_pipeline_stream_disable to return an error code if failed
> - don't add a new field to subdevice "stream_counter" when calling s_stream, since this counter is updated only in
> the helper functions, and might be confusing that it is generally not an indication of the number of calls to s_stream.
> Also, except of rkisp1, and vimc, it seems that the other drivers that might use the new helpers don't use a counter.
> 
> new added - patch 3: the call to media_pipeline_start should be called before calling s_stream on the subdevices in order
> to validate the links and prevents them from changing, this patch fixes it.
> 
> patch 4: (use the helpers in rkisp1). The helpers now don't have a counter for the number of calls to s_stream, so rkisp1
> should check if the other capture is streaming and in that case call s_stream only for its resizer.
> 
> patch 5: - (use the helpers in vimc)
> - test the return value of v4l2_pipeline_stream_disable
> - the call to the helerps now doesn't need the pipe parameter.
> 
> Overview of patches in V3:
> --------------------------
> 
> Patch 1/5 adds a new iterator function in media-controller to follow links from sink to
> source only.
> 
> Patch 2/5 adds the helpers in v4l2-common.c,
> 
> Patch 3/5 calles media_pipeline_start before calling s_stream on the subdevices
> 
> Patch 4/5 cleanup rkisp1 driver to use the helpers.
> 
> Patch 5/5 cleanup vimc driver to use the helpers.
> 
> Changes in V3:
> ====================
> Following up Niklas' comments in V2 https://patchwork.kernel.org/patch/11473681/#23270823
> 
> * I removed the limitation in topologies with entities with multiple enabled
> links to its sink pads in the topology.
> Now it enables all subdevs in the pipeline that have an enabled link going
> from sink to source while walking from the video device, so it can be
> also useful for rcar-vin driver.
> 
> To implement this, I added back in the series the patch from v1:
>     "media: mc-entity.c: add media_graph_walk_next_stream()"
> 
> * "size" was renamed to "max_size" in function v4l2_pipeline_subdevs_get()
> to reflect the maximum number of elements that can fit in the subdevs array,
> with proper documentation.
> 
> * v4l2_pipeline_subdevs_get() returns a negative number for error, instead
> of returning 0 and printing a warning.
> 
> * I also add if defined(CONFIG_MEDIA_CONTROLLER) around helpers to avoid
> compiling errors.
> 
> Overview of patches in V3:
> --------------------------
> 
> Patch 1/4 adds a new iterator function in media-controller to follow links from sink to
> source only.
> 
> Patch 2/4 adds the helpers in v4l2-common.c, allowing nested calls by
> adding stream_count in the subdevice struct.
> 
> Patch 3/4 cleanup rkisp1 driver to use the helpers.
> 
> Patch 4/4 cleanup vimc driver to use the helpers.
> 
> Changes in V2:
> ====================
> The first version was calling the s_stream() callbacks from sensor to
> capture.
> 
> This was generating errors in the Scarlet Chromebook, when the sensor
> was being enabled before the ISP.
> 
> It make sense to enable subdevices from capture to sensor instead (which
> is what most drivers do already).
> 
> This v2 drops the changes from mc-entity.c, and re-implement helpers in
> v4l2-common.c
> 
> Overview of patches in V2:
> --------------------------
> 
> Path 1/3 adds the helpers in v4l2-common.c, allowing nested calls by
> adding stream_count in the subdevice struct.
> 
> Patch 2/3 cleanup rkisp1 driver to use the helpers.
> 
> Patch 3/3 cleanup vimc driver to use the helpers.
> 
> Dafna Hirschfeld (1):
>   media: staging: rkisp1: validate links before powering and streaming
> 
> Helen Koike (4):
>   media: mc-entity.c: add media_graph_walk_next_stream()
>   media: v4l2-common: add helper functions to call s_stream() callbacks
>   media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable}
>     helpers
>   media: vimc: use v4l2_pipeline_stream_{enable,disable} helpers
> 
>  drivers/media/mc/mc-entity.c                  |  34 ++++-
>  .../media/test-drivers/vimc/vimc-capture.c    |  31 +++--
>  .../media/test-drivers/vimc/vimc-streamer.c   |  49 +------
>  drivers/media/v4l2-core/v4l2-common.c         |  99 ++++++++++++++
>  drivers/staging/media/rkisp1/rkisp1-capture.c | 125 ++++++------------
>  include/media/media-entity.h                  |  15 +++
>  include/media/v4l2-common.h                   |  39 ++++++
>  7 files changed, 253 insertions(+), 139 deletions(-)
> 

Re: [PATCH v4 2/5] media: v4l2-common: add helper functions to call s_stream() callbacks

[Sakari Ailus]

Hi Dafna,

My apologies for not reviewing this earlier.

On Fri, May 22, 2020 at 09:55:19AM +0200, Dafna Hirschfeld wrote:
> From: Helen Koike <helen.koike@collabora.com>
> 
> Add v4l2_pipeline_stream_{enable,disable} helper functions to iterate
> through the subdevices in a given stream (i.e following links from sink
> to source) and call .s_stream() callback.
> 
> If .s_stream(true) fails, call .s_stream(false) in the reverse order.
> 
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> ---
>  drivers/media/v4l2-core/v4l2-common.c | 99 +++++++++++++++++++++++++++
>  include/media/v4l2-common.h           | 39 +++++++++++
>  2 files changed, 138 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-common.c b/drivers/media/v4l2-core/v4l2-common.c
> index 9e8eb45a5b03..734db2bf5ca9 100644
> --- a/drivers/media/v4l2-core/v4l2-common.c
> +++ b/drivers/media/v4l2-core/v4l2-common.c
> @@ -442,3 +442,102 @@ int v4l2_fill_pixfmt(struct v4l2_pix_format *pixfmt, u32 pixelformat,
>  	return 0;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_fill_pixfmt);
> +
> +#if defined(CONFIG_MEDIA_CONTROLLER)
> +
> +/*
> + * v4l2_pipeline_subdevs_get - Assemble a list of subdevices in a pipeline
> + * @subdevs: the array to be filled.
> + * @max_size: max number of elements that can fit in subdevs array.
> + *
> + * Walk from a video node, following links from sink to source and fill the
> + * array with subdevices in the path.
> + * The walker performs a depth-first traversal, which means that, in a topology
> + * sd1->sd2->sd3->vdev, sd1 will be the first element placed in the array.
> + *
> + * Return the number of subdevices filled in the array.
> + */
> +static int v4l2_pipeline_subdevs_get(struct video_device *vdev,
> +				     struct v4l2_subdev **subdevs,
> +				     unsigned int max_size)
> +{
> +	struct media_entity *entity = &vdev->entity;
> +	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_graph graph;
> +	int idx = 0;
> +	int ret;
> +
> +	ret = media_graph_walk_init(&graph, mdev);
> +	if (ret)
> +		return ret;
> +
> +	media_graph_walk_start(&graph, entity);
> +
> +	while ((entity = media_graph_walk_next_stream(&graph))) {
> +		if (!is_media_entity_v4l2_subdev(entity))
> +			continue;
> +		if (WARN_ON(idx >= max_size))
> +			return -EINVAL;
> +		subdevs[idx++] = media_entity_to_v4l2_subdev(entity);
> +	}
> +
> +	media_graph_walk_cleanup(&graph);
> +
> +	return idx;
> +}
> +
> +static void v4l2_subdevs_stream_disable(struct v4l2_subdev **subdevs, int size)
> +{
> +	int i;
> +
> +	for (i = 0; i < size; i++) {
> +		struct v4l2_subdev *sd = subdevs[i];
> +
> +		dev_dbg(sd->dev,
> +			"disabling stream for '%s'\n", sd->entity.name);
> +		v4l2_subdev_call(sd, video, s_stream, false);
> +	}
> +}
> +
> +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev)
> +{
> +	struct media_device *mdev = vdev->entity.graph_obj.mdev;
> +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
> +	struct v4l2_subdev *sd;
> +	int i, size, ret;
> +
> +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
> +	if (size <= 0)
> +		return size;
> +
> +	/* Call s_stream() in reverse order to enable sensors last */
> +	for (i = size - 1; i >= 0; i--) {
> +		sd = subdevs[i];
> +		dev_dbg(mdev->dev,
> +			"enabling stream for '%s'\n", sd->entity.name);
> +		ret = v4l2_subdev_call(sd, video, s_stream, true);

The s_stream callback is only called on the very next sub-device upstream
in the pipeline, not on any further sub-devices. This is because a driver
for the device knows best in which order things need to be set up.

This could include, for instance, telling a sensor to place its CSI-2
transmitter to LP-11 state.

> +		if (ret && ret != -ENOIOCTLCMD) {
> +			v4l2_subdevs_stream_disable(subdevs + i + 1,
> +						    size - i - 1);
> +			return ret;
> +		}
> +	}
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_enable);
> +
> +int v4l2_pipeline_stream_disable(struct video_device *vdev)
> +{
> +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
> +	int size;
> +
> +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
> +	if (size < 0)
> +		return size;
> +
> +	v4l2_subdevs_stream_disable(subdevs, size);
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_disable);
> +
> +#endif /* CONFIG_MEDIA_CONTROLLER */
> diff --git a/include/media/v4l2-common.h b/include/media/v4l2-common.h
> index 150ee16ebd81..a278bcf3c5bc 100644
> --- a/include/media/v4l2-common.h
> +++ b/include/media/v4l2-common.h
> @@ -539,4 +539,43 @@ static inline void v4l2_buffer_set_timestamp(struct v4l2_buffer *buf,
>  	buf->timestamp.tv_usec = ts.tv_nsec / NSEC_PER_USEC;
>  }
>  
> +#if defined(CONFIG_MEDIA_CONTROLLER)
> +
> +/**
> + * v4l2_pipeline_stream_enable - Call .s_stream(true) callbacks in the stream
> + * @vdev: Starting video device.
> + *
> + * Call .s_stream(true) callback in all the subdevices participating in the
> + * stream, i.e. following links from sink to source.
> + *
> + * .s_stream(true) is also called from sink to source, i.e. in a topology
> + * sd1->sd2->sd3->vdev, .s_stream(true) of sd3 is called first.
> + *
> + * Calls to this function can be nested, in which case the same number of
> + * v4l2_pipeline_stream_disable() calls will be required to disable streaming
> + * through subdevices in the pipeline.
> + * The  pipeline pointer must be identical for all nested calls to
> + * v4l2_pipeline_stream_enable().
> + */
> +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev);
> +
> +/**
> + * v4l2_pipeline_stream_disable - Call .s_stream(false) callbacks in the stream
> + * @vdev: Starting video device.
> + *
> + * Call .s_stream(false) callback in all the subdevices participating in the
> + * stream, i.e. following links from sink to source.
> + *
> + * s_stream(false) is called in a reverse order from
> + * v4l2_pipeline_stream_enable(), i.e. in a topology sd1->sd2->sd3->vdev,
> + * .s_stream(false) of sd1 is called first.
> + *
> + * If multiple calls to v4l2_pipeline_stream_enable() have been made, the same
> + * number of calls to this function are required to disable streaming through
> + * subdevices in the pipeline.
> + */
> +int v4l2_pipeline_stream_disable(struct video_device *vdev);
> +
> +#endif /* CONFIG_MEDIA_CONTROLLER */
> +
>  #endif /* V4L2_COMMON_H_ */

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v4 2/5] media: v4l2-common: add helper functions to call s_stream() callbacks

[Dafna Hirschfeld]

On 25.05.20 11:23, Sakari Ailus wrote:
> Hi Dafna,
> 
> My apologies for not reviewing this earlier.
No problem :)

> 
> On Fri, May 22, 2020 at 09:55:19AM +0200, Dafna Hirschfeld wrote:
>> From: Helen Koike <helen.koike@collabora.com>
>>
>> Add v4l2_pipeline_stream_{enable,disable} helper functions to iterate
>> through the subdevices in a given stream (i.e following links from sink
>> to source) and call .s_stream() callback.
>>
>> If .s_stream(true) fails, call .s_stream(false) in the reverse order.
>>
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-common.c | 99 +++++++++++++++++++++++++++
>>   include/media/v4l2-common.h           | 39 +++++++++++
>>   2 files changed, 138 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-common.c b/drivers/media/v4l2-core/v4l2-common.c
>> index 9e8eb45a5b03..734db2bf5ca9 100644
>> --- a/drivers/media/v4l2-core/v4l2-common.c
>> +++ b/drivers/media/v4l2-core/v4l2-common.c
>> @@ -442,3 +442,102 @@ int v4l2_fill_pixfmt(struct v4l2_pix_format *pixfmt, u32 pixelformat,
>>   	return 0;
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_fill_pixfmt);
>> +
>> +#if defined(CONFIG_MEDIA_CONTROLLER)
>> +
>> +/*
>> + * v4l2_pipeline_subdevs_get - Assemble a list of subdevices in a pipeline
>> + * @subdevs: the array to be filled.
>> + * @max_size: max number of elements that can fit in subdevs array.
>> + *
>> + * Walk from a video node, following links from sink to source and fill the
>> + * array with subdevices in the path.
>> + * The walker performs a depth-first traversal, which means that, in a topology
>> + * sd1->sd2->sd3->vdev, sd1 will be the first element placed in the array.
>> + *
>> + * Return the number of subdevices filled in the array.
>> + */
>> +static int v4l2_pipeline_subdevs_get(struct video_device *vdev,
>> +				     struct v4l2_subdev **subdevs,
>> +				     unsigned int max_size)
>> +{
>> +	struct media_entity *entity = &vdev->entity;
>> +	struct media_device *mdev = entity->graph_obj.mdev;
>> +	struct media_graph graph;
>> +	int idx = 0;
>> +	int ret;
>> +
>> +	ret = media_graph_walk_init(&graph, mdev);
>> +	if (ret)
>> +		return ret;
>> +
>> +	media_graph_walk_start(&graph, entity);
>> +
>> +	while ((entity = media_graph_walk_next_stream(&graph))) {
>> +		if (!is_media_entity_v4l2_subdev(entity))
>> +			continue;
>> +		if (WARN_ON(idx >= max_size))
>> +			return -EINVAL;
>> +		subdevs[idx++] = media_entity_to_v4l2_subdev(entity);
>> +	}
>> +
>> +	media_graph_walk_cleanup(&graph);
>> +
>> +	return idx;
>> +}
>> +
>> +static void v4l2_subdevs_stream_disable(struct v4l2_subdev **subdevs, int size)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < size; i++) {
>> +		struct v4l2_subdev *sd = subdevs[i];
>> +
>> +		dev_dbg(sd->dev,
>> +			"disabling stream for '%s'\n", sd->entity.name);
>> +		v4l2_subdev_call(sd, video, s_stream, false);
>> +	}
>> +}
>> +
>> +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev)
>> +{
>> +	struct media_device *mdev = vdev->entity.graph_obj.mdev;
>> +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
>> +	struct v4l2_subdev *sd;
>> +	int i, size, ret;
>> +
>> +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
>> +	if (size <= 0)
>> +		return size;
>> +
>> +	/* Call s_stream() in reverse order to enable sensors last */
>> +	for (i = size - 1; i >= 0; i--) {
>> +		sd = subdevs[i];
>> +		dev_dbg(mdev->dev,
>> +			"enabling stream for '%s'\n", sd->entity.name);
>> +		ret = v4l2_subdev_call(sd, video, s_stream, true);
> 
> The s_stream callback is only called on the very next sub-device upstream
> in the pipeline, not on any further sub-devices. This is because a driver
> for the device knows best in which order things need to be set up.
> 
This is only a helper function, drivers can choose not to use it.
In many cases the same driver implements many subdevices so the driver
knows what should be done also for subdevices deeper in the stream.
In the cover letter of the patch there is a list of links to code parts
that can be replaced by this helper functions.
The cover letter: https://patchwork.kernel.org/cover/11564901/

Thanks,
Dafna

> This could include, for instance, telling a sensor to place its CSI-2
> transmitter to LP-11 state.
> 
>> +		if (ret && ret != -ENOIOCTLCMD) {
>> +			v4l2_subdevs_stream_disable(subdevs + i + 1,
>> +						    size - i - 1);
>> +			return ret;
>> +		}
>> +	}
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_enable);
>> +
>> +int v4l2_pipeline_stream_disable(struct video_device *vdev)
>> +{
>> +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
>> +	int size;
>> +
>> +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
>> +	if (size < 0)
>> +		return size;
>> +
>> +	v4l2_subdevs_stream_disable(subdevs, size);
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_disable);
>> +
>> +#endif /* CONFIG_MEDIA_CONTROLLER */
>> diff --git a/include/media/v4l2-common.h b/include/media/v4l2-common.h
>> index 150ee16ebd81..a278bcf3c5bc 100644
>> --- a/include/media/v4l2-common.h
>> +++ b/include/media/v4l2-common.h
>> @@ -539,4 +539,43 @@ static inline void v4l2_buffer_set_timestamp(struct v4l2_buffer *buf,
>>   	buf->timestamp.tv_usec = ts.tv_nsec / NSEC_PER_USEC;
>>   }
>>   
>> +#if defined(CONFIG_MEDIA_CONTROLLER)
>> +
>> +/**
>> + * v4l2_pipeline_stream_enable - Call .s_stream(true) callbacks in the stream
>> + * @vdev: Starting video device.
>> + *
>> + * Call .s_stream(true) callback in all the subdevices participating in the
>> + * stream, i.e. following links from sink to source.
>> + *
>> + * .s_stream(true) is also called from sink to source, i.e. in a topology
>> + * sd1->sd2->sd3->vdev, .s_stream(true) of sd3 is called first.
>> + *
>> + * Calls to this function can be nested, in which case the same number of
>> + * v4l2_pipeline_stream_disable() calls will be required to disable streaming
>> + * through subdevices in the pipeline.
>> + * The  pipeline pointer must be identical for all nested calls to
>> + * v4l2_pipeline_stream_enable().
>> + */
>> +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev);
>> +
>> +/**
>> + * v4l2_pipeline_stream_disable - Call .s_stream(false) callbacks in the stream
>> + * @vdev: Starting video device.
>> + *
>> + * Call .s_stream(false) callback in all the subdevices participating in the
>> + * stream, i.e. following links from sink to source.
>> + *
>> + * s_stream(false) is called in a reverse order from
>> + * v4l2_pipeline_stream_enable(), i.e. in a topology sd1->sd2->sd3->vdev,
>> + * .s_stream(false) of sd1 is called first.
>> + *
>> + * If multiple calls to v4l2_pipeline_stream_enable() have been made, the same
>> + * number of calls to this function are required to disable streaming through
>> + * subdevices in the pipeline.
>> + */
>> +int v4l2_pipeline_stream_disable(struct video_device *vdev);
>> +
>> +#endif /* CONFIG_MEDIA_CONTROLLER */
>> +
>>   #endif /* V4L2_COMMON_H_ */
> 

Re: [PATCH v4 2/5] media: v4l2-common: add helper functions to call s_stream() callbacks

[Sakari Ailus]

Hi Dafna,

On Mon, May 25, 2020 at 11:42:37AM +0200, Dafna Hirschfeld wrote:
> 
> 
> On 25.05.20 11:23, Sakari Ailus wrote:
> > Hi Dafna,
> > 
> > My apologies for not reviewing this earlier.
> No problem :)
> 
> > 
> > On Fri, May 22, 2020 at 09:55:19AM +0200, Dafna Hirschfeld wrote:
> > > From: Helen Koike <helen.koike@collabora.com>
> > > 
> > > Add v4l2_pipeline_stream_{enable,disable} helper functions to iterate
> > > through the subdevices in a given stream (i.e following links from sink
> > > to source) and call .s_stream() callback.
> > > 
> > > If .s_stream(true) fails, call .s_stream(false) in the reverse order.
> > > 
> > > Signed-off-by: Helen Koike <helen.koike@collabora.com>
> > > Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> > > ---
> > >   drivers/media/v4l2-core/v4l2-common.c | 99 +++++++++++++++++++++++++++
> > >   include/media/v4l2-common.h           | 39 +++++++++++
> > >   2 files changed, 138 insertions(+)
> > > 
> > > diff --git a/drivers/media/v4l2-core/v4l2-common.c b/drivers/media/v4l2-core/v4l2-common.c
> > > index 9e8eb45a5b03..734db2bf5ca9 100644
> > > --- a/drivers/media/v4l2-core/v4l2-common.c
> > > +++ b/drivers/media/v4l2-core/v4l2-common.c
> > > @@ -442,3 +442,102 @@ int v4l2_fill_pixfmt(struct v4l2_pix_format *pixfmt, u32 pixelformat,
> > >   	return 0;
> > >   }
> > >   EXPORT_SYMBOL_GPL(v4l2_fill_pixfmt);
> > > +
> > > +#if defined(CONFIG_MEDIA_CONTROLLER)
> > > +
> > > +/*
> > > + * v4l2_pipeline_subdevs_get - Assemble a list of subdevices in a pipeline
> > > + * @subdevs: the array to be filled.
> > > + * @max_size: max number of elements that can fit in subdevs array.
> > > + *
> > > + * Walk from a video node, following links from sink to source and fill the
> > > + * array with subdevices in the path.
> > > + * The walker performs a depth-first traversal, which means that, in a topology
> > > + * sd1->sd2->sd3->vdev, sd1 will be the first element placed in the array.
> > > + *
> > > + * Return the number of subdevices filled in the array.
> > > + */
> > > +static int v4l2_pipeline_subdevs_get(struct video_device *vdev,
> > > +				     struct v4l2_subdev **subdevs,
> > > +				     unsigned int max_size)
> > > +{
> > > +	struct media_entity *entity = &vdev->entity;
> > > +	struct media_device *mdev = entity->graph_obj.mdev;
> > > +	struct media_graph graph;
> > > +	int idx = 0;
> > > +	int ret;
> > > +
> > > +	ret = media_graph_walk_init(&graph, mdev);
> > > +	if (ret)
> > > +		return ret;
> > > +
> > > +	media_graph_walk_start(&graph, entity);
> > > +
> > > +	while ((entity = media_graph_walk_next_stream(&graph))) {
> > > +		if (!is_media_entity_v4l2_subdev(entity))
> > > +			continue;
> > > +		if (WARN_ON(idx >= max_size))
> > > +			return -EINVAL;
> > > +		subdevs[idx++] = media_entity_to_v4l2_subdev(entity);
> > > +	}
> > > +
> > > +	media_graph_walk_cleanup(&graph);
> > > +
> > > +	return idx;
> > > +}
> > > +
> > > +static void v4l2_subdevs_stream_disable(struct v4l2_subdev **subdevs, int size)
> > > +{
> > > +	int i;
> > > +
> > > +	for (i = 0; i < size; i++) {
> > > +		struct v4l2_subdev *sd = subdevs[i];
> > > +
> > > +		dev_dbg(sd->dev,
> > > +			"disabling stream for '%s'\n", sd->entity.name);
> > > +		v4l2_subdev_call(sd, video, s_stream, false);
> > > +	}
> > > +}
> > > +
> > > +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev)
> > > +{
> > > +	struct media_device *mdev = vdev->entity.graph_obj.mdev;
> > > +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
> > > +	struct v4l2_subdev *sd;
> > > +	int i, size, ret;
> > > +
> > > +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
> > > +	if (size <= 0)
> > > +		return size;
> > > +
> > > +	/* Call s_stream() in reverse order to enable sensors last */
> > > +	for (i = size - 1; i >= 0; i--) {
> > > +		sd = subdevs[i];
> > > +		dev_dbg(mdev->dev,
> > > +			"enabling stream for '%s'\n", sd->entity.name);
> > > +		ret = v4l2_subdev_call(sd, video, s_stream, true);
> > 
> > The s_stream callback is only called on the very next sub-device upstream
> > in the pipeline, not on any further sub-devices. This is because a driver
> > for the device knows best in which order things need to be set up.
> > 
> This is only a helper function, drivers can choose not to use it.
> In many cases the same driver implements many subdevices so the driver
> knows what should be done also for subdevices deeper in the stream.

Can it be used on devices that do not operate from memory to memory? I.e.
how do you ensure the s_stream is not called on further sub-devices than
those that are adjacent to what this driver controls?

One option could be to check sd->dev and skip devices that are further away
but for that you'd also need to know how these sub-devices have been
reached. The graph walk does not currently provide this information.

It's true that many drivers implement similar functionality for streaming.
It's non-trivial but often makes use of some device specific knowledge, for
instance in which order things take place (such as whether the upstream
sub-device is started before the downstream one). These implementations
also do not use graph walk for the above reasons.

> In the cover letter of the patch there is a list of links to code parts
> that can be replaced by this helper functions.
> The cover letter: https://patchwork.kernel.org/cover/11564901/
> 
> Thanks,
> Dafna
> 
> > This could include, for instance, telling a sensor to place its CSI-2
> > transmitter to LP-11 state.
> > 
> > > +		if (ret && ret != -ENOIOCTLCMD) {
> > > +			v4l2_subdevs_stream_disable(subdevs + i + 1,
> > > +						    size - i - 1);
> > > +			return ret;
> > > +		}
> > > +	}
> > > +	return 0;
> > > +}
> > > +EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_enable);
> > > +
> > > +int v4l2_pipeline_stream_disable(struct video_device *vdev)
> > > +{
> > > +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
> > > +	int size;
> > > +
> > > +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
> > > +	if (size < 0)
> > > +		return size;
> > > +
> > > +	v4l2_subdevs_stream_disable(subdevs, size);
> > > +	return 0;
> > > +}
> > > +EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_disable);
> > > +
> > > +#endif /* CONFIG_MEDIA_CONTROLLER */
> > > diff --git a/include/media/v4l2-common.h b/include/media/v4l2-common.h
> > > index 150ee16ebd81..a278bcf3c5bc 100644
> > > --- a/include/media/v4l2-common.h
> > > +++ b/include/media/v4l2-common.h
> > > @@ -539,4 +539,43 @@ static inline void v4l2_buffer_set_timestamp(struct v4l2_buffer *buf,
> > >   	buf->timestamp.tv_usec = ts.tv_nsec / NSEC_PER_USEC;
> > >   }
> > > +#if defined(CONFIG_MEDIA_CONTROLLER)
> > > +
> > > +/**
> > > + * v4l2_pipeline_stream_enable - Call .s_stream(true) callbacks in the stream
> > > + * @vdev: Starting video device.
> > > + *
> > > + * Call .s_stream(true) callback in all the subdevices participating in the
> > > + * stream, i.e. following links from sink to source.
> > > + *
> > > + * .s_stream(true) is also called from sink to source, i.e. in a topology
> > > + * sd1->sd2->sd3->vdev, .s_stream(true) of sd3 is called first.
> > > + *
> > > + * Calls to this function can be nested, in which case the same number of
> > > + * v4l2_pipeline_stream_disable() calls will be required to disable streaming
> > > + * through subdevices in the pipeline.
> > > + * The  pipeline pointer must be identical for all nested calls to
> > > + * v4l2_pipeline_stream_enable().
> > > + */
> > > +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev);
> > > +
> > > +/**
> > > + * v4l2_pipeline_stream_disable - Call .s_stream(false) callbacks in the stream
> > > + * @vdev: Starting video device.
> > > + *
> > > + * Call .s_stream(false) callback in all the subdevices participating in the
> > > + * stream, i.e. following links from sink to source.
> > > + *
> > > + * s_stream(false) is called in a reverse order from
> > > + * v4l2_pipeline_stream_enable(), i.e. in a topology sd1->sd2->sd3->vdev,
> > > + * .s_stream(false) of sd1 is called first.
> > > + *
> > > + * If multiple calls to v4l2_pipeline_stream_enable() have been made, the same
> > > + * number of calls to this function are required to disable streaming through
> > > + * subdevices in the pipeline.
> > > + */
> > > +int v4l2_pipeline_stream_disable(struct video_device *vdev);
> > > +
> > > +#endif /* CONFIG_MEDIA_CONTROLLER */
> > > +
> > >   #endif /* V4L2_COMMON_H_ */
> > 

-- 
Sakari Ailus

Re: [PATCH v4 2/5] media: v4l2-common: add helper functions to call s_stream() callbacks

[Dafna Hirschfeld]

On 25.05.20 12:03, Sakari Ailus wrote:
> Hi Dafna,
> 
> On Mon, May 25, 2020 at 11:42:37AM +0200, Dafna Hirschfeld wrote:
>>
>>
>> On 25.05.20 11:23, Sakari Ailus wrote:
>>> Hi Dafna,
>>>
>>> My apologies for not reviewing this earlier.
>> No problem :)
>>
>>>
>>> On Fri, May 22, 2020 at 09:55:19AM +0200, Dafna Hirschfeld wrote:
>>>> From: Helen Koike <helen.koike@collabora.com>
>>>>
>>>> Add v4l2_pipeline_stream_{enable,disable} helper functions to iterate
>>>> through the subdevices in a given stream (i.e following links from sink
>>>> to source) and call .s_stream() callback.
>>>>
>>>> If .s_stream(true) fails, call .s_stream(false) in the reverse order.
>>>>
>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>> ---
>>>>    drivers/media/v4l2-core/v4l2-common.c | 99 +++++++++++++++++++++++++++
>>>>    include/media/v4l2-common.h           | 39 +++++++++++
>>>>    2 files changed, 138 insertions(+)
>>>>
>>>> diff --git a/drivers/media/v4l2-core/v4l2-common.c b/drivers/media/v4l2-core/v4l2-common.c
>>>> index 9e8eb45a5b03..734db2bf5ca9 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-common.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-common.c
>>>> @@ -442,3 +442,102 @@ int v4l2_fill_pixfmt(struct v4l2_pix_format *pixfmt, u32 pixelformat,
>>>>    	return 0;
>>>>    }
>>>>    EXPORT_SYMBOL_GPL(v4l2_fill_pixfmt);
>>>> +
>>>> +#if defined(CONFIG_MEDIA_CONTROLLER)
>>>> +
>>>> +/*
>>>> + * v4l2_pipeline_subdevs_get - Assemble a list of subdevices in a pipeline
>>>> + * @subdevs: the array to be filled.
>>>> + * @max_size: max number of elements that can fit in subdevs array.
>>>> + *
>>>> + * Walk from a video node, following links from sink to source and fill the
>>>> + * array with subdevices in the path.
>>>> + * The walker performs a depth-first traversal, which means that, in a topology
>>>> + * sd1->sd2->sd3->vdev, sd1 will be the first element placed in the array.
>>>> + *
>>>> + * Return the number of subdevices filled in the array.
>>>> + */
>>>> +static int v4l2_pipeline_subdevs_get(struct video_device *vdev,
>>>> +				     struct v4l2_subdev **subdevs,
>>>> +				     unsigned int max_size)
>>>> +{
>>>> +	struct media_entity *entity = &vdev->entity;
>>>> +	struct media_device *mdev = entity->graph_obj.mdev;
>>>> +	struct media_graph graph;
>>>> +	int idx = 0;
>>>> +	int ret;
>>>> +
>>>> +	ret = media_graph_walk_init(&graph, mdev);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	media_graph_walk_start(&graph, entity);
>>>> +
>>>> +	while ((entity = media_graph_walk_next_stream(&graph))) {
>>>> +		if (!is_media_entity_v4l2_subdev(entity))
>>>> +			continue;
>>>> +		if (WARN_ON(idx >= max_size))
>>>> +			return -EINVAL;
>>>> +		subdevs[idx++] = media_entity_to_v4l2_subdev(entity);
>>>> +	}
>>>> +
>>>> +	media_graph_walk_cleanup(&graph);
>>>> +
>>>> +	return idx;
>>>> +}
>>>> +
>>>> +static void v4l2_subdevs_stream_disable(struct v4l2_subdev **subdevs, int size)
>>>> +{
>>>> +	int i;
>>>> +
>>>> +	for (i = 0; i < size; i++) {
>>>> +		struct v4l2_subdev *sd = subdevs[i];
>>>> +
>>>> +		dev_dbg(sd->dev,
>>>> +			"disabling stream for '%s'\n", sd->entity.name);
>>>> +		v4l2_subdev_call(sd, video, s_stream, false);
>>>> +	}
>>>> +}
>>>> +
>>>> +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev)
>>>> +{
>>>> +	struct media_device *mdev = vdev->entity.graph_obj.mdev;
>>>> +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
>>>> +	struct v4l2_subdev *sd;
>>>> +	int i, size, ret;
>>>> +
>>>> +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
>>>> +	if (size <= 0)
>>>> +		return size;
>>>> +
>>>> +	/* Call s_stream() in reverse order to enable sensors last */
>>>> +	for (i = size - 1; i >= 0; i--) {
>>>> +		sd = subdevs[i];
>>>> +		dev_dbg(mdev->dev,
>>>> +			"enabling stream for '%s'\n", sd->entity.name);
>>>> +		ret = v4l2_subdev_call(sd, video, s_stream, true);
>>>
>>> The s_stream callback is only called on the very next sub-device upstream
>>> in the pipeline, not on any further sub-devices. This is because a driver
>>> for the device knows best in which order things need to be set up.
>>>
>> This is only a helper function, drivers can choose not to use it.
>> In many cases the same driver implements many subdevices so the driver
>> knows what should be done also for subdevices deeper in the stream.
> 
> Can it be used on devices that do not operate from memory to memory? I.e.
> how do you ensure the s_stream is not called on further sub-devices than
> those that are adjacent to what this driver controls?
Oh, I think I see your point, for example in case of an isp driver and a sensor driver.
The sensor driver can also implement several subdevices that the isp driver is not
aware of and it is the responsibility of the sensor driver to call them.
Is this a common case? Still there are the code parts in drivers that are implementing
calling s_stream in a loop similar to those in the helper functions.
Maybe they are "buggy"?
For example this code part:
https://git.linuxtv.org/media_tree.git/tree/drivers/media/platform/qcom/camss/camss-video.c#n430
is called by the `start_stream` of a video device and I see that the
`camss_probe` function that register it also calls
v4l2_async_notifier_register, which means there might have a subdevice
from a different driver.

Anyway, for the rkisp1 it is probably problematic since it is connected to a sensor
implemented by another driver.
  
> 
> One option could be to check sd->dev and skip devices that are further away
> but for that you'd also need to know how these sub-devices have been
> reached. The graph walk does not currently provide this information.
I think it is possible to use the sd->dev (or maybe sd-owner?)
since we always only go down the stream, so when we reach a subdevice from
a different driver we can stop. (not sure though)

Thanks,
Dafna

> 
> It's true that many drivers implement similar functionality for streaming.
> It's non-trivial but often makes use of some device specific knowledge, for
> instance in which order things take place (such as whether the upstream
> sub-device is started before the downstream one). These implementations
> also do not use graph walk for the above reasons.
> 
>> In the cover letter of the patch there is a list of links to code parts
>> that can be replaced by this helper functions.
>> The cover letter: https://patchwork.kernel.org/cover/11564901/
>>
>> Thanks,
>> Dafna
>>
>>> This could include, for instance, telling a sensor to place its CSI-2
>>> transmitter to LP-11 state.
>>>
>>>> +		if (ret && ret != -ENOIOCTLCMD) {
>>>> +			v4l2_subdevs_stream_disable(subdevs + i + 1,
>>>> +						    size - i - 1);
>>>> +			return ret;
>>>> +		}
>>>> +	}
>>>> +	return 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_enable);
>>>> +
>>>> +int v4l2_pipeline_stream_disable(struct video_device *vdev)
>>>> +{
>>>> +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
>>>> +	int size;
>>>> +
>>>> +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
>>>> +	if (size < 0)
>>>> +		return size;
>>>> +
>>>> +	v4l2_subdevs_stream_disable(subdevs, size);
>>>> +	return 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_disable);
>>>> +
>>>> +#endif /* CONFIG_MEDIA_CONTROLLER */
>>>> diff --git a/include/media/v4l2-common.h b/include/media/v4l2-common.h
>>>> index 150ee16ebd81..a278bcf3c5bc 100644
>>>> --- a/include/media/v4l2-common.h
>>>> +++ b/include/media/v4l2-common.h
>>>> @@ -539,4 +539,43 @@ static inline void v4l2_buffer_set_timestamp(struct v4l2_buffer *buf,
>>>>    	buf->timestamp.tv_usec = ts.tv_nsec / NSEC_PER_USEC;
>>>>    }
>>>> +#if defined(CONFIG_MEDIA_CONTROLLER)
>>>> +
>>>> +/**
>>>> + * v4l2_pipeline_stream_enable - Call .s_stream(true) callbacks in the stream
>>>> + * @vdev: Starting video device.
>>>> + *
>>>> + * Call .s_stream(true) callback in all the subdevices participating in the
>>>> + * stream, i.e. following links from sink to source.
>>>> + *
>>>> + * .s_stream(true) is also called from sink to source, i.e. in a topology
>>>> + * sd1->sd2->sd3->vdev, .s_stream(true) of sd3 is called first.
>>>> + *
>>>> + * Calls to this function can be nested, in which case the same number of
>>>> + * v4l2_pipeline_stream_disable() calls will be required to disable streaming
>>>> + * through subdevices in the pipeline.
>>>> + * The  pipeline pointer must be identical for all nested calls to
>>>> + * v4l2_pipeline_stream_enable().
>>>> + */
>>>> +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev);
>>>> +
>>>> +/**
>>>> + * v4l2_pipeline_stream_disable - Call .s_stream(false) callbacks in the stream
>>>> + * @vdev: Starting video device.
>>>> + *
>>>> + * Call .s_stream(false) callback in all the subdevices participating in the
>>>> + * stream, i.e. following links from sink to source.
>>>> + *
>>>> + * s_stream(false) is called in a reverse order from
>>>> + * v4l2_pipeline_stream_enable(), i.e. in a topology sd1->sd2->sd3->vdev,
>>>> + * .s_stream(false) of sd1 is called first.
>>>> + *
>>>> + * If multiple calls to v4l2_pipeline_stream_enable() have been made, the same
>>>> + * number of calls to this function are required to disable streaming through
>>>> + * subdevices in the pipeline.
>>>> + */
>>>> +int v4l2_pipeline_stream_disable(struct video_device *vdev);
>>>> +
>>>> +#endif /* CONFIG_MEDIA_CONTROLLER */
>>>> +
>>>>    #endif /* V4L2_COMMON_H_ */
>>>
> 

Re: [PATCH v4 0/5] media: add v4l2_pipeline_stream_{enable,disable}

[Tomasz Figa]

On Fri, May 22, 2020 at 11:06 AM Helen Koike <helen.koike@collabora.com> wrote:
>
> Hi Dafna,
>
> Thanks for working on this.
>
> On 5/22/20 4:55 AM, Dafna Hirschfeld wrote:
> > Hi,
> > This is v4 of the patchset that was sent by Helen Koike.
> >
> > Media drivers need to iterate through the pipeline and call .s_stream()
> > callbacks in the subdevices.
> >
> > Instead of repeating code, add helpers for this.
> >
> > These helpers will go walk through the pipeline only visiting entities
> > that participates in the stream, i.e. it follows links from sink to source
> > (and not the opposite).
> > For example, in a topology like this https://bit.ly/3b2MxjI
> > calling v4l2_pipeline_stream_enable() from rkisp1_mainpath won't call
> > .s_stream(true) for rkisp1_resizer_selfpath.
> >
> > stream_count variable was added in v4l2_subdevice to handle nested calls
> > to the helpers.
> > This is useful when the driver allows streaming from more then one
> > capture device sharing subdevices.
>
> If I understand correctly, this isn't  true anymore right? Nested calls aren't
> possible anymore since this version doesn't contain stream_count in struct v4l2_subdevice.
>
> Documentation of v4l2_pipeline_stream_*() should also be updated.
>
> Just to be clear, without the nested call, vimc will require to add its own
> counters, patch https://patchwork.kernel.org/patch/10948833/ will be
> required again to allow multi streaming.
>
> Also, patch "media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable}"
> is cleaner in the previous version (with stream_count in struct v4l2_subdevice).
>
> All drivers that allows multi streaming will need to implement some special handling.
>
> Adding stream_count in struct v4l2_subdevice still seems cleaner to me. I'd like to hear
> what others think.

I certainly would see this reference counting done in generic code,
because requiring every driver to do it simply adds to the endless
amount of boiler plate that V4L2 currently requires from the drivers.
:(

I wonder if it wouldn't be possible to redesign the framework so that
.s_stream() at the subdev level is only called when it's expected to
either start or stop this particular subdev and driver's
implementation can simply execute the requested action.

Best regards,
Tomasz

Re: [PATCH v4 0/5] media: add v4l2_pipeline_stream_{enable,disable}

[Laurent Pinchart]

Hi Tomasz,

On Tue, May 26, 2020 at 06:11:00PM +0200, Tomasz Figa wrote:
> On Fri, May 22, 2020 at 11:06 AM Helen Koike <helen.koike@collabora.com> wrote:
> > On 5/22/20 4:55 AM, Dafna Hirschfeld wrote:
> > > Hi,
> > > This is v4 of the patchset that was sent by Helen Koike.
> > >
> > > Media drivers need to iterate through the pipeline and call .s_stream()
> > > callbacks in the subdevices.
> > >
> > > Instead of repeating code, add helpers for this.
> > >
> > > These helpers will go walk through the pipeline only visiting entities
> > > that participates in the stream, i.e. it follows links from sink to source
> > > (and not the opposite).
> > > For example, in a topology like this https://bit.ly/3b2MxjI
> > > calling v4l2_pipeline_stream_enable() from rkisp1_mainpath won't call
> > > .s_stream(true) for rkisp1_resizer_selfpath.
> > >
> > > stream_count variable was added in v4l2_subdevice to handle nested calls
> > > to the helpers.
> > > This is useful when the driver allows streaming from more then one
> > > capture device sharing subdevices.
> >
> > If I understand correctly, this isn't  true anymore right? Nested calls aren't
> > possible anymore since this version doesn't contain stream_count in struct v4l2_subdevice.
> >
> > Documentation of v4l2_pipeline_stream_*() should also be updated.
> >
> > Just to be clear, without the nested call, vimc will require to add its own
> > counters, patch https://patchwork.kernel.org/patch/10948833/ will be
> > required again to allow multi streaming.
> >
> > Also, patch "media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable}"
> > is cleaner in the previous version (with stream_count in struct v4l2_subdevice).
> >
> > All drivers that allows multi streaming will need to implement some special handling.
> >
> > Adding stream_count in struct v4l2_subdevice still seems cleaner to me. I'd like to hear
> > what others think.
> 
> I certainly would see this reference counting done in generic code,
> because requiring every driver to do it simply adds to the endless
> amount of boiler plate that V4L2 currently requires from the drivers.
> :(
> 
> I wonder if it wouldn't be possible to redesign the framework so that
> .s_stream() at the subdev level is only called when it's expected to
> either start or stop this particular subdev and driver's
> implementation can simply execute the requested action.

I'd very much like that. Note that I think a few drivers abuse the on
parameter to the function to pass other values than 0 or 1. We'd have to
fix those first (or maybe it has been done already, it's been a long
time since I last checked).

-- 
Regards,

Laurent Pinchart

Re: [PATCH v4 0/5] media: add v4l2_pipeline_stream_{enable,disable}

[Dafna Hirschfeld]

Hi Tomasz, Helen, Laurent

On 26.05.20 20:57, Laurent Pinchart wrote:
> Hi Tomasz,
> 
> On Tue, May 26, 2020 at 06:11:00PM +0200, Tomasz Figa wrote:
>> On Fri, May 22, 2020 at 11:06 AM Helen Koike <helen.koike@collabora.com> wrote:
>>> On 5/22/20 4:55 AM, Dafna Hirschfeld wrote:
>>>> Hi,
>>>> This is v4 of the patchset that was sent by Helen Koike.
>>>>
>>>> Media drivers need to iterate through the pipeline and call .s_stream()
>>>> callbacks in the subdevices.
>>>>
>>>> Instead of repeating code, add helpers for this.
>>>>
>>>> These helpers will go walk through the pipeline only visiting entities
>>>> that participates in the stream, i.e. it follows links from sink to source
>>>> (and not the opposite).
>>>> For example, in a topology like this https://bit.ly/3b2MxjI
>>>> calling v4l2_pipeline_stream_enable() from rkisp1_mainpath won't call
>>>> .s_stream(true) for rkisp1_resizer_selfpath.
>>>>
>>>> stream_count variable was added in v4l2_subdevice to handle nested calls
>>>> to the helpers.
>>>> This is useful when the driver allows streaming from more then one
>>>> capture device sharing subdevices.
>>>
>>> If I understand correctly, this isn't  true anymore right? Nested calls aren't
>>> possible anymore since this version doesn't contain stream_count in struct v4l2_subdevice.
>>>
>>> Documentation of v4l2_pipeline_stream_*() should also be updated.
>>>
>>> Just to be clear, without the nested call, vimc will require to add its own
>>> counters, patch https://patchwork.kernel.org/patch/10948833/ will be
>>> required again to allow multi streaming.
>>>
>>> Also, patch "media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable}"
>>> is cleaner in the previous version (with stream_count in struct v4l2_subdevice).
>>>
>>> All drivers that allows multi streaming will need to implement some special handling.
>>>
>>> Adding stream_count in struct v4l2_subdevice still seems cleaner to me. I'd like to hear
>>> what others think.
>>
>> I certainly would see this reference counting done in generic code,
>> because requiring every driver to do it simply adds to the endless

It is required only for drivers that support multistreaming. I don't know much
about other driver except of the ones I am working on, is it a common case?

>> amount of boiler plate that V4L2 currently requires from the drivers.
>> :(
>>
>> I wonder if it wouldn't be possible to redesign the framework so that
>> .s_stream() at the subdev level is only called when it's expected to
>> either start or stop this particular subdev and driver's
>> implementation can simply execute the requested action.

You mean that a generic code similar to the helper functions in this patchset
will be used for all drivers, so that drivers don't call s_stream for subdevices
anymore?
Anyway, this patchset just adds helper functions, it does not redesign the code.
Maybe the stream_count can be updated in the v4l2_subdev_call macro ?
This why it can be used not just for the helper functions.

Thanks,
Dafna

> 
> I'd very much like that. Note that I think a few drivers abuse the on
> parameter to the function to pass other values than 0 or 1. We'd have to
> fix those first (or maybe it has been done already, it's been a long
> time since I last checked).
> 

Re: [PATCH v4 0/5] media: add v4l2_pipeline_stream_{enable,disable}

[Tomasz Figa]

On Thu, May 28, 2020 at 6:21 PM Dafna Hirschfeld
<dafna.hirschfeld@collabora.com> wrote:
>
> Hi Tomasz, Helen, Laurent
>
> On 26.05.20 20:57, Laurent Pinchart wrote:
> > Hi Tomasz,
> >
> > On Tue, May 26, 2020 at 06:11:00PM +0200, Tomasz Figa wrote:
> >> On Fri, May 22, 2020 at 11:06 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>> On 5/22/20 4:55 AM, Dafna Hirschfeld wrote:
> >>>> Hi,
> >>>> This is v4 of the patchset that was sent by Helen Koike.
> >>>>
> >>>> Media drivers need to iterate through the pipeline and call .s_stream()
> >>>> callbacks in the subdevices.
> >>>>
> >>>> Instead of repeating code, add helpers for this.
> >>>>
> >>>> These helpers will go walk through the pipeline only visiting entities
> >>>> that participates in the stream, i.e. it follows links from sink to source
> >>>> (and not the opposite).
> >>>> For example, in a topology like this https://bit.ly/3b2MxjI
> >>>> calling v4l2_pipeline_stream_enable() from rkisp1_mainpath won't call
> >>>> .s_stream(true) for rkisp1_resizer_selfpath.
> >>>>
> >>>> stream_count variable was added in v4l2_subdevice to handle nested calls
> >>>> to the helpers.
> >>>> This is useful when the driver allows streaming from more then one
> >>>> capture device sharing subdevices.
> >>>
> >>> If I understand correctly, this isn't  true anymore right? Nested calls aren't
> >>> possible anymore since this version doesn't contain stream_count in struct v4l2_subdevice.
> >>>
> >>> Documentation of v4l2_pipeline_stream_*() should also be updated.
> >>>
> >>> Just to be clear, without the nested call, vimc will require to add its own
> >>> counters, patch https://patchwork.kernel.org/patch/10948833/ will be
> >>> required again to allow multi streaming.
> >>>
> >>> Also, patch "media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable}"
> >>> is cleaner in the previous version (with stream_count in struct v4l2_subdevice).
> >>>
> >>> All drivers that allows multi streaming will need to implement some special handling.
> >>>
> >>> Adding stream_count in struct v4l2_subdevice still seems cleaner to me. I'd like to hear
> >>> what others think.
> >>
> >> I certainly would see this reference counting done in generic code,
> >> because requiring every driver to do it simply adds to the endless
>
> It is required only for drivers that support multistreaming. I don't know much
> about other driver except of the ones I am working on, is it a common case?
>

I'm not very familiar with the older camera I/F drivers, but multiple
streams isn't anything unusual for camera hardware. I recall the old
Samsung FIMC already having support for separate preview and capture
outputs.

Also adding the reference counting on framework level probably
wouldn't really hurt drivers which only have 1 stream anyway.

> >> amount of boiler plate that V4L2 currently requires from the drivers.
> >> :(
> >>
> >> I wonder if it wouldn't be possible to redesign the framework so that
> >> .s_stream() at the subdev level is only called when it's expected to
> >> either start or stop this particular subdev and driver's
> >> implementation can simply execute the requested action.
>
> You mean that a generic code similar to the helper functions in this patchset
> will be used for all drivers, so that drivers don't call s_stream for subdevices
> anymore?
> Anyway, this patchset just adds helper functions, it does not redesign the code.
> Maybe the stream_count can be updated in the v4l2_subdev_call macro ?
> This why it can be used not just for the helper functions.

Sorry, just thinking out loud. Generally if we look at other kAPIs,
such as the drm_crtc_helper_funcs [1] or regulator_ops [2], the driver
provided implementation doesn't have to care about duplicate
enable/disable requests.

[1] https://elixir.bootlin.com/linux/v5.7-rc7/source/include/drm/drm_modeset_helper_vtables.h#L61
[2] https://elixir.bootlin.com/linux/v5.7-rc7/source/include/linux/regulator/driver.h#L144

If we could prohibit calling v4l2_subdev_ops::s_stream() by the
drivers directly and instead add something like
v4l2_subdev_s_stream(), the latter could do reference counting on its
own and then only call v4l2_subdev_ops::s_stream() when the reference
count changes between 0 and 1.

One problem I see with this series is that I'm not sure if it's always
guaranteed that all the drivers in the pipeline would actually use the
generic helpers. If there is a driver in the pipeline which just calls
v4l2_subdev_ops::s_stream() on some other subdev on its own, it
wouldn't be aware of the reference count and bad things could happen
(e.g. the subdev stopped despite the count being > 0).

However, I'm afraid this is more of the kAPI design issue and could be
require quite a significant effort to be straightened out.

Best regards,
Tomasz

>
> Thanks,
> Dafna
>
> >
> > I'd very much like that. Note that I think a few drivers abuse the on
> > parameter to the function to pass other values than 0 or 1. We'd have to
> > fix those first (or maybe it has been done already, it's been a long
> > time since I last checked).
> >

Re: [PATCH v4 0/5] media: add v4l2_pipeline_stream_{enable,disable}

[Tomasz Figa]

[Fixing Niklas's address.]

On Fri, May 29, 2020 at 3:26 PM Tomasz Figa <tfiga@chromium.org> wrote:
>
> On Thu, May 28, 2020 at 6:21 PM Dafna Hirschfeld
> <dafna.hirschfeld@collabora.com> wrote:
> >
> > Hi Tomasz, Helen, Laurent
> >
> > On 26.05.20 20:57, Laurent Pinchart wrote:
> > > Hi Tomasz,
> > >
> > > On Tue, May 26, 2020 at 06:11:00PM +0200, Tomasz Figa wrote:
> > >> On Fri, May 22, 2020 at 11:06 AM Helen Koike <helen.koike@collabora.com> wrote:
> > >>> On 5/22/20 4:55 AM, Dafna Hirschfeld wrote:
> > >>>> Hi,
> > >>>> This is v4 of the patchset that was sent by Helen Koike.
> > >>>>
> > >>>> Media drivers need to iterate through the pipeline and call .s_stream()
> > >>>> callbacks in the subdevices.
> > >>>>
> > >>>> Instead of repeating code, add helpers for this.
> > >>>>
> > >>>> These helpers will go walk through the pipeline only visiting entities
> > >>>> that participates in the stream, i.e. it follows links from sink to source
> > >>>> (and not the opposite).
> > >>>> For example, in a topology like this https://bit.ly/3b2MxjI
> > >>>> calling v4l2_pipeline_stream_enable() from rkisp1_mainpath won't call
> > >>>> .s_stream(true) for rkisp1_resizer_selfpath.
> > >>>>
> > >>>> stream_count variable was added in v4l2_subdevice to handle nested calls
> > >>>> to the helpers.
> > >>>> This is useful when the driver allows streaming from more then one
> > >>>> capture device sharing subdevices.
> > >>>
> > >>> If I understand correctly, this isn't  true anymore right? Nested calls aren't
> > >>> possible anymore since this version doesn't contain stream_count in struct v4l2_subdevice.
> > >>>
> > >>> Documentation of v4l2_pipeline_stream_*() should also be updated.
> > >>>
> > >>> Just to be clear, without the nested call, vimc will require to add its own
> > >>> counters, patch https://patchwork.kernel.org/patch/10948833/ will be
> > >>> required again to allow multi streaming.
> > >>>
> > >>> Also, patch "media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable}"
> > >>> is cleaner in the previous version (with stream_count in struct v4l2_subdevice).
> > >>>
> > >>> All drivers that allows multi streaming will need to implement some special handling.
> > >>>
> > >>> Adding stream_count in struct v4l2_subdevice still seems cleaner to me. I'd like to hear
> > >>> what others think.
> > >>
> > >> I certainly would see this reference counting done in generic code,
> > >> because requiring every driver to do it simply adds to the endless
> >
> > It is required only for drivers that support multistreaming. I don't know much
> > about other driver except of the ones I am working on, is it a common case?
> >
>
> I'm not very familiar with the older camera I/F drivers, but multiple
> streams isn't anything unusual for camera hardware. I recall the old
> Samsung FIMC already having support for separate preview and capture
> outputs.
>
> Also adding the reference counting on framework level probably
> wouldn't really hurt drivers which only have 1 stream anyway.
>
> > >> amount of boiler plate that V4L2 currently requires from the drivers.
> > >> :(
> > >>
> > >> I wonder if it wouldn't be possible to redesign the framework so that
> > >> .s_stream() at the subdev level is only called when it's expected to
> > >> either start or stop this particular subdev and driver's
> > >> implementation can simply execute the requested action.
> >
> > You mean that a generic code similar to the helper functions in this patchset
> > will be used for all drivers, so that drivers don't call s_stream for subdevices
> > anymore?
> > Anyway, this patchset just adds helper functions, it does not redesign the code.
> > Maybe the stream_count can be updated in the v4l2_subdev_call macro ?
> > This why it can be used not just for the helper functions.
>
> Sorry, just thinking out loud. Generally if we look at other kAPIs,
> such as the drm_crtc_helper_funcs [1] or regulator_ops [2], the driver
> provided implementation doesn't have to care about duplicate
> enable/disable requests.
>
> [1] https://elixir.bootlin.com/linux/v5.7-rc7/source/include/drm/drm_modeset_helper_vtables.h#L61
> [2] https://elixir.bootlin.com/linux/v5.7-rc7/source/include/linux/regulator/driver.h#L144
>
> If we could prohibit calling v4l2_subdev_ops::s_stream() by the
> drivers directly and instead add something like
> v4l2_subdev_s_stream(), the latter could do reference counting on its
> own and then only call v4l2_subdev_ops::s_stream() when the reference
> count changes between 0 and 1.
>
> One problem I see with this series is that I'm not sure if it's always
> guaranteed that all the drivers in the pipeline would actually use the
> generic helpers. If there is a driver in the pipeline which just calls
> v4l2_subdev_ops::s_stream() on some other subdev on its own, it
> wouldn't be aware of the reference count and bad things could happen
> (e.g. the subdev stopped despite the count being > 0).
>
> However, I'm afraid this is more of the kAPI design issue and could be
> require quite a significant effort to be straightened out.
>
> Best regards,
> Tomasz
>
> >
> > Thanks,
> > Dafna
> >
> > >
> > > I'd very much like that. Note that I think a few drivers abuse the on
> > > parameter to the function to pass other values than 0 or 1. We'd have to
> > > fix those first (or maybe it has been done already, it's been a long
> > > time since I last checked).
> > >

Re: [PATCH v4 0/5] media: add v4l2_pipeline_stream_{enable,disable}

[Helen Koike]

Hi,

On 5/29/20 10:27 AM, Tomasz Figa wrote:
> [Fixing Niklas's address.]
> 
> On Fri, May 29, 2020 at 3:26 PM Tomasz Figa <tfiga@chromium.org> wrote:
>>
>> On Thu, May 28, 2020 at 6:21 PM Dafna Hirschfeld
>> <dafna.hirschfeld@collabora.com> wrote:
>>>
>>> Hi Tomasz, Helen, Laurent
>>>
>>> On 26.05.20 20:57, Laurent Pinchart wrote:
>>>> Hi Tomasz,
>>>>
>>>> On Tue, May 26, 2020 at 06:11:00PM +0200, Tomasz Figa wrote:
>>>>> On Fri, May 22, 2020 at 11:06 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>>> On 5/22/20 4:55 AM, Dafna Hirschfeld wrote:
>>>>>>> Hi,
>>>>>>> This is v4 of the patchset that was sent by Helen Koike.
>>>>>>>
>>>>>>> Media drivers need to iterate through the pipeline and call .s_stream()
>>>>>>> callbacks in the subdevices.
>>>>>>>
>>>>>>> Instead of repeating code, add helpers for this.
>>>>>>>
>>>>>>> These helpers will go walk through the pipeline only visiting entities
>>>>>>> that participates in the stream, i.e. it follows links from sink to source
>>>>>>> (and not the opposite).
>>>>>>> For example, in a topology like this https://bit.ly/3b2MxjI
>>>>>>> calling v4l2_pipeline_stream_enable() from rkisp1_mainpath won't call
>>>>>>> .s_stream(true) for rkisp1_resizer_selfpath.
>>>>>>>
>>>>>>> stream_count variable was added in v4l2_subdevice to handle nested calls
>>>>>>> to the helpers.
>>>>>>> This is useful when the driver allows streaming from more then one
>>>>>>> capture device sharing subdevices.
>>>>>>
>>>>>> If I understand correctly, this isn't  true anymore right? Nested calls aren't
>>>>>> possible anymore since this version doesn't contain stream_count in struct v4l2_subdevice.
>>>>>>
>>>>>> Documentation of v4l2_pipeline_stream_*() should also be updated.
>>>>>>
>>>>>> Just to be clear, without the nested call, vimc will require to add its own
>>>>>> counters, patch https://patchwork.kernel.org/patch/10948833/ will be
>>>>>> required again to allow multi streaming.
>>>>>>
>>>>>> Also, patch "media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable}"
>>>>>> is cleaner in the previous version (with stream_count in struct v4l2_subdevice).
>>>>>>
>>>>>> All drivers that allows multi streaming will need to implement some special handling.
>>>>>>
>>>>>> Adding stream_count in struct v4l2_subdevice still seems cleaner to me. I'd like to hear
>>>>>> what others think.
>>>>>
>>>>> I certainly would see this reference counting done in generic code,
>>>>> because requiring every driver to do it simply adds to the endless
>>>
>>> It is required only for drivers that support multistreaming. I don't know much
>>> about other driver except of the ones I am working on, is it a common case?
>>>
>>
>> I'm not very familiar with the older camera I/F drivers, but multiple
>> streams isn't anything unusual for camera hardware. I recall the old
>> Samsung FIMC already having support for separate preview and capture
>> outputs.
>>
>> Also adding the reference counting on framework level probably
>> wouldn't really hurt drivers which only have 1 stream anyway.
>>
>>>>> amount of boiler plate that V4L2 currently requires from the drivers.
>>>>> :(
>>>>>
>>>>> I wonder if it wouldn't be possible to redesign the framework so that
>>>>> .s_stream() at the subdev level is only called when it's expected to
>>>>> either start or stop this particular subdev and driver's
>>>>> implementation can simply execute the requested action.
>>>
>>> You mean that a generic code similar to the helper functions in this patchset
>>> will be used for all drivers, so that drivers don't call s_stream for subdevices
>>> anymore?
>>> Anyway, this patchset just adds helper functions, it does not redesign the code.
>>> Maybe the stream_count can be updated in the v4l2_subdev_call macro ?
>>> This why it can be used not just for the helper functions.
>>
>> Sorry, just thinking out loud. Generally if we look at other kAPIs,
>> such as the drm_crtc_helper_funcs [1] or regulator_ops [2], the driver
>> provided implementation doesn't have to care about duplicate
>> enable/disable requests.

Thanks for this pointer.

>>
>> [1] https://elixir.bootlin.com/linux/v5.7-rc7/source/include/drm/drm_modeset_helper_vtables.h#L61
>> [2] https://elixir.bootlin.com/linux/v5.7-rc7/source/include/linux/regulator/driver.h#L144
>>
>> If we could prohibit calling v4l2_subdev_ops::s_stream() by the
>> drivers directly and instead add something like
>> v4l2_subdev_s_stream(), the latter could do reference counting on its
>> own and then only call v4l2_subdev_ops::s_stream() when the reference
>> count changes between 0 and 1.

This is basically how v3 was implemented https://patchwork.kernel.org/patch/11489583/

And the main concern (from what I understood) was to add a stream_count
under struct v4l2_subdev, that is only touched by the helpers, so this
stream_count field would be useless for drivers not using the helpers.
which, imho, it is not a big problem.

I think we gain more with a common implementation.

>>
>> One problem I see with this series is that I'm not sure if it's always
>> guaranteed that all the drivers in the pipeline would actually use the
>> generic helpers.

I'm not sure we should always guarantee usage of generic helpers, since
drivers may want to initialize subdevices in a specific order.

>> If there is a driver in the pipeline which just calls
>> v4l2_subdev_ops::s_stream() on some other subdev on its own, it
>> wouldn't be aware of the reference count and bad things could happen
>> (e.g. the subdev stopped despite the count being > 0).

I don't think this is a problem, since v4l2_subdev_ops::s_stream() are
usually triggered by a STREAM_ON on a video node. So or the video node driver
uses the helpers, or it calls v4l2_subdev_ops::s_stream() on subdevices directly.

Unless if, we could have a case where we have multiple video nodes in the
same topology that is implemented by different drivers, which seems odd
to me.

Regards,
Helen

>>
>> However, I'm afraid this is more of the kAPI design issue and could be
>> require quite a significant effort to be straightened out.
>>
>> Best regards,
>> Tomasz
>>
>>>
>>> Thanks,
>>> Dafna
>>>
>>>>
>>>> I'd very much like that. Note that I think a few drivers abuse the on
>>>> parameter to the function to pass other values than 0 or 1. We'd have to
>>>> fix those first (or maybe it has been done already, it's been a long
>>>> time since I last checked).
>>>>

Re: [PATCH v4 0/5] media: add v4l2_pipeline_stream_{enable,disable}

[Tomasz Figa]

On Fri, May 29, 2020 at 3:49 PM Helen Koike <helen.koike@collabora.com> wrote:
>
> Hi,
>
> On 5/29/20 10:27 AM, Tomasz Figa wrote:
> > [Fixing Niklas's address.]
> >
> > On Fri, May 29, 2020 at 3:26 PM Tomasz Figa <tfiga@chromium.org> wrote:
> >>
> >> On Thu, May 28, 2020 at 6:21 PM Dafna Hirschfeld
> >> <dafna.hirschfeld@collabora.com> wrote:
> >>>
> >>> Hi Tomasz, Helen, Laurent
> >>>
> >>> On 26.05.20 20:57, Laurent Pinchart wrote:
> >>>> Hi Tomasz,
> >>>>
> >>>> On Tue, May 26, 2020 at 06:11:00PM +0200, Tomasz Figa wrote:
> >>>>> On Fri, May 22, 2020 at 11:06 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>>> On 5/22/20 4:55 AM, Dafna Hirschfeld wrote:
> >>>>>>> Hi,
> >>>>>>> This is v4 of the patchset that was sent by Helen Koike.
> >>>>>>>
> >>>>>>> Media drivers need to iterate through the pipeline and call .s_stream()
> >>>>>>> callbacks in the subdevices.
> >>>>>>>
> >>>>>>> Instead of repeating code, add helpers for this.
> >>>>>>>
> >>>>>>> These helpers will go walk through the pipeline only visiting entities
> >>>>>>> that participates in the stream, i.e. it follows links from sink to source
> >>>>>>> (and not the opposite).
> >>>>>>> For example, in a topology like this https://bit.ly/3b2MxjI
> >>>>>>> calling v4l2_pipeline_stream_enable() from rkisp1_mainpath won't call
> >>>>>>> .s_stream(true) for rkisp1_resizer_selfpath.
> >>>>>>>
> >>>>>>> stream_count variable was added in v4l2_subdevice to handle nested calls
> >>>>>>> to the helpers.
> >>>>>>> This is useful when the driver allows streaming from more then one
> >>>>>>> capture device sharing subdevices.
> >>>>>>
> >>>>>> If I understand correctly, this isn't  true anymore right? Nested calls aren't
> >>>>>> possible anymore since this version doesn't contain stream_count in struct v4l2_subdevice.
> >>>>>>
> >>>>>> Documentation of v4l2_pipeline_stream_*() should also be updated.
> >>>>>>
> >>>>>> Just to be clear, without the nested call, vimc will require to add its own
> >>>>>> counters, patch https://patchwork.kernel.org/patch/10948833/ will be
> >>>>>> required again to allow multi streaming.
> >>>>>>
> >>>>>> Also, patch "media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable}"
> >>>>>> is cleaner in the previous version (with stream_count in struct v4l2_subdevice).
> >>>>>>
> >>>>>> All drivers that allows multi streaming will need to implement some special handling.
> >>>>>>
> >>>>>> Adding stream_count in struct v4l2_subdevice still seems cleaner to me. I'd like to hear
> >>>>>> what others think.
> >>>>>
> >>>>> I certainly would see this reference counting done in generic code,
> >>>>> because requiring every driver to do it simply adds to the endless
> >>>
> >>> It is required only for drivers that support multistreaming. I don't know much
> >>> about other driver except of the ones I am working on, is it a common case?
> >>>
> >>
> >> I'm not very familiar with the older camera I/F drivers, but multiple
> >> streams isn't anything unusual for camera hardware. I recall the old
> >> Samsung FIMC already having support for separate preview and capture
> >> outputs.
> >>
> >> Also adding the reference counting on framework level probably
> >> wouldn't really hurt drivers which only have 1 stream anyway.
> >>
> >>>>> amount of boiler plate that V4L2 currently requires from the drivers.
> >>>>> :(
> >>>>>
> >>>>> I wonder if it wouldn't be possible to redesign the framework so that
> >>>>> .s_stream() at the subdev level is only called when it's expected to
> >>>>> either start or stop this particular subdev and driver's
> >>>>> implementation can simply execute the requested action.
> >>>
> >>> You mean that a generic code similar to the helper functions in this patchset
> >>> will be used for all drivers, so that drivers don't call s_stream for subdevices
> >>> anymore?
> >>> Anyway, this patchset just adds helper functions, it does not redesign the code.
> >>> Maybe the stream_count can be updated in the v4l2_subdev_call macro ?
> >>> This why it can be used not just for the helper functions.
> >>
> >> Sorry, just thinking out loud. Generally if we look at other kAPIs,
> >> such as the drm_crtc_helper_funcs [1] or regulator_ops [2], the driver
> >> provided implementation doesn't have to care about duplicate
> >> enable/disable requests.
>
> Thanks for this pointer.
>
> >>
> >> [1] https://elixir.bootlin.com/linux/v5.7-rc7/source/include/drm/drm_modeset_helper_vtables.h#L61
> >> [2] https://elixir.bootlin.com/linux/v5.7-rc7/source/include/linux/regulator/driver.h#L144
> >>
> >> If we could prohibit calling v4l2_subdev_ops::s_stream() by the
> >> drivers directly and instead add something like
> >> v4l2_subdev_s_stream(), the latter could do reference counting on its
> >> own and then only call v4l2_subdev_ops::s_stream() when the reference
> >> count changes between 0 and 1.
>
> This is basically how v3 was implemented https://patchwork.kernel.org/patch/11489583/

Not exactly. There are 2 separate problems being addressed here:
1) Iterating over the pipeline and starting streaming on all the entities,
2) Preventing duplicate calls to s_stream().

v3 attempted to address problem 2) by the way of addressing problem
1). However problem 2) can also occur on its own, outside of the
pipeline start/stop, because s_stream is a standalone subdev
operation, exposed both to the userspace via UAPI and to the drivers
via the pseudo-kAPI (v4l2_subdev_call()). If 2) was solved on the
level of the kAPI, i.e. removing v4l2_subdev_call() and replacing it
with dedicated functions for operations like s_stream, refcounting
could be implemented inside of such functions, without the need to do
it in the pipeline iteration helpers.

>
> And the main concern (from what I understood) was to add a stream_count
> under struct v4l2_subdev, that is only touched by the helpers, so this
> stream_count field would be useless for drivers not using the helpers.
> which, imho, it is not a big problem.

I believe that's exactly because the two problems I mentioned above
are actually separate problems and a solution for 1) can't solve 2)
fully.

>
> I think we gain more with a common implementation.
>
> >>
> >> One problem I see with this series is that I'm not sure if it's always
> >> guaranteed that all the drivers in the pipeline would actually use the
> >> generic helpers.
>
> I'm not sure we should always guarantee usage of generic helpers, since
> drivers may want to initialize subdevices in a specific order.
>

If we treat the 2 problems separately, the helpers for 1) could be
still optional, but the new calls for 2) would be mandatory.

> >> If there is a driver in the pipeline which just calls
> >> v4l2_subdev_ops::s_stream() on some other subdev on its own, it
> >> wouldn't be aware of the reference count and bad things could happen
> >> (e.g. the subdev stopped despite the count being > 0).
>
> I don't think this is a problem, since v4l2_subdev_ops::s_stream() are
> usually triggered by a STREAM_ON on a video node. So or the video node driver
> uses the helpers, or it calls v4l2_subdev_ops::s_stream() on subdevices directly.
>

That's not a kAPI guarantee. Any driver is free to call
v4l2_subdev_call(..., s_stream) whenever it wants.

> Unless if, we could have a case where we have multiple video nodes in the
> same topology that is implemented by different drivers, which seems odd
> to me.

That's not the only case. There are some devices, such as the adv748x
CSI-2 transmitter which manage the rest of the pipeline on their own,
e.g.

https://elixir.bootlin.com/linux/v5.7-rc7/source/drivers/media/i2c/adv748x/adv748x-csi2.c#L116

In this case, the generic helpers invoked by the ISP driver would
compete with the explicit configuration done by the driver. However,
if we solve the problem 2) at the kAPI level, that wouldn't be an
issue anymore, because the generic helpers and explicit calls would
simply grab additional reference counts.

Best regards,
Tomasz

>
> Regards,
> Helen
>
> >>
> >> However, I'm afraid this is more of the kAPI design issue and could be
> >> require quite a significant effort to be straightened out.
> >>
> >> Best regards,
> >> Tomasz
> >>
> >>>
> >>> Thanks,
> >>> Dafna
> >>>
> >>>>
> >>>> I'd very much like that. Note that I think a few drivers abuse the on
> >>>> parameter to the function to pass other values than 0 or 1. We'd have to
> >>>> fix those first (or maybe it has been done already, it's been a long
> >>>> time since I last checked).
> >>>>

Re: [PATCH v4 3/5] media: staging: rkisp1: validate links before powering and streaming

[Tomasz Figa]

On Fri, May 22, 2020 at 09:55:20AM +0200, Dafna Hirschfeld wrote:
> In function rkisp1_vb2_start_streaming, the call to
> media_pipeline_start should be the first thing in order
> to validate the links and prevents their state from being modified
> before power up and streaming.
> 
> Adjust stop streaming to the same logic, call media_pipeline_stop
> after we disable streaming on the entities in the topology.
> 
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> ---
>  drivers/staging/media/rkisp1/rkisp1-capture.c | 21 ++++++++++---------
>  1 file changed, 11 insertions(+), 10 deletions(-)
> 

Reviewed-by: Tomasz Figa <tfiga@chromium.org>

Best regards,
Tomasz

Re: [PATCH v4 4/5] media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable} helpers

[Tomasz Figa]

Hi Dafna,

On Fri, May 22, 2020 at 09:55:21AM +0200, Dafna Hirschfeld wrote:
> From: Helen Koike <helen.koike@collabora.com>
> 
> Use v4l2_pipeline_stream_{enable,disable} to call .s_stream()
> subdevice callbacks through the pipeline.
> Those helpers are called only if the other capture is not streaming.
> 
> If the other capture is streaming then he already did that for us
> so we call s_stream only on the resizer that is connected to the
> capture node.
> 
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> ---
>  drivers/staging/media/rkisp1/rkisp1-capture.c | 104 ++++++------------
>  1 file changed, 32 insertions(+), 72 deletions(-)
> 

Thank you for the patch. Please see my comments inline.

[snip]
> +static int rkisp1_s_stream_subdev(struct rkisp1_capture *cap, int enable)
> +{
> +	struct rkisp1_device *rkisp1 = cap->rkisp1;
> +	struct rkisp1_capture *other = &rkisp1->capture_devs[cap->id ^ 1];
> +	int ret;
> +
> +	/*
> +	 * if the other capture is already streaming then we only need to
> +	 * call s_stream of our reszier
> +	 */
> +	if (other->is_streaming) {
> +		struct v4l2_subdev *rsz_sd  = &rkisp1->resizer_devs[cap->id].sd;
> +
> +		ret = v4l2_subdev_call(rsz_sd, video, s_stream, enable);
> +		if (ret && ret != -ENOIOCTLCMD)
> +			dev_err(rkisp1->dev,
> +				"stream %s resizer '%s' failed (%d)\n",
> +				enable ? "on" : "off", rsz_sd->name, ret);

Do we need this special case? Wouldn't v4l2_pipeline_stream_*() simply
increment reference counters for the other entities?

> +	} else {
> +		if (enable)
> +			ret = v4l2_pipeline_stream_enable(&cap->vnode.vdev);
> +		else
> +			ret = v4l2_pipeline_stream_disable(&cap->vnode.vdev);

I wonder if this doesn't ask for just making the helper
v4l2_pipeline_s_stream(..., int enable).

Best regards,
Tomasz

Re: [PATCH v4 4/5] media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable} helpers

[Dafna Hirschfeld]

On 10.06.20 19:03, Tomasz Figa wrote:
> Hi Dafna,
> 
> On Fri, May 22, 2020 at 09:55:21AM +0200, Dafna Hirschfeld wrote:
>> From: Helen Koike <helen.koike@collabora.com>
>>
>> Use v4l2_pipeline_stream_{enable,disable} to call .s_stream()
>> subdevice callbacks through the pipeline.
>> Those helpers are called only if the other capture is not streaming.
>>
>> If the other capture is streaming then he already did that for us
>> so we call s_stream only on the resizer that is connected to the
>> capture node.
>>
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>> ---
>>   drivers/staging/media/rkisp1/rkisp1-capture.c | 104 ++++++------------
>>   1 file changed, 32 insertions(+), 72 deletions(-)
>>
> 
> Thank you for the patch. Please see my comments inline.
> 
> [snip]
>> +static int rkisp1_s_stream_subdev(struct rkisp1_capture *cap, int enable)
>> +{
>> +	struct rkisp1_device *rkisp1 = cap->rkisp1;
>> +	struct rkisp1_capture *other = &rkisp1->capture_devs[cap->id ^ 1];
>> +	int ret;
>> +
>> +	/*
>> +	 * if the other capture is already streaming then we only need to
>> +	 * call s_stream of our reszier
>> +	 */
>> +	if (other->is_streaming) {
>> +		struct v4l2_subdev *rsz_sd  = &rkisp1->resizer_devs[cap->id].sd;
>> +
>> +		ret = v4l2_subdev_call(rsz_sd, video, s_stream, enable);
>> +		if (ret && ret != -ENOIOCTLCMD)
>> +			dev_err(rkisp1->dev,
>> +				"stream %s resizer '%s' failed (%d)\n",
>> +				enable ? "on" : "off", rsz_sd->name, ret);
> 
> Do we need this special case? Wouldn't v4l2_pipeline_stream_*() simply
> increment reference counters for the other entities?

I removed the stream count in v4 of the patchset since I thought it
might be problematic/confusing to add a field "stream_count" in
"struct v4l2_subdev" that is used and updated only by those helper functions

What do you think?

There is also the issue that both you and Sakari Ailus mentioned that
an isp driver can't know the subtopology of a sensor driver and how it handle the
s_stream callback on it's entities.

Thanks,
Dafna


> 
>> +	} else {
>> +		if (enable)
>> +			ret = v4l2_pipeline_stream_enable(&cap->vnode.vdev);
>> +		else
>> +			ret = v4l2_pipeline_stream_disable(&cap->vnode.vdev);
> 
> I wonder if this doesn't ask for just making the helper
> v4l2_pipeline_s_stream(..., int enable).> 
> Best regards,
> Tomasz
> 

Re: [PATCH v4 4/5] media: staging: rkisp1: cap: use v4l2_pipeline_stream_{enable,disable} helpers

[Tomasz Figa]

On Wed, Jun 10, 2020 at 07:22:04PM +0200, Dafna Hirschfeld wrote:
> 
> 
> On 10.06.20 19:03, Tomasz Figa wrote:
> > Hi Dafna,
> > 
> > On Fri, May 22, 2020 at 09:55:21AM +0200, Dafna Hirschfeld wrote:
> > > From: Helen Koike <helen.koike@collabora.com>
> > > 
> > > Use v4l2_pipeline_stream_{enable,disable} to call .s_stream()
> > > subdevice callbacks through the pipeline.
> > > Those helpers are called only if the other capture is not streaming.
> > > 
> > > If the other capture is streaming then he already did that for us
> > > so we call s_stream only on the resizer that is connected to the
> > > capture node.
> > > 
> > > Signed-off-by: Helen Koike <helen.koike@collabora.com>
> > > Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> > > ---
> > >   drivers/staging/media/rkisp1/rkisp1-capture.c | 104 ++++++------------
> > >   1 file changed, 32 insertions(+), 72 deletions(-)
> > > 
> > 
> > Thank you for the patch. Please see my comments inline.
> > 
> > [snip]
> > > +static int rkisp1_s_stream_subdev(struct rkisp1_capture *cap, int enable)
> > > +{
> > > +	struct rkisp1_device *rkisp1 = cap->rkisp1;
> > > +	struct rkisp1_capture *other = &rkisp1->capture_devs[cap->id ^ 1];
> > > +	int ret;
> > > +
> > > +	/*
> > > +	 * if the other capture is already streaming then we only need to
> > > +	 * call s_stream of our reszier
> > > +	 */
> > > +	if (other->is_streaming) {
> > > +		struct v4l2_subdev *rsz_sd  = &rkisp1->resizer_devs[cap->id].sd;
> > > +
> > > +		ret = v4l2_subdev_call(rsz_sd, video, s_stream, enable);
> > > +		if (ret && ret != -ENOIOCTLCMD)
> > > +			dev_err(rkisp1->dev,
> > > +				"stream %s resizer '%s' failed (%d)\n",
> > > +				enable ? "on" : "off", rsz_sd->name, ret);
> > 
> > Do we need this special case? Wouldn't v4l2_pipeline_stream_*() simply
> > increment reference counters for the other entities?
> 
> I removed the stream count in v4 of the patchset since I thought it
> might be problematic/confusing to add a field "stream_count" in
> "struct v4l2_subdev" that is used and updated only by those helper functions
> 
> What do you think?
> 
> There is also the issue that both you and Sakari Ailus mentioned that
> an isp driver can't know the subtopology of a sensor driver and how it handle the
> s_stream callback on it's entities.

Ah, okay, so we settled on removing the refcounting from the helpers.
Fair enough. Sorry for the noise.

Feel free to add my Reviewed-by.

Best regards,
Tomasz

Re: [PATCH v4 2/5] media: v4l2-common: add helper functions to call s_stream() callbacks

[Hans Verkuil]

On 22/05/2020 09:55, Dafna Hirschfeld wrote:
> From: Helen Koike <helen.koike@collabora.com>
> 
> Add v4l2_pipeline_stream_{enable,disable} helper functions to iterate
> through the subdevices in a given stream (i.e following links from sink
> to source) and call .s_stream() callback.
> 
> If .s_stream(true) fails, call .s_stream(false) in the reverse order.
> 
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> ---
>  drivers/media/v4l2-core/v4l2-common.c | 99 +++++++++++++++++++++++++++
>  include/media/v4l2-common.h           | 39 +++++++++++
>  2 files changed, 138 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-common.c b/drivers/media/v4l2-core/v4l2-common.c
> index 9e8eb45a5b03..734db2bf5ca9 100644
> --- a/drivers/media/v4l2-core/v4l2-common.c
> +++ b/drivers/media/v4l2-core/v4l2-common.c
> @@ -442,3 +442,102 @@ int v4l2_fill_pixfmt(struct v4l2_pix_format *pixfmt, u32 pixelformat,
>  	return 0;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_fill_pixfmt);
> +
> +#if defined(CONFIG_MEDIA_CONTROLLER)
> +
> +/*
> + * v4l2_pipeline_subdevs_get - Assemble a list of subdevices in a pipeline
> + * @subdevs: the array to be filled.
> + * @max_size: max number of elements that can fit in subdevs array.
> + *
> + * Walk from a video node, following links from sink to source and fill the
> + * array with subdevices in the path.
> + * The walker performs a depth-first traversal, which means that, in a topology
> + * sd1->sd2->sd3->vdev, sd1 will be the first element placed in the array.
> + *
> + * Return the number of subdevices filled in the array.
> + */
> +static int v4l2_pipeline_subdevs_get(struct video_device *vdev,
> +				     struct v4l2_subdev **subdevs,
> +				     unsigned int max_size)
> +{
> +	struct media_entity *entity = &vdev->entity;
> +	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_graph graph;
> +	int idx = 0;
> +	int ret;
> +
> +	ret = media_graph_walk_init(&graph, mdev);
> +	if (ret)
> +		return ret;
> +
> +	media_graph_walk_start(&graph, entity);
> +
> +	while ((entity = media_graph_walk_next_stream(&graph))) {
> +		if (!is_media_entity_v4l2_subdev(entity))
> +			continue;
> +		if (WARN_ON(idx >= max_size))
> +			return -EINVAL;
> +		subdevs[idx++] = media_entity_to_v4l2_subdev(entity);
> +	}
> +
> +	media_graph_walk_cleanup(&graph);
> +
> +	return idx;
> +}
> +
> +static void v4l2_subdevs_stream_disable(struct v4l2_subdev **subdevs, int size)
> +{
> +	int i;
> +
> +	for (i = 0; i < size; i++) {
> +		struct v4l2_subdev *sd = subdevs[i];
> +
> +		dev_dbg(sd->dev,
> +			"disabling stream for '%s'\n", sd->entity.name);
> +		v4l2_subdev_call(sd, video, s_stream, false);
> +	}
> +}
> +
> +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev)
> +{
> +	struct media_device *mdev = vdev->entity.graph_obj.mdev;
> +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
> +	struct v4l2_subdev *sd;
> +	int i, size, ret;
> +
> +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
> +	if (size <= 0)
> +		return size;
> +
> +	/* Call s_stream() in reverse order to enable sensors last */
> +	for (i = size - 1; i >= 0; i--) {
> +		sd = subdevs[i];
> +		dev_dbg(mdev->dev,
> +			"enabling stream for '%s'\n", sd->entity.name);
> +		ret = v4l2_subdev_call(sd, video, s_stream, true);
> +		if (ret && ret != -ENOIOCTLCMD) {
> +			v4l2_subdevs_stream_disable(subdevs + i + 1,
> +						    size - i - 1);
> +			return ret;
> +		}
> +	}
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_enable);
> +
> +int v4l2_pipeline_stream_disable(struct video_device *vdev)
> +{
> +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
> +	int size;
> +
> +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
> +	if (size < 0)
> +		return size;
> +
> +	v4l2_subdevs_stream_disable(subdevs, size);
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_disable);
> +
> +#endif /* CONFIG_MEDIA_CONTROLLER */
> diff --git a/include/media/v4l2-common.h b/include/media/v4l2-common.h
> index 150ee16ebd81..a278bcf3c5bc 100644
> --- a/include/media/v4l2-common.h
> +++ b/include/media/v4l2-common.h
> @@ -539,4 +539,43 @@ static inline void v4l2_buffer_set_timestamp(struct v4l2_buffer *buf,
>  	buf->timestamp.tv_usec = ts.tv_nsec / NSEC_PER_USEC;
>  }
>  
> +#if defined(CONFIG_MEDIA_CONTROLLER)
> +
> +/**
> + * v4l2_pipeline_stream_enable - Call .s_stream(true) callbacks in the stream
> + * @vdev: Starting video device.
> + *
> + * Call .s_stream(true) callback in all the subdevices participating in the
> + * stream, i.e. following links from sink to source.
> + *
> + * .s_stream(true) is also called from sink to source, i.e. in a topology
> + * sd1->sd2->sd3->vdev, .s_stream(true) of sd3 is called first.
> + *
> + * Calls to this function can be nested, in which case the same number of

Since there is no refcounting (as opposed to v3) you cannot nest these calls,
so the comments for both these functions should be updated.

> + * v4l2_pipeline_stream_disable() calls will be required to disable streaming
> + * through subdevices in the pipeline.
> + * The  pipeline pointer must be identical for all nested calls to
> + * v4l2_pipeline_stream_enable().
> + */
> +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev);
> +
> +/**
> + * v4l2_pipeline_stream_disable - Call .s_stream(false) callbacks in the stream
> + * @vdev: Starting video device.
> + *
> + * Call .s_stream(false) callback in all the subdevices participating in the
> + * stream, i.e. following links from sink to source.
> + *
> + * s_stream(false) is called in a reverse order from
> + * v4l2_pipeline_stream_enable(), i.e. in a topology sd1->sd2->sd3->vdev,
> + * .s_stream(false) of sd1 is called first.
> + *
> + * If multiple calls to v4l2_pipeline_stream_enable() have been made, the same
> + * number of calls to this function are required to disable streaming through
> + * subdevices in the pipeline.
> + */
> +int v4l2_pipeline_stream_disable(struct video_device *vdev);

I'm inclined to make this a void function. Typically a 'disable' function should
always succeed because there really is not much you can do if it fails.

If v4l2_pipeline_subdevs_get() fails, then just WARN_ON. This really should not
happen.

Another problem (Tomasz described that in one of his replies) is with drivers like
adv748x that manage the rest of the pipeline on their own.

The problem is that (AFAIK) you can't see this when walking the graph. What you
would like to know is if an entity controls the upstream entities by having that
entity set a flag or something. Then these pipeline functions would stop at that
entity and let that entity do the work for the upstream entities.

If it is not too much work to support this, then it would be nice to have this.
If it is a lot of work, then it should be documented that these functions are
likely not suitable for such pipelines.

I was also wondering how this function relates to media_pipeline_start(). Which
function should be called first (I think media_pipeline_start() has to come before
v4l2_pipeline_stream_enable(), although patch 4/5 appears to do the opposite).
And I was also wondering if the stream_count (incremented by media_pipeline_start())
is something that v4l2_pipeline_stream_enable/disable could use: i.e. s_stream only
needs to be called if stream_count == 1.

In fact, perhaps v4l2_pipeline_stream_enable/disable should call __media_pipeline_start/stop?

Regards,

	Hans

> +
> +#endif /* CONFIG_MEDIA_CONTROLLER */
> +
>  #endif /* V4L2_COMMON_H_ */
> 

Re: [PATCH v4 2/5] media: v4l2-common: add helper functions to call s_stream() callbacks

[Sakari Ailus]

Hi Dafna,

Apologies for the late reply...

On Mon, May 25, 2020 at 12:45:56PM +0200, Dafna Hirschfeld wrote:
> 
> 
> On 25.05.20 12:03, Sakari Ailus wrote:
> > Hi Dafna,
> > 
> > On Mon, May 25, 2020 at 11:42:37AM +0200, Dafna Hirschfeld wrote:
> > > 
> > > 
> > > On 25.05.20 11:23, Sakari Ailus wrote:
> > > > Hi Dafna,
> > > > 
> > > > My apologies for not reviewing this earlier.
> > > No problem :)
> > > 
> > > > 
> > > > On Fri, May 22, 2020 at 09:55:19AM +0200, Dafna Hirschfeld wrote:
> > > > > From: Helen Koike <helen.koike@collabora.com>
> > > > > 
> > > > > Add v4l2_pipeline_stream_{enable,disable} helper functions to iterate
> > > > > through the subdevices in a given stream (i.e following links from sink
> > > > > to source) and call .s_stream() callback.
> > > > > 
> > > > > If .s_stream(true) fails, call .s_stream(false) in the reverse order.
> > > > > 
> > > > > Signed-off-by: Helen Koike <helen.koike@collabora.com>
> > > > > Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> > > > > ---
> > > > >    drivers/media/v4l2-core/v4l2-common.c | 99 +++++++++++++++++++++++++++
> > > > >    include/media/v4l2-common.h           | 39 +++++++++++
> > > > >    2 files changed, 138 insertions(+)
> > > > > 
> > > > > diff --git a/drivers/media/v4l2-core/v4l2-common.c b/drivers/media/v4l2-core/v4l2-common.c
> > > > > index 9e8eb45a5b03..734db2bf5ca9 100644
> > > > > --- a/drivers/media/v4l2-core/v4l2-common.c
> > > > > +++ b/drivers/media/v4l2-core/v4l2-common.c
> > > > > @@ -442,3 +442,102 @@ int v4l2_fill_pixfmt(struct v4l2_pix_format *pixfmt, u32 pixelformat,
> > > > >    	return 0;
> > > > >    }
> > > > >    EXPORT_SYMBOL_GPL(v4l2_fill_pixfmt);
> > > > > +
> > > > > +#if defined(CONFIG_MEDIA_CONTROLLER)
> > > > > +
> > > > > +/*
> > > > > + * v4l2_pipeline_subdevs_get - Assemble a list of subdevices in a pipeline
> > > > > + * @subdevs: the array to be filled.
> > > > > + * @max_size: max number of elements that can fit in subdevs array.
> > > > > + *
> > > > > + * Walk from a video node, following links from sink to source and fill the
> > > > > + * array with subdevices in the path.
> > > > > + * The walker performs a depth-first traversal, which means that, in a topology
> > > > > + * sd1->sd2->sd3->vdev, sd1 will be the first element placed in the array.
> > > > > + *
> > > > > + * Return the number of subdevices filled in the array.
> > > > > + */
> > > > > +static int v4l2_pipeline_subdevs_get(struct video_device *vdev,
> > > > > +				     struct v4l2_subdev **subdevs,
> > > > > +				     unsigned int max_size)
> > > > > +{
> > > > > +	struct media_entity *entity = &vdev->entity;
> > > > > +	struct media_device *mdev = entity->graph_obj.mdev;
> > > > > +	struct media_graph graph;
> > > > > +	int idx = 0;
> > > > > +	int ret;
> > > > > +
> > > > > +	ret = media_graph_walk_init(&graph, mdev);
> > > > > +	if (ret)
> > > > > +		return ret;
> > > > > +
> > > > > +	media_graph_walk_start(&graph, entity);
> > > > > +
> > > > > +	while ((entity = media_graph_walk_next_stream(&graph))) {
> > > > > +		if (!is_media_entity_v4l2_subdev(entity))
> > > > > +			continue;
> > > > > +		if (WARN_ON(idx >= max_size))
> > > > > +			return -EINVAL;
> > > > > +		subdevs[idx++] = media_entity_to_v4l2_subdev(entity);
> > > > > +	}
> > > > > +
> > > > > +	media_graph_walk_cleanup(&graph);
> > > > > +
> > > > > +	return idx;
> > > > > +}
> > > > > +
> > > > > +static void v4l2_subdevs_stream_disable(struct v4l2_subdev **subdevs, int size)
> > > > > +{
> > > > > +	int i;
> > > > > +
> > > > > +	for (i = 0; i < size; i++) {
> > > > > +		struct v4l2_subdev *sd = subdevs[i];
> > > > > +
> > > > > +		dev_dbg(sd->dev,
> > > > > +			"disabling stream for '%s'\n", sd->entity.name);
> > > > > +		v4l2_subdev_call(sd, video, s_stream, false);
> > > > > +	}
> > > > > +}
> > > > > +
> > > > > +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev)
> > > > > +{
> > > > > +	struct media_device *mdev = vdev->entity.graph_obj.mdev;
> > > > > +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
> > > > > +	struct v4l2_subdev *sd;
> > > > > +	int i, size, ret;
> > > > > +
> > > > > +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
> > > > > +	if (size <= 0)
> > > > > +		return size;
> > > > > +
> > > > > +	/* Call s_stream() in reverse order to enable sensors last */
> > > > > +	for (i = size - 1; i >= 0; i--) {
> > > > > +		sd = subdevs[i];
> > > > > +		dev_dbg(mdev->dev,
> > > > > +			"enabling stream for '%s'\n", sd->entity.name);
> > > > > +		ret = v4l2_subdev_call(sd, video, s_stream, true);
> > > > 
> > > > The s_stream callback is only called on the very next sub-device upstream
> > > > in the pipeline, not on any further sub-devices. This is because a driver
> > > > for the device knows best in which order things need to be set up.
> > > > 
> > > This is only a helper function, drivers can choose not to use it.
> > > In many cases the same driver implements many subdevices so the driver
> > > knows what should be done also for subdevices deeper in the stream.
> > 
> > Can it be used on devices that do not operate from memory to memory? I.e.
> > how do you ensure the s_stream is not called on further sub-devices than
> > those that are adjacent to what this driver controls?

> Oh, I think I see your point, for example in case of an isp driver and a
> sensor driver. The sensor driver can also implement several subdevices
> that the isp driver is not aware of and it is the responsibility of the
> sensor driver to call them. Is this a common case? Still there are the

Yes. For a sensor driver it might not matter much though, as it's just a
single device. But on pipelines with e.g. CSI-2 to parallel converters and
CSI-2 aggregators it will make a big difference.

> code parts in drivers that are implementing calling s_stream in a loop
> similar to those in the helper functions. Maybe they are "buggy"? For
> example this code part:

Yes. Please see the omap3isp or ipu3-cio2 driver, both do that correctly.
Well, for iou3-cio2 this is easy as the driver itself exposes just a single
sub-device.

> https://git.linuxtv.org/media_tree.git/tree/drivers/media/platform/qcom/camss/camss-video.c#n430
> is called by the `start_stream` of a video device and I see that the
> `camss_probe` function that register it also calls
> v4l2_async_notifier_register, which means there might have a subdevice
> from a different driver.
> 
> Anyway, for the rkisp1 it is probably problematic since it is connected to a sensor
> implemented by another driver.
> > 
> > One option could be to check sd->dev and skip devices that are further away
> > but for that you'd also need to know how these sub-devices have been
> > reached. The graph walk does not currently provide this information.
> I think it is possible to use the sd->dev (or maybe sd-owner?)
> since we always only go down the stream, so when we reach a subdevice from
> a different driver we can stop. (not sure though)

Please use sd->dev as the omap3isp does.

-- 
Sakari Ailus

Re: [PATCH v4 2/5] media: v4l2-common: add helper functions to call s_stream() callbacks

[Dafna Hirschfeld]

On 22.06.20 11:00, Hans Verkuil wrote:
> On 22/05/2020 09:55, Dafna Hirschfeld wrote:
>> From: Helen Koike <helen.koike@collabora.com>
>>
>> Add v4l2_pipeline_stream_{enable,disable} helper functions to iterate
>> through the subdevices in a given stream (i.e following links from sink
>> to source) and call .s_stream() callback.
>>
>> If .s_stream(true) fails, call .s_stream(false) in the reverse order.
>>
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-common.c | 99 +++++++++++++++++++++++++++
>>   include/media/v4l2-common.h           | 39 +++++++++++
>>   2 files changed, 138 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-common.c b/drivers/media/v4l2-core/v4l2-common.c
>> index 9e8eb45a5b03..734db2bf5ca9 100644
>> --- a/drivers/media/v4l2-core/v4l2-common.c
>> +++ b/drivers/media/v4l2-core/v4l2-common.c
>> @@ -442,3 +442,102 @@ int v4l2_fill_pixfmt(struct v4l2_pix_format *pixfmt, u32 pixelformat,
>>   	return 0;
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_fill_pixfmt);
>> +
>> +#if defined(CONFIG_MEDIA_CONTROLLER)
>> +
>> +/*
>> + * v4l2_pipeline_subdevs_get - Assemble a list of subdevices in a pipeline
>> + * @subdevs: the array to be filled.
>> + * @max_size: max number of elements that can fit in subdevs array.
>> + *
>> + * Walk from a video node, following links from sink to source and fill the
>> + * array with subdevices in the path.
>> + * The walker performs a depth-first traversal, which means that, in a topology
>> + * sd1->sd2->sd3->vdev, sd1 will be the first element placed in the array.
>> + *
>> + * Return the number of subdevices filled in the array.
>> + */
>> +static int v4l2_pipeline_subdevs_get(struct video_device *vdev,
>> +				     struct v4l2_subdev **subdevs,
>> +				     unsigned int max_size)
>> +{
>> +	struct media_entity *entity = &vdev->entity;
>> +	struct media_device *mdev = entity->graph_obj.mdev;
>> +	struct media_graph graph;
>> +	int idx = 0;
>> +	int ret;
>> +
>> +	ret = media_graph_walk_init(&graph, mdev);
>> +	if (ret)
>> +		return ret;
>> +
>> +	media_graph_walk_start(&graph, entity);
>> +
>> +	while ((entity = media_graph_walk_next_stream(&graph))) {
>> +		if (!is_media_entity_v4l2_subdev(entity))
>> +			continue;
>> +		if (WARN_ON(idx >= max_size))
>> +			return -EINVAL;
>> +		subdevs[idx++] = media_entity_to_v4l2_subdev(entity);
>> +	}
>> +
>> +	media_graph_walk_cleanup(&graph);
>> +
>> +	return idx;
>> +}
>> +
>> +static void v4l2_subdevs_stream_disable(struct v4l2_subdev **subdevs, int size)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < size; i++) {
>> +		struct v4l2_subdev *sd = subdevs[i];
>> +
>> +		dev_dbg(sd->dev,
>> +			"disabling stream for '%s'\n", sd->entity.name);
>> +		v4l2_subdev_call(sd, video, s_stream, false);
>> +	}
>> +}
>> +
>> +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev)
>> +{
>> +	struct media_device *mdev = vdev->entity.graph_obj.mdev;
>> +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
>> +	struct v4l2_subdev *sd;
>> +	int i, size, ret;
>> +
>> +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
>> +	if (size <= 0)
>> +		return size;
>> +
>> +	/* Call s_stream() in reverse order to enable sensors last */
>> +	for (i = size - 1; i >= 0; i--) {
>> +		sd = subdevs[i];
>> +		dev_dbg(mdev->dev,
>> +			"enabling stream for '%s'\n", sd->entity.name);
>> +		ret = v4l2_subdev_call(sd, video, s_stream, true);
>> +		if (ret && ret != -ENOIOCTLCMD) {
>> +			v4l2_subdevs_stream_disable(subdevs + i + 1,
>> +						    size - i - 1);
>> +			return ret;
>> +		}
>> +	}
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_enable);
>> +
>> +int v4l2_pipeline_stream_disable(struct video_device *vdev)
>> +{
>> +	struct v4l2_subdev *subdevs[MEDIA_ENTITY_ENUM_MAX_DEPTH];
>> +	int size;
>> +
>> +	size = v4l2_pipeline_subdevs_get(vdev, subdevs, ARRAY_SIZE(subdevs));
>> +	if (size < 0)
>> +		return size;
>> +
>> +	v4l2_subdevs_stream_disable(subdevs, size);
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_pipeline_stream_disable);
>> +
>> +#endif /* CONFIG_MEDIA_CONTROLLER */
>> diff --git a/include/media/v4l2-common.h b/include/media/v4l2-common.h
>> index 150ee16ebd81..a278bcf3c5bc 100644
>> --- a/include/media/v4l2-common.h
>> +++ b/include/media/v4l2-common.h
>> @@ -539,4 +539,43 @@ static inline void v4l2_buffer_set_timestamp(struct v4l2_buffer *buf,
>>   	buf->timestamp.tv_usec = ts.tv_nsec / NSEC_PER_USEC;
>>   }
>>   
>> +#if defined(CONFIG_MEDIA_CONTROLLER)
>> +
>> +/**
>> + * v4l2_pipeline_stream_enable - Call .s_stream(true) callbacks in the stream
>> + * @vdev: Starting video device.
>> + *
>> + * Call .s_stream(true) callback in all the subdevices participating in the
>> + * stream, i.e. following links from sink to source.
>> + *
>> + * .s_stream(true) is also called from sink to source, i.e. in a topology
>> + * sd1->sd2->sd3->vdev, .s_stream(true) of sd3 is called first.
>> + *
>> + * Calls to this function can be nested, in which case the same number of
> 
> Since there is no refcounting (as opposed to v3) you cannot nest these calls,
> so the comments for both these functions should be updated.
> 
>> + * v4l2_pipeline_stream_disable() calls will be required to disable streaming
>> + * through subdevices in the pipeline.
>> + * The  pipeline pointer must be identical for all nested calls to
>> + * v4l2_pipeline_stream_enable().
>> + */
>> +__must_check int v4l2_pipeline_stream_enable(struct video_device *vdev);
>> +
>> +/**
>> + * v4l2_pipeline_stream_disable - Call .s_stream(false) callbacks in the stream
>> + * @vdev: Starting video device.
>> + *
>> + * Call .s_stream(false) callback in all the subdevices participating in the
>> + * stream, i.e. following links from sink to source.
>> + *
>> + * s_stream(false) is called in a reverse order from
>> + * v4l2_pipeline_stream_enable(), i.e. in a topology sd1->sd2->sd3->vdev,
>> + * .s_stream(false) of sd1 is called first.
>> + *
>> + * If multiple calls to v4l2_pipeline_stream_enable() have been made, the same
>> + * number of calls to this function are required to disable streaming through
>> + * subdevices in the pipeline.
>> + */
>> +int v4l2_pipeline_stream_disable(struct video_device *vdev);
> 
> I'm inclined to make this a void function. Typically a 'disable' function should
> always succeed because there really is not much you can do if it fails.
> 
> If v4l2_pipeline_subdevs_get() fails, then just WARN_ON. This really should not
> happen.
> 
> Another problem (Tomasz described that in one of his replies) is with drivers like
> adv748x that manage the rest of the pipeline on their own.
> 
> The problem is that (AFAIK) you can't see this when walking the graph. What you
> would like to know is if an entity controls the upstream entities by having that
> entity set a flag or something. Then these pipeline functions would stop at that
> entity and let that entity do the work for the upstream entities.
> 
> If it is not too much work to support this, then it would be nice to have this.

Hi, I can do it using the sd->dev field like Sakari suggested.
So the loop iterate only until sd->dev changes.

> If it is a lot of work, then it should be documented that these functions are
> likely not suitable for such pipelines.
> 
> I was also wondering how this function relates to media_pipeline_start(). Which
> function should be called first (I think media_pipeline_start() has to come before
> v4l2_pipeline_stream_enable(), although patch 4/5 appears to do the opposite).

I also think media_pipeline_start should be called first since it
calls the link_validate callbacks. In the patchset I have patch 3/5 calling media_pipeline_start
first.

> And I was also wondering if the stream_count (incremented by media_pipeline_start())
> is something that v4l2_pipeline_stream_enable/disable could use: i.e. s_stream only
> needs to be called if stream_count == 1.

Yes, I think this is also possible.

Thanks,
Dafna

> 
> In fact, perhaps v4l2_pipeline_stream_enable/disable should call __media_pipeline_start/stop?
> 
> Regards,
> 
> 	Hans
> 
>> +
>> +#endif /* CONFIG_MEDIA_CONTROLLER */
>> +
>>   #endif /* V4L2_COMMON_H_ */
>>
>