[PATCH 00/20] Add multiplexed media pads to support CSI-2 virtual channels

[PATCH 00/20] Add multiplexed media pads to support CSI-2 virtual channels

[Niklas Söderlund]

Hi,

This series is a RFC for how I think one could add CSI-2 virtual channel 
support to the V4L2 framework. The problem is that there is no way to in 
the media framework describe and control links between subdevices which 
carry more then one video stream, for example a CSI-2 bus which can have 
4 virtual channels carrying different video streams.

This series adds a new pad flag which would indicate that a pad carries 
multiplexed streams, adds a new s_stream() operation to the pad 
operations structure which takes a new argument 'stream'. This new 
s_stream() operation then is both pad and stream aware. It also extends 
struct v4l2_mbus_frame_desc_entry with a new sub-struct to describe how 
a CSI-2 link multiplexes virtual channels. I also include one 
implementation based on Renesas R-Car which makes use of these patches 
as I think they help with understanding but they have no impact on the 
RFC feature itself.

The idea is that on both sides of the multiplexed media link there are 
one multiplexer subdevice and one demultiplexer subdevice. These two 
subdevices can't do any format conversions, there sole purpose is to 
(de)multiplex the CSI-2 link. If there is hardware which can do both 
CSI-2 multiplexing and format conversions they can be modeled as two 
subdevices from the same device driver and using the still pending 
incremental async mechanism to connect the external pads. The reason 
there is no format conversion is important as the multiplexed pads can't 
have a format in the current V4L2 model, get/set_fmt are not aware of 
streams.

        +------------------+              +------------------+
     +-------+  subdev 1   |              |  subdev 2   +-------+
  +--+ Pad 1 |             |              |             | Pad 3 +---+
     +--+----+   +---------+---+      +---+---------+   +----+--+
        |        | Muxed pad A +------+ Muxed pad B |        |
     +--+----+   +---------+---+      +---+---------+   +----+--+
  +--+ Pad 2 |             |              |             | Pad 4 +---+
     +-------+             |              |             +-------+
        +------------------+              +------------------+

In the simple example above Pad 1 is routed to Pad 3 and Pad 2 to Pad 4, 
and the video data for both of them travels the link between pad A and 
B. One shortcoming of this RFC is that there currently are no way to 
express to user-space which pad is routed to which stream of the 
multiplexed link. But inside the kernel this is known and format 
validation is done by comparing the format of Pad 1 to Pad 3 and Pad 2 
to Pad 4 by the V4L2 framework. But it would be nice for the user to 
also be able to get this information while setting up the MC graph in 
user-space.

Obviously there are things that are not perfect in this RFC, one is the 
above mentioned lack of user-space visibility of that Pad 1 is in fact 
routed to Pad 3. Others are lack of Documentation/ work and I'm sure 
there are error path shortcuts which are not fully thought out. One big 
question is also if the s_stream() operation added to ops structure 
should be a compliment to the existing ones in video and audio ops or 
aim to replace the one in video ops. I'm also unsure of the CSI2 flag of 
struct v4l2_mbus_frame_desc_entry don't really belong in struct 
v4l2_mbus_frame_desc. And I'm sure there are lots of other stuff that's 
why this is a RFC...

A big thanks to Laurent and Sakari for being really nice and taking time 
helping me grasp all the possibilities and issues with this problem, all 
cred to them and all blame to me for misunderstanding there guidance :-)

This series based on the latest R-Car CSI-2 and VIN patches which can be 
found at [1], but that is a dependency only for the driver specific
implementation which acts as an example of implementation. For the V4L2 
framework patches the media-tree is the base.

1. https://git.ragnatech.se/linux#rcar-vin-elinux-v12

Niklas Söderlund (20):
  media.h: add MEDIA_PAD_FL_MUXED flag
  v4l2-subdev.h: add pad and stream aware s_stream
  v4l2-subdev.h: add CSI-2 bus description to struct
    v4l2_mbus_frame_desc_entry
  v4l2-core: check that both pads in a link are muxed if one are
  v4l2-core: verify all streams formats on multiplexed links
  rcar-vin: use the pad and stream aware s_stream
  rcar-csi2: declare sink pad as multiplexed
  rcar-csi2: switch to pad and stream aware s_stream
  rcar-csi2: figure out remote pad and stream which are starting
  rcar-csi2: count usage for each source pad
  rcar-csi2: when starting CSI-2 receiver use frame descriptor
    information
  rcar-csi2: only allow formats on source pads
  rcar-csi2: implement get_frame_desc
  adv748x: add module param for virtual channel
  adv748x: declare source pad as multiplexed
  adv748x: add translation from pixelcode to CSI-2 datatype
  adv748x: implement get_frame_desc
  adv748x: switch to pad and stream aware s_stream
  adv748x: only allow formats on sink pads
  arm64: dts: renesas: salvator: use VC1 for CVBS

 arch/arm64/boot/dts/renesas/salvator-common.dtsi |   2 +-
 drivers/media/i2c/adv748x/adv748x-core.c         |  10 +
 drivers/media/i2c/adv748x/adv748x-csi2.c         |  78 +++++++-
 drivers/media/i2c/adv748x/adv748x.h              |   1 +
 drivers/media/platform/rcar-vin/rcar-csi2.c      | 239 ++++++++++++++++-------
 drivers/media/platform/rcar-vin/rcar-dma.c       |   6 +-
 drivers/media/v4l2-core/v4l2-subdev.c            |  65 ++++++
 include/media/v4l2-subdev.h                      |  16 ++
 include/uapi/linux/media.h                       |   1 +
 9 files changed, 341 insertions(+), 77 deletions(-)

-- 
2.13.3

[PATCH 01/20] media.h: add MEDIA_PAD_FL_MUXED flag

[Niklas Söderlund]

Add flag to indicate that a pad can mux more then one stream. The user
can use the pad operation get_frame_desc to query the pad about how the
pad is muxed.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 include/uapi/linux/media.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index 4865f1e713398b63..49d692e1182b59a1 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -263,6 +263,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK		(1 << 0)
 #define MEDIA_PAD_FL_SOURCE		(1 << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT	(1 << 2)
+#define MEDIA_PAD_FL_MUXED		(1 << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
-- 
2.13.3

[PATCH 02/20] v4l2-subdev.h: add pad and stream aware s_stream

[Niklas Söderlund]

To be able to start and stop individual streams of a multiplexed pad the
s_stream operation needs to be both pad and stream aware. Add a new
operation to pad ops to facilitate this.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 include/media/v4l2-subdev.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 7e63124450a963da..c258b1524f94f8ff 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -643,6 +643,9 @@ struct v4l2_subdev_pad_config {
  *
  * @set_frame_desc: set the low level media bus frame parameters, @fd array
  *                  may be adjusted by the subdev driver to device capabilities.
+ *
+ * @s_stream: used to notify the driver that a stream will start or has
+ *	stopped.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -683,6 +686,8 @@ struct v4l2_subdev_pad_ops {
 			      struct v4l2_mbus_frame_desc *fd);
 	int (*set_frame_desc)(struct v4l2_subdev *sd, unsigned int pad,
 			      struct v4l2_mbus_frame_desc *fd);
+	int (*s_stream)(struct v4l2_subdev *sd, unsigned int pad,
+			unsigned int stream, int enable);
 };
 
 /**
-- 
2.13.3

[PATCH 03/20] v4l2-subdev.h: add CSI-2 bus description to struct v4l2_mbus_frame_desc_entry

[Niklas Söderlund]

Pads that carry muxed CSI-2 streams needs a way to describe which
'normal' pad is muxed to/from which stream of the multiplexed pad.
Extend struct v4l2_mbus_frame_desc_entry to carry this information.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 include/media/v4l2-subdev.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index c258b1524f94f8ff..dc855135d94eafdb 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -292,6 +292,9 @@ struct v4l2_subdev_audio_ops {
  * receiver should use 1D DMA.
  */
 #define V4L2_MBUS_FRAME_DESC_FL_BLOB		(1U << 1)
+/* Indicates the the discriptor describes a muxed CSI-2 bus */
+#define V4L2_MBUS_FRAME_DESC_FL_CSI2		(1U << 2)
+
 
 /**
  * struct v4l2_mbus_frame_desc_entry - media bus frame description structure
@@ -301,11 +304,19 @@ struct v4l2_subdev_audio_ops {
  * @pixelcode: media bus pixel code, valid if FRAME_DESC_FL_BLOB is not set
  * @length: number of octets per frame, valid if V4L2_MBUS_FRAME_DESC_FL_BLOB
  *	    is set
+ * @csi2: CSI-2 bus specific parameters, valid if V4L2_MBUS_FRAME_DESC_FL_CSI2
+ *        is set
  */
 struct v4l2_mbus_frame_desc_entry {
 	u16 flags;
 	u32 pixelcode;
 	u32 length;
+
+	struct {
+		unsigned int channel;
+		unsigned int datatype;
+		unsigned int pad;
+	} csi2;
 };
 
 #define V4L2_FRAME_DESC_ENTRY_MAX	4
-- 
2.13.3

[PATCH 04/20] v4l2-core: check that both pads in a link are muxed if one are

[Niklas Söderlund]

Since multiplexed pads carry multiple streams it's not possible to
verify the format for a specific stream at this time. Instead make sure
both pads are marked as multiplexed and skip the format checking.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 43fefa73e0a3f64f..d6c1a3b777dd2fcd 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -547,6 +547,15 @@ int v4l2_subdev_link_validate(struct media_link *link)
 	struct v4l2_subdev_format sink_fmt, source_fmt;
 	int rval;
 
+	/* Require both pads in a link to be multiplexed if one is */
+	if ((link->source->flags | link->sink->flags) & MEDIA_PAD_FL_MUXED) {
+		if ((link->source->flags & MEDIA_PAD_FL_MUXED) == 0)
+			return -EINVAL;
+		if ((link->sink->flags & MEDIA_PAD_FL_MUXED) == 0)
+			return -EINVAL;
+		return 0;
+	}
+
 	rval = v4l2_subdev_link_validate_get_format(
 		link->source, &source_fmt);
 	if (rval < 0)
-- 
2.13.3

[PATCH 05/20] v4l2-core: verify all streams formats on multiplexed links

[Niklas Söderlund]

Extend the format validation for multiplexed pads to verify all streams
which are part of the multiplexed link. This might take the verification
to an extreme as it could be argued that one should be able to configure
and start just one stream without having to configure other streams
which the user never intends to start.

It could also be argued that the multiplexer and demultiplexer needs to
know about all formats which could be activated before any stream could
be started. In such case the number of streams described in the frame
descriptor should be dynamic and only possible and configured streams
should be reported which would then solve this issue.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 72 +++++++++++++++++++++++++++++++----
 1 file changed, 64 insertions(+), 8 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index d6c1a3b777dd2fcd..43cd2b5e3d8ea323 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -541,20 +541,76 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 	return -EINVAL;
 }
 
+static int v4l2_subdev_link_validate_muxed(struct media_link *link)
+{
+	struct v4l2_mbus_frame_desc sink_fd, source_fd;
+	struct v4l2_subdev *sink_sd, *source_sd;
+	unsigned int i;
+	int ret;
+
+	/* Require both pads in a link to be multiplexed */
+	if ((link->source->flags & MEDIA_PAD_FL_MUXED) == 0 ||
+	    (link->sink->flags & MEDIA_PAD_FL_MUXED) == 0)
+		return -EINVAL;
+
+	sink_sd = media_entity_to_v4l2_subdev(link->sink->entity);
+	source_sd = media_entity_to_v4l2_subdev(link->source->entity);
+
+	/* If not both provide frame descs there is not much to be done */
+	ret = v4l2_subdev_call(sink_sd, pad, get_frame_desc,
+			       link->sink->index, &sink_fd);
+	if (ret < 0)
+		return 0;
+	ret = v4l2_subdev_call(source_sd, pad, get_frame_desc,
+			       link->source->index, &source_fd);
+	if (ret < 0)
+		return 0;
+
+	/* Check both side multiplex same number of streams */
+	if (sink_fd.num_entries != source_fd.num_entries)
+		return -EINVAL;
+
+	/* Verify all formats of the multiplexed pads by examining the
+	 * format of the pads which are routed to them. Maybe this is
+	 * a bad idea...
+	 */
+	for (i = 0; i < sink_fd.num_entries; i++) {
+		struct v4l2_subdev_format sink_fmt, source_fmt;
+
+		sink_fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+		sink_fmt.pad = sink_fd.entry[i].csi2.pad;
+		ret = v4l2_subdev_call(sink_sd, pad, get_fmt, NULL, &sink_fmt);
+		if (ret < 0)
+			return 0;
+
+		source_fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+		source_fmt.pad = source_fd.entry[i].csi2.pad;
+		ret = v4l2_subdev_call(source_sd, pad, get_fmt, NULL,
+				       &source_fmt);
+		if (ret < 0)
+			return 0;
+
+		ret = v4l2_subdev_call(sink_sd, pad, link_validate, link,
+					&source_fmt, &sink_fmt);
+		if (ret == -ENOIOCTLCMD)
+			ret = v4l2_subdev_link_validate_default(sink_sd, link,
+								&source_fmt,
+								&sink_fmt);
+		if (ret)
+			return -EPIPE;
+	}
+
+	return 0;
+}
+
 int v4l2_subdev_link_validate(struct media_link *link)
 {
 	struct v4l2_subdev *sink;
 	struct v4l2_subdev_format sink_fmt, source_fmt;
 	int rval;
 
-	/* Require both pads in a link to be multiplexed if one is */
-	if ((link->source->flags | link->sink->flags) & MEDIA_PAD_FL_MUXED) {
-		if ((link->source->flags & MEDIA_PAD_FL_MUXED) == 0)
-			return -EINVAL;
-		if ((link->sink->flags & MEDIA_PAD_FL_MUXED) == 0)
-			return -EINVAL;
-		return 0;
-	}
+	if ((link->source->flags | link->sink->flags) & MEDIA_PAD_FL_MUXED)
+		return v4l2_subdev_link_validate_muxed(link);
 
 	rval = v4l2_subdev_link_validate_get_format(
 		link->source, &source_fmt);
-- 
2.13.3

[PATCH 06/20] rcar-vin: use the pad and stream aware s_stream

[Niklas Söderlund]

To work with multiplexed streams the pad and stream aware s_stream
operation needs to be used.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/platform/rcar-vin/rcar-dma.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
index 67a82ac6dc88c0da..395cbf96b40aeb8d 100644
--- a/drivers/media/platform/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/rcar-vin/rcar-dma.c
@@ -1235,16 +1235,18 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 		if (media_pipeline_start(&vin->vdev.entity, pipe))
 			return -EPIPE;
 
-		ret = v4l2_subdev_call(sd, video, s_stream, 1);
+		ret = v4l2_subdev_call(sd, pad, s_stream, pad->index, 0, 1);
 		if (ret == -ENOIOCTLCMD)
 			ret = 0;
 		if (ret)
 			media_pipeline_stop(&vin->vdev.entity);
 	} else {
 		media_pipeline_stop(&vin->vdev.entity);
-		ret = v4l2_subdev_call(sd, video, s_stream, 0);
+		ret = v4l2_subdev_call(sd, pad, s_stream, pad->index, 0, 0);
 	}
 
+	vin_dbg(vin, "pad: %u stream: 0 enable: %d\n", pad->index, on);
+
 	return ret;
 }
 
-- 
2.13.3

[PATCH 07/20] rcar-csi2: declare sink pad as multiplexed

[Niklas Söderlund]

The sink pad will receive multiplexed streams over a CSI-2 bus, mark the
pad as muxed.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/platform/rcar-vin/rcar-csi2.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c
index 8e424b7adf9cc39c..58c8e9ec8ab756fc 100644
--- a/drivers/media/platform/rcar-vin/rcar-csi2.c
+++ b/drivers/media/platform/rcar-vin/rcar-csi2.c
@@ -908,7 +908,8 @@ static int rcar_csi2_probe(struct platform_device *pdev)
 	priv->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
 	priv->subdev.entity.ops = &rcar_csi2_entity_ops;
 
-	priv->pads[RCAR_CSI2_SINK].flags = MEDIA_PAD_FL_SINK;
+	priv->pads[RCAR_CSI2_SINK].flags =
+		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_MUXED;
 	for (i = RCAR_CSI2_SOURCE_VC0; i < NR_OF_RCAR_CSI2_PAD; i++)
 		priv->pads[i].flags = MEDIA_PAD_FL_SOURCE;
 
-- 
2.13.3

[PATCH 08/20] rcar-csi2: switch to pad and stream aware s_stream

[Niklas Söderlund]

Switch the driver to implement the pad and stream aware s_stream
operation. This is needed to enable to support to start and stop
individual streams on a multiplexed pad.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/platform/rcar-vin/rcar-csi2.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c
index 58c8e9ec8ab756fc..46da45aea18fe3d3 100644
--- a/drivers/media/platform/rcar-vin/rcar-csi2.c
+++ b/drivers/media/platform/rcar-vin/rcar-csi2.c
@@ -552,12 +552,19 @@ static int rcar_csi2_sd_info(struct rcar_csi2 *priv, struct v4l2_subdev **sd)
 	return 0;
 }
 
-static int rcar_csi2_s_stream(struct v4l2_subdev *sd, int enable)
+static int rcar_csi2_s_stream(struct v4l2_subdev *sd, unsigned int pad,
+			      unsigned int stream, int enable)
 {
 	struct rcar_csi2 *priv = sd_to_csi2(sd);
 	struct v4l2_subdev *nextsd;
 	int ret;
 
+	if (pad < RCAR_CSI2_SOURCE_VC0 || pad > RCAR_CSI2_SOURCE_VC3)
+		return -EINVAL;
+
+	if (stream != 0)
+		return -EINVAL;
+
 	mutex_lock(&priv->lock);
 
 	ret = rcar_csi2_sd_info(priv, &nextsd);
@@ -625,10 +632,6 @@ static int rcar_csi2_get_pad_format(struct v4l2_subdev *sd,
 	return 0;
 }
 
-static const struct v4l2_subdev_video_ops rcar_csi2_video_ops = {
-	.s_stream = rcar_csi2_s_stream,
-};
-
 static const struct v4l2_subdev_core_ops rcar_csi2_subdev_core_ops = {
 	.s_power = rcar_csi2_s_power,
 };
@@ -636,10 +639,10 @@ static const struct v4l2_subdev_core_ops rcar_csi2_subdev_core_ops = {
 static const struct v4l2_subdev_pad_ops rcar_csi2_pad_ops = {
 	.set_fmt = rcar_csi2_set_pad_format,
 	.get_fmt = rcar_csi2_get_pad_format,
+	.s_stream = rcar_csi2_s_stream,
 };
 
 static const struct v4l2_subdev_ops rcar_csi2_subdev_ops = {
-	.video	= &rcar_csi2_video_ops,
 	.core	= &rcar_csi2_subdev_core_ops,
 	.pad	= &rcar_csi2_pad_ops,
 };
-- 
2.13.3

[PATCH 09/20] rcar-csi2: figure out remote pad and stream which are starting

[Niklas Söderlund]

Using the information in v4l2_mbus_frame_desc figure out which remote
pad and stream should be started or stopped.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/platform/rcar-vin/rcar-csi2.c | 40 +++++++++++++++++++++++++----
 1 file changed, 35 insertions(+), 5 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c
index 46da45aea18fe3d3..8b4db70b0dd5d0fb 100644
--- a/drivers/media/platform/rcar-vin/rcar-csi2.c
+++ b/drivers/media/platform/rcar-vin/rcar-csi2.c
@@ -533,9 +533,14 @@ static void rcar_csi2_stop(struct rcar_csi2 *priv)
 	rcar_csi2_reset(priv);
 }
 
-static int rcar_csi2_sd_info(struct rcar_csi2 *priv, struct v4l2_subdev **sd)
+static int rcar_csi2_sd_info(struct rcar_csi2 *priv, unsigned int channel,
+			     struct v4l2_subdev **nextsd, unsigned int *nextpad,
+			     unsigned int *nextstream)
 {
+	struct v4l2_mbus_frame_desc_entry *entry;
+	struct v4l2_mbus_frame_desc fd;
 	struct media_pad *pad;
+	unsigned int i;
 
 	pad = media_entity_remote_pad(&priv->pads[RCAR_CSI2_SINK]);
 	if (!pad) {
@@ -543,31 +548,56 @@ static int rcar_csi2_sd_info(struct rcar_csi2 *priv, struct v4l2_subdev **sd)
 		return -ENODEV;
 	}
 
-	*sd = media_entity_to_v4l2_subdev(pad->entity);
-	if (!*sd) {
+	*nextsd = media_entity_to_v4l2_subdev(pad->entity);
+	if (!*nextsd) {
 		dev_err(priv->dev, "Could not find remote subdevice\n");
 		return -ENODEV;
 	}
 
-	return 0;
+	*nextpad = pad->index;
+
+	if (v4l2_subdev_call(*nextsd, pad, get_frame_desc, *nextpad, &fd)) {
+		dev_err(priv->dev, "Could not read frame desc\n");
+		return -EINVAL;
+	}
+
+	for (i = 0; i < fd.num_entries; i++) {
+		entry = &fd.entry[i];
+
+		if ((entry->flags & V4L2_MBUS_FRAME_DESC_FL_CSI2) == 0)
+			return -EINVAL;
+
+		if (entry->csi2.channel == channel) {
+			*nextstream = i;
+			return 0;
+		}
+	}
+
+	dev_err(priv->dev, "Could not find stream for channel %u\n",
+		channel);
+	return -EINVAL;
 }
 
 static int rcar_csi2_s_stream(struct v4l2_subdev *sd, unsigned int pad,
 			      unsigned int stream, int enable)
 {
 	struct rcar_csi2 *priv = sd_to_csi2(sd);
+	unsigned int channel, nextpad, nextstream;
 	struct v4l2_subdev *nextsd;
 	int ret;
 
 	if (pad < RCAR_CSI2_SOURCE_VC0 || pad > RCAR_CSI2_SOURCE_VC3)
 		return -EINVAL;
 
+	/* Figure out CSI-2 channel from pad. First source is VC0, etc */
+	channel = pad - 1;
+
 	if (stream != 0)
 		return -EINVAL;
 
 	mutex_lock(&priv->lock);
 
-	ret = rcar_csi2_sd_info(priv, &nextsd);
+	ret = rcar_csi2_sd_info(priv, channel, &nextsd, &nextpad, &nextstream);
 	if (ret)
 		goto out;
 
-- 
2.13.3

[PATCH 10/20] rcar-csi2: count usage for each source pad

[Niklas Söderlund]

The R-Car CSI-2 hardware can output the same virtual channel
simultaneously to more then one R-Car VIN. For this reason we need to
move the usage counting from the global device to each source pad.

If a source pads usage count go from 0 to 1 we need to signal that a new
stream should start, likewise if it go from 1 to 0 we need to stop a
stream. At the same time we don't want to start or stop the whole R-Car
CSI-2 device but at the first or last stream is started.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/platform/rcar-vin/rcar-csi2.c | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c
index 8b4db70b0dd5d0fb..12a8ea7a219ac7d3 100644
--- a/drivers/media/platform/rcar-vin/rcar-csi2.c
+++ b/drivers/media/platform/rcar-vin/rcar-csi2.c
@@ -294,7 +294,7 @@ struct rcar_csi2 {
 	struct v4l2_mbus_framefmt mf;
 
 	struct mutex lock;
-	int stream_count;
+	int stream_count[4];
 
 	struct {
 		struct v4l2_async_subdev asd;
@@ -582,7 +582,7 @@ static int rcar_csi2_s_stream(struct v4l2_subdev *sd, unsigned int pad,
 			      unsigned int stream, int enable)
 {
 	struct rcar_csi2 *priv = sd_to_csi2(sd);
-	unsigned int channel, nextpad, nextstream;
+	unsigned int i, channel, nextpad, nextstream, count = 0;
 	struct v4l2_subdev *nextsd;
 	int ret;
 
@@ -601,18 +601,25 @@ static int rcar_csi2_s_stream(struct v4l2_subdev *sd, unsigned int pad,
 	if (ret)
 		goto out;
 
-	priv->stream_count += enable ? 1 : -1;
+	priv->stream_count[channel] += enable ? 1 : -1;
 
-	if (enable && priv->stream_count == 1) {
+	for (i = 0; i < 4; i++)
+		count += priv->stream_count[i];
+
+	if (enable && count == 1) {
 		ret =  rcar_csi2_start(priv);
 		if (ret)
 			goto out;
-		ret = v4l2_subdev_call(nextsd, video, s_stream, 1);
-
-	} else if (!enable && !priv->stream_count) {
+	} else if (!enable && !count) {
 		rcar_csi2_stop(priv);
-		ret = v4l2_subdev_call(nextsd, video, s_stream, 0);
 	}
+
+	if (enable && priv->stream_count[channel] == 1)
+		ret = v4l2_subdev_call(nextsd, pad, s_stream, nextpad,
+				       nextstream, 1);
+	else if (!enable && !priv->stream_count[channel])
+		ret = v4l2_subdev_call(nextsd, pad, s_stream, nextpad,
+				       nextstream, 0);
 out:
 	mutex_unlock(&priv->lock);
 
@@ -912,7 +919,9 @@ static int rcar_csi2_probe(struct platform_device *pdev)
 	priv->dev = &pdev->dev;
 
 	mutex_init(&priv->lock);
-	priv->stream_count = 0;
+
+	for (i = 0; i < 4; i++)
+		priv->stream_count[i] = 0;
 
 	ret = rcar_csi2_parse_dt_settings(priv);
 	if (ret)
-- 
2.13.3

[PATCH 11/20] rcar-csi2: when starting CSI-2 receiver use frame descriptor information

[Niklas Söderlund]

The driver now have access to frame descriptor information, use it. Only
enable the virtual channels which are described in the frame descriptor
and calculate the PHY PLL based on all streams described.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/platform/rcar-vin/rcar-csi2.c | 85 ++++++++++++++---------------
 1 file changed, 42 insertions(+), 43 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c
index 12a8ea7a219ac7d3..b8ec03d595ba8ac4 100644
--- a/drivers/media/platform/rcar-vin/rcar-csi2.c
+++ b/drivers/media/platform/rcar-vin/rcar-csi2.c
@@ -242,24 +242,22 @@ static const struct phypll_hsfreqrange hsfreqrange_m3w_h3es1[] = {
 #define CSI0CLKFREQRANGE(n)		((n & 0x3f) << 16)
 
 struct rcar_csi2_format {
-	unsigned int code;
 	unsigned int datatype;
 	unsigned int bpp;
 };
 
 static const struct rcar_csi2_format rcar_csi2_formats[] = {
-	{ .code = MEDIA_BUS_FMT_RGB888_1X24,	.datatype = 0x24, .bpp = 24 },
-	{ .code = MEDIA_BUS_FMT_UYVY8_1X16,	.datatype = 0x1e, .bpp = 16 },
-	{ .code = MEDIA_BUS_FMT_UYVY8_2X8,	.datatype = 0x1e, .bpp = 16 },
-	{ .code = MEDIA_BUS_FMT_YUYV10_2X10,	.datatype = 0x1e, .bpp = 16 },
+	{ .datatype = 0x1e, .bpp = 16 },
+	{ .datatype = 0x24, .bpp = 24 },
 };
 
-static const struct rcar_csi2_format *rcar_csi2_code_to_fmt(unsigned int code)
+static const struct rcar_csi2_format
+*rcar_csi2_datatype_to_fmt(unsigned int datatype)
 {
 	int i;
 
 	for (i = 0; i < ARRAY_SIZE(rcar_csi2_formats); i++)
-		if (rcar_csi2_formats[i].code == code)
+		if (rcar_csi2_formats[i].datatype == datatype)
 			return rcar_csi2_formats + i;
 	return NULL;
 }
@@ -348,12 +346,14 @@ static int rcar_csi2_wait_phy_start(struct rcar_csi2 *priv)
 
 static int rcar_csi2_calc_phypll(struct rcar_csi2 *priv,
 				 struct v4l2_subdev *source,
-				 struct v4l2_mbus_framefmt *mf,
+				 struct v4l2_mbus_frame_desc *fd,
 				 u32 *phypll)
 {
+	struct v4l2_mbus_frame_desc_entry *entry;
 	const struct phypll_hsfreqrange *hsfreq;
 	const struct rcar_csi2_format *format;
 	struct v4l2_ctrl *ctrl;
+	unsigned int i, bpp = 0;
 	u64 mbps;
 
 	ctrl = v4l2_ctrl_find(source->ctrl_handler, V4L2_CID_PIXEL_RATE);
@@ -363,13 +363,20 @@ static int rcar_csi2_calc_phypll(struct rcar_csi2 *priv,
 		return -EINVAL;
 	}
 
-	format = rcar_csi2_code_to_fmt(mf->code);
-	if (!format) {
-		dev_err(priv->dev, "Unknown format: %d\n", mf->code);
-		return -EINVAL;
+	for (i = 0; i < fd->num_entries; i++) {
+		entry = &fd->entry[i];
+
+		format = rcar_csi2_datatype_to_fmt(entry->csi2.datatype);
+		if (!format) {
+			dev_err(priv->dev, "Unknown datatype: %d\n",
+				entry->csi2.datatype);
+			return -EINVAL;
+		}
+
+		bpp += format->bpp;
 	}
 
-	mbps = v4l2_ctrl_g_ctrl_int64(ctrl) * format->bpp;
+	mbps = v4l2_ctrl_g_ctrl_int64(ctrl) * bpp;
 	do_div(mbps, priv->lanes * 1000000);
 
 	for (hsfreq = priv->info->hsfreqrange; hsfreq->mbps != 0; hsfreq++)
@@ -391,11 +398,10 @@ static int rcar_csi2_calc_phypll(struct rcar_csi2 *priv,
 
 static int rcar_csi2_start(struct rcar_csi2 *priv)
 {
-	const struct rcar_csi2_format *format;
-	struct v4l2_subdev_format fmt;
+	struct v4l2_mbus_frame_desc_entry *entry;
 	struct media_pad *pad, *source_pad;
 	struct v4l2_subdev *source = NULL;
-	struct v4l2_mbus_framefmt *mf = &fmt.format;
+	struct v4l2_mbus_frame_desc fd;
 	u32 phycnt, phypll, tmp;
 	u32 vcdt = 0, vcdt2 = 0;
 	unsigned int i;
@@ -417,40 +423,33 @@ static int rcar_csi2_start(struct rcar_csi2 *priv)
 	dev_dbg(priv->dev, "Using source %s pad: %u\n", source->name,
 		source_pad->index);
 
-	fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
-	fmt.pad = source_pad->index;
-	ret = v4l2_subdev_call(source, pad, get_fmt, NULL, &fmt);
-	if (ret)
-		return ret;
+	if (v4l2_subdev_call(source, pad, get_frame_desc, source_pad->index,
+			     &fd)) {
+		dev_err(priv->dev, "Could not read frame desc\n");
+		return -EINVAL;
+	}
 
-	dev_dbg(priv->dev, "Input size (%dx%d%c)\n", mf->width,
-		mf->height, mf->field == V4L2_FIELD_NONE ? 'p' : 'i');
-
-	/*
-	 * Enable all Virtual Channels
-	 *
-	 * NOTE: It's not possible to get individual format for each
-	 *       source virtual channel. Once this is possible in V4L2
-	 *       it should be used here.
-	 */
-	for (i = 0; i < 4; i++) {
-		format = rcar_csi2_code_to_fmt(mf->code);
-		if (!format) {
-			dev_err(priv->dev, "Unsupported media bus format: %d\n",
-				mf->code);
+	for (i = 0; i < fd.num_entries; i++) {
+		entry = &fd.entry[i];
+
+		if ((entry->flags & V4L2_MBUS_FRAME_DESC_FL_CSI2) == 0)
 			return -EINVAL;
-		}
 
-		tmp = VCDT_SEL_VC(i) | VCDT_VCDTN_EN | VCDT_SEL_DTN_ON |
-			VCDT_SEL_DT(format->datatype);
+		tmp = VCDT_SEL_VC(entry->csi2.channel) | VCDT_VCDTN_EN |
+			VCDT_SEL_DTN_ON | VCDT_SEL_DT(entry->csi2.datatype);
 
 		/* Store in correct reg and offset */
-		if (i < 2)
-			vcdt |= tmp << ((i % 2) * 16);
+		if (entry->csi2.channel < 2)
+			vcdt |= tmp << ((entry->csi2.channel % 2) * 16);
 		else
-			vcdt2 |= tmp << ((i % 2) * 16);
+			vcdt2 |= tmp << ((entry->csi2.channel % 2) * 16);
+
+		dev_dbg(priv->dev, "VC%d datatype: 0x%x\n",
+			entry->csi2.channel, entry->csi2.datatype);
 	}
 
+	dev_dbg(priv->dev, "VCDT: 0x%08x VCDT2: 0x%08x\n", vcdt, vcdt2);
+
 	switch (priv->lanes) {
 	case 1:
 		phycnt = PHYCNT_ENABLECLK | PHYCNT_ENABLE_0;
@@ -466,7 +465,7 @@ static int rcar_csi2_start(struct rcar_csi2 *priv)
 		return -EINVAL;
 	}
 
-	ret = rcar_csi2_calc_phypll(priv, source, mf, &phypll);
+	ret = rcar_csi2_calc_phypll(priv, source, &fd, &phypll);
 	if (ret)
 		return ret;
 
-- 
2.13.3

[PATCH 12/20] rcar-csi2: only allow formats on source pads

[Niklas Söderlund]

The driver is now pad and stream aware, only allow to get/set format on
source pads. Also record a different format for each source pad since
it's no longer true that they are all the same.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/platform/rcar-vin/rcar-csi2.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c
index b8ec03d595ba8ac4..034d37bc66e93ac7 100644
--- a/drivers/media/platform/rcar-vin/rcar-csi2.c
+++ b/drivers/media/platform/rcar-vin/rcar-csi2.c
@@ -289,7 +289,7 @@ struct rcar_csi2 {
 	struct v4l2_subdev subdev;
 	struct media_pad pads[NR_OF_RCAR_CSI2_PAD];
 
-	struct v4l2_mbus_framefmt mf;
+	struct v4l2_mbus_framefmt mf[4];
 
 	struct mutex lock;
 	int stream_count[4];
@@ -644,10 +644,14 @@ static int rcar_csi2_set_pad_format(struct v4l2_subdev *sd,
 	struct rcar_csi2 *priv = sd_to_csi2(sd);
 	struct v4l2_mbus_framefmt *framefmt;
 
+	if (format->pad < RCAR_CSI2_SOURCE_VC0 ||
+	    format->pad > RCAR_CSI2_SOURCE_VC3)
+		return -EINVAL;
+
 	if (format->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
-		priv->mf = format->format;
+		priv->mf[format->pad] = format->format;
 	} else {
-		framefmt = v4l2_subdev_get_try_format(sd, cfg, 0);
+		framefmt = v4l2_subdev_get_try_format(sd, cfg, format->pad);
 		*framefmt = format->format;
 	}
 
@@ -660,10 +664,15 @@ static int rcar_csi2_get_pad_format(struct v4l2_subdev *sd,
 {
 	struct rcar_csi2 *priv = sd_to_csi2(sd);
 
+	if (format->pad < RCAR_CSI2_SOURCE_VC0 ||
+	    format->pad > RCAR_CSI2_SOURCE_VC3)
+		return -EINVAL;
+
 	if (format->which == V4L2_SUBDEV_FORMAT_ACTIVE)
-		format->format = priv->mf;
+		format->format = priv->mf[format->pad];
 	else
-		format->format = *v4l2_subdev_get_try_format(sd, cfg, 0);
+		format->format = *v4l2_subdev_get_try_format(sd, cfg,
+							     format->pad);
 
 	return 0;
 }
-- 
2.13.3

[PATCH 13/20] rcar-csi2: implement get_frame_desc

[Niklas Söderlund]

The frame descriptor of the CSI-2 receiver mirrors that of the
transmitter in all aspects but the routing from the multiplexed sink pad
to the 'normal' source pads. Simply fetch the frame descriptor of the
transmitter and update the routing.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/platform/rcar-vin/rcar-csi2.c | 56 +++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c
index 034d37bc66e93ac7..e01f334364d50754 100644
--- a/drivers/media/platform/rcar-vin/rcar-csi2.c
+++ b/drivers/media/platform/rcar-vin/rcar-csi2.c
@@ -677,6 +677,61 @@ static int rcar_csi2_get_pad_format(struct v4l2_subdev *sd,
 	return 0;
 }
 
+static int rcar_csi2_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
+				    struct v4l2_mbus_frame_desc *fd)
+{
+	struct rcar_csi2 *priv = sd_to_csi2(sd);
+	struct v4l2_subdev *nextsd;
+	struct media_pad *nextpad;
+	unsigned int i;
+
+	/* Only valid for sink pad */
+	if (pad != RCAR_CSI2_SINK || fd == NULL)
+		return -EINVAL;
+
+	/* Figoure out remote subdev and pad */
+	nextpad = media_entity_remote_pad(&priv->pads[RCAR_CSI2_SINK]);
+	if (!nextpad) {
+		dev_err(priv->dev, "Could not find remote pad\n");
+		return -EPIPE;
+	}
+
+	nextsd = media_entity_to_v4l2_subdev(nextpad->entity);
+	if (!nextsd) {
+		dev_err(priv->dev, "Could not find remote subdevice\n");
+		return -ENODEV;
+	}
+
+	/*
+	 * Our frame descriptor will mirror that of the remote device
+	 * in all aspects but which of our source pads are routed to
+	 * which muxed channel. So start with a copy of the remote
+	 * description and then update the pads
+	 */
+
+	if (v4l2_subdev_call(nextsd, pad, get_frame_desc, nextpad->index, fd)) {
+		dev_err(priv->dev, "Could not read frame desc\n");
+		return -EINVAL;
+	}
+
+	for (i = 0; i < fd->num_entries; i++) {
+		/* Make sure it's a muxed CSI2 bus */
+		if ((fd->entry[i].flags & V4L2_MBUS_FRAME_DESC_FL_CSI2) == 0)
+			return -EINVAL;
+
+		if (fd->entry[i].csi2.channel >= 4) {
+			dev_err(priv->dev, "CSI-2 channel value to large: %u\n",
+				fd->entry[i].csi2.channel);
+			return -EINVAL;
+		}
+
+		/* Figure out pad from CSI-2 channel. First source pad is 1 */
+		fd->entry[i].csi2.pad = fd->entry[i].csi2.channel + 1;
+	}
+
+	return 0;
+}
+
 static const struct v4l2_subdev_core_ops rcar_csi2_subdev_core_ops = {
 	.s_power = rcar_csi2_s_power,
 };
@@ -684,6 +739,7 @@ static const struct v4l2_subdev_core_ops rcar_csi2_subdev_core_ops = {
 static const struct v4l2_subdev_pad_ops rcar_csi2_pad_ops = {
 	.set_fmt = rcar_csi2_set_pad_format,
 	.get_fmt = rcar_csi2_get_pad_format,
+	.get_frame_desc = rcar_csi2_get_frame_desc,
 	.s_stream = rcar_csi2_s_stream,
 };
 
-- 
2.13.3

[PATCH 14/20] adv748x: add module param for virtual channel

[Niklas Söderlund]

The hardware can output on any of the 4 (0-3) Virtual Channels of the
CSI-2 bus. Add a module parameter each for TXA and TXB to allow the user
to specify which channel should be used.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/i2c/adv748x/adv748x-core.c | 10 ++++++++++
 drivers/media/i2c/adv748x/adv748x-csi2.c |  2 +-
 drivers/media/i2c/adv748x/adv748x.h      |  1 +
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/media/i2c/adv748x/adv748x-core.c b/drivers/media/i2c/adv748x/adv748x-core.c
index aeb6ae80cb184eb4..72ec585f44385b01 100644
--- a/drivers/media/i2c/adv748x/adv748x-core.c
+++ b/drivers/media/i2c/adv748x/adv748x-core.c
@@ -31,6 +31,9 @@
 
 #include "adv748x.h"
 
+static unsigned int txavc;
+static unsigned int txbvc;
+
 /* -----------------------------------------------------------------------------
  * Register manipulation
  */
@@ -751,6 +754,7 @@ static int adv748x_probe(struct i2c_client *client,
 	}
 
 	/* Initialise TXA */
+	state->txa.vc = txavc;
 	ret = adv748x_csi2_init(state, &state->txa);
 	if (ret) {
 		adv_err(state, "Failed to probe TXA");
@@ -758,6 +762,7 @@ static int adv748x_probe(struct i2c_client *client,
 	}
 
 	/* Initialise TXB */
+	state->txb.vc = txbvc;
 	ret = adv748x_csi2_init(state, &state->txb);
 	if (ret) {
 		adv_err(state, "Failed to probe TXB");
@@ -827,6 +832,11 @@ static struct i2c_driver adv748x_driver = {
 
 module_i2c_driver(adv748x_driver);
 
+module_param(txavc, uint, 0644);
+MODULE_PARM_DESC(txavc, "Virtual Channel for TXA");
+module_param(txbvc, uint, 0644);
+MODULE_PARM_DESC(txbvc, "Virtual Channel for TXB");
+
 MODULE_AUTHOR("Kieran Bingham <kieran.bingham@ideasonboard.com>");
 MODULE_DESCRIPTION("ADV748X video decoder");
 MODULE_LICENSE("GPL v2");
diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index 979825d4a419b2da..2bec0cd0a00f1d5c 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -280,7 +280,7 @@ int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx)
 	}
 
 	/* Initialise the virtual channel */
-	adv748x_csi2_set_virtual_channel(tx, 0);
+	adv748x_csi2_set_virtual_channel(tx, tx->vc);
 
 	adv748x_subdev_init(&tx->sd, state, &adv748x_csi2_ops,
 			    MEDIA_ENT_F_UNKNOWN,
diff --git a/drivers/media/i2c/adv748x/adv748x.h b/drivers/media/i2c/adv748x/adv748x.h
index cc4151b5b31e2418..b3ef22e48e04ea37 100644
--- a/drivers/media/i2c/adv748x/adv748x.h
+++ b/drivers/media/i2c/adv748x/adv748x.h
@@ -92,6 +92,7 @@ enum adv748x_csi2_pads {
 
 struct adv748x_csi2 {
 	struct adv748x_state *state;
+	unsigned int vc;
 	struct v4l2_mbus_framefmt format;
 	unsigned int page;
 
-- 
2.13.3

[PATCH 15/20] adv748x: declare source pad as multiplexed

[Niklas Söderlund]

The source pad will receive multiplexed streams over a CSI-2 bus, mark
the pad as muxed.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/i2c/adv748x/adv748x-csi2.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index 2bec0cd0a00f1d5c..b76c2be8da4271fb 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -293,7 +293,8 @@ int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx)
 	tx->sd.internal_ops = &adv748x_csi2_internal_ops;
 
 	tx->pads[ADV748X_CSI2_SINK].flags = MEDIA_PAD_FL_SINK;
-	tx->pads[ADV748X_CSI2_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
+	tx->pads[ADV748X_CSI2_SOURCE].flags =
+		MEDIA_PAD_FL_SOURCE | MEDIA_PAD_FL_MUXED;
 
 	ret = media_entity_pads_init(&tx->sd.entity, ADV748X_CSI2_NR_PADS,
 				     tx->pads);
-- 
2.13.3

[PATCH 16/20] adv748x: add translation from pixelcode to CSI-2 datatype

[Niklas Söderlund]

This will be needed to fill out the frame descriptor information
correctly.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/i2c/adv748x/adv748x-csi2.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index b76c2be8da4271fb..a77069fc1adc1eca 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -18,6 +18,28 @@
 
 #include "adv748x.h"
 
+struct adv748x_csi2_format {
+	unsigned int code;
+	unsigned int datatype;
+};
+
+static const struct adv748x_csi2_format adv748x_csi2_formats[] = {
+	{ .code = MEDIA_BUS_FMT_RGB888_1X24,    .datatype = 0x24, },
+	{ .code = MEDIA_BUS_FMT_UYVY8_1X16,     .datatype = 0x1e, },
+	{ .code = MEDIA_BUS_FMT_UYVY8_2X8,      .datatype = 0x1e, },
+	{ .code = MEDIA_BUS_FMT_YUYV10_2X10,    .datatype = 0x1e, },
+};
+
+static unsigned int adv748x_csi2_code_to_datatype(unsigned int code)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(adv748x_csi2_formats); i++)
+		if (adv748x_csi2_formats[i].code == code)
+			return adv748x_csi2_formats[i].datatype;
+	return 0;
+}
+
 static bool is_txa(struct adv748x_csi2 *tx)
 {
 	return tx == &tx->state->txa;
-- 
2.13.3

[PATCH 17/20] adv748x: implement get_frame_desc

[Niklas Söderlund]

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/i2c/adv748x/adv748x-csi2.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index a77069fc1adc1eca..58a9d9105148c15b 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -225,9 +225,38 @@ static int adv748x_csi2_set_format(struct v4l2_subdev *sd,
 	return ret;
 }
 
+static int adv748x_csi2_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
+				       struct v4l2_mbus_frame_desc *fd)
+{
+	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
+	struct adv748x_state *state = tx->state;
+	struct v4l2_mbus_framefmt *mbusformat;
+
+	if (pad != ADV748X_CSI2_SOURCE || fd == NULL)
+		return -EINVAL;
+
+	mbusformat = adv748x_csi2_get_pad_format(sd, NULL, ADV748X_CSI2_SINK,
+						 V4L2_SUBDEV_FORMAT_ACTIVE);
+	if (!mbusformat)
+		return -EINVAL;
+
+	mutex_lock(&state->mutex);
+	fd->entry[0].flags = V4L2_MBUS_FRAME_DESC_FL_CSI2;
+	fd->entry[0].pixelcode = mbusformat->code;
+	fd->entry[0].csi2.channel = tx->vc;
+	fd->entry[0].csi2.datatype =
+		adv748x_csi2_code_to_datatype(fd->entry[0].pixelcode);
+	fd->entry[0].csi2.pad = ADV748X_CSI2_SINK;
+	fd->num_entries = 1;
+	mutex_unlock(&state->mutex);
+
+	return 0;
+}
+
 static const struct v4l2_subdev_pad_ops adv748x_csi2_pad_ops = {
 	.get_fmt = adv748x_csi2_get_format,
 	.set_fmt = adv748x_csi2_set_format,
+	.get_frame_desc = adv748x_csi2_get_frame_desc,
 };
 
 /* -----------------------------------------------------------------------------
-- 
2.13.3

[PATCH 18/20] adv748x: switch to pad and stream aware s_stream

[Niklas Söderlund]

Switch the driver to implement the pad and stream aware s_stream
operation. This is needed to enable to support to start and stop
individual streams on a multiplexed pad.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/i2c/adv748x/adv748x-csi2.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index 58a9d9105148c15b..39d7877ff615fed1 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -128,22 +128,26 @@ static const struct v4l2_subdev_internal_ops adv748x_csi2_internal_ops = {
  * v4l2_subdev_video_ops
  */
 
-static int adv748x_csi2_s_stream(struct v4l2_subdev *sd, int enable)
+static int adv748x_csi2_s_stream(struct v4l2_subdev *sd, unsigned int pad,
+				 unsigned int stream, int enable)
 {
 	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
+	struct adv748x_state *state = tx->state;
 	struct v4l2_subdev *src;
 
+	if (pad != ADV748X_CSI2_SOURCE || stream != 0)
+		return -EINVAL;
+
 	src = adv748x_get_remote_sd(&tx->pads[ADV748X_CSI2_SINK]);
 	if (!src)
 		return -EPIPE;
 
+	adv_dbg(state, "%s: pad: %u stream: %u enable: %d\n", sd->name,
+		pad, stream, enable);
+
 	return v4l2_subdev_call(src, video, s_stream, enable);
 }
 
-static const struct v4l2_subdev_video_ops adv748x_csi2_video_ops = {
-	.s_stream = adv748x_csi2_s_stream,
-};
-
 /* -----------------------------------------------------------------------------
  * v4l2_subdev_pad_ops
  *
@@ -257,6 +261,7 @@ static const struct v4l2_subdev_pad_ops adv748x_csi2_pad_ops = {
 	.get_fmt = adv748x_csi2_get_format,
 	.set_fmt = adv748x_csi2_set_format,
 	.get_frame_desc = adv748x_csi2_get_frame_desc,
+	.s_stream = adv748x_csi2_s_stream,
 };
 
 /* -----------------------------------------------------------------------------
@@ -264,7 +269,6 @@ static const struct v4l2_subdev_pad_ops adv748x_csi2_pad_ops = {
  */
 
 static const struct v4l2_subdev_ops adv748x_csi2_ops = {
-	.video = &adv748x_csi2_video_ops,
 	.pad = &adv748x_csi2_pad_ops,
 };
 
-- 
2.13.3

[PATCH 19/20] adv748x: only allow formats on sink pads

[Niklas Söderlund]

The driver is now pad and stream aware, only allow to get/set format on
sink pads. Also record a different format for each sink pad since it's
no longer true that they are all the same

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/i2c/adv748x/adv748x-csi2.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index 39d7877ff615fed1..9af33edfa7bed0a0 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -176,6 +176,9 @@ static int adv748x_csi2_get_format(struct v4l2_subdev *sd,
 	struct adv748x_state *state = tx->state;
 	struct v4l2_mbus_framefmt *mbusformat;
 
+	if (sdformat->pad != ADV748X_CSI2_SINK)
+		return -EINVAL;
+
 	mbusformat = adv748x_csi2_get_pad_format(sd, cfg, sdformat->pad,
 						 sdformat->which);
 	if (!mbusformat)
@@ -199,6 +202,9 @@ static int adv748x_csi2_set_format(struct v4l2_subdev *sd,
 	struct v4l2_mbus_framefmt *mbusformat;
 	int ret = 0;
 
+	if (sdformat->pad != ADV748X_CSI2_SINK)
+		return -EINVAL;
+
 	mbusformat = adv748x_csi2_get_pad_format(sd, cfg, sdformat->pad,
 						 sdformat->which);
 	if (!mbusformat)
-- 
2.13.3

[PATCH 20/20] arm64: dts: renesas: salvator: use VC1 for CVBS

[Niklas Söderlund]

In order to test Virtual Channels use VC1 for CVBS input from the
adv748x.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 arch/arm64/boot/dts/renesas/salvator-common.dtsi | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/boot/dts/renesas/salvator-common.dtsi b/arch/arm64/boot/dts/renesas/salvator-common.dtsi
index 7b67efcb1d22090a..8047fe1df065d63b 100644
--- a/arch/arm64/boot/dts/renesas/salvator-common.dtsi
+++ b/arch/arm64/boot/dts/renesas/salvator-common.dtsi
@@ -41,7 +41,7 @@
 	};
 
 	chosen {
-		bootargs = "ignore_loglevel rw root=/dev/nfs ip=dhcp";
+		bootargs = "ignore_loglevel rw root=/dev/nfs ip=dhcp adv748x.txbvc=1";
 		stdout-path = "serial0:115200n8";
 	};
 
-- 
2.13.3

Re: [PATCH 20/20] arm64: dts: renesas: salvator: use VC1 for CVBS

[Simon Horman]

On Fri, Aug 11, 2017 at 11:57:03AM +0200, Niklas Söderlund wrote:
> In order to test Virtual Channels use VC1 for CVBS input from the
> adv748x.
> 
> Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> ---
>  arch/arm64/boot/dts/renesas/salvator-common.dtsi | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/boot/dts/renesas/salvator-common.dtsi b/arch/arm64/boot/dts/renesas/salvator-common.dtsi
> index 7b67efcb1d22090a..8047fe1df065d63b 100644
> --- a/arch/arm64/boot/dts/renesas/salvator-common.dtsi
> +++ b/arch/arm64/boot/dts/renesas/salvator-common.dtsi
> @@ -41,7 +41,7 @@
>  	};
>  
>  	chosen {
> -		bootargs = "ignore_loglevel rw root=/dev/nfs ip=dhcp";
> +		bootargs = "ignore_loglevel rw root=/dev/nfs ip=dhcp adv748x.txbvc=1";
>  		stdout-path = "serial0:115200n8";
>  	};

Hi Niklas,

I'm somewhat surprised to see what appears to be a new module parameter.
I'm not going to reject this but did you give consideration to doing this
another way?

In any case I have marked this as "Deferred" pending acceptance of the
driver change. If you think it can go in now then I'm open to discussion.

Re: [PATCH 20/20] arm64: dts: renesas: salvator: use VC1 for CVBS

[Niklas Söderlund]

Hi Simon,

On 2017-08-30 09:36:37 +0200, Simon Horman wrote:
> On Fri, Aug 11, 2017 at 11:57:03AM +0200, Niklas Söderlund wrote:
> > In order to test Virtual Channels use VC1 for CVBS input from the
> > adv748x.
> > 
> > Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> > ---
> >  arch/arm64/boot/dts/renesas/salvator-common.dtsi | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/arch/arm64/boot/dts/renesas/salvator-common.dtsi b/arch/arm64/boot/dts/renesas/salvator-common.dtsi
> > index 7b67efcb1d22090a..8047fe1df065d63b 100644
> > --- a/arch/arm64/boot/dts/renesas/salvator-common.dtsi
> > +++ b/arch/arm64/boot/dts/renesas/salvator-common.dtsi
> > @@ -41,7 +41,7 @@
> >  	};
> >  
> >  	chosen {
> > -		bootargs = "ignore_loglevel rw root=/dev/nfs ip=dhcp";
> > +		bootargs = "ignore_loglevel rw root=/dev/nfs ip=dhcp adv748x.txbvc=1";
> >  		stdout-path = "serial0:115200n8";
> >  	};
> 
> Hi Niklas,
> 
> I'm somewhat surprised to see what appears to be a new module parameter.
> I'm not going to reject this but did you give consideration to doing this
> another way?

This is my fault when sending this series out it should be marked as RFC 
as it's stated in the cover-letter. This new module parameter is not 
intended to be unstreamed, not even the driver parts. It's only usage is 
to be able to easy test the multiplexed media pad using the onboard 
Salvator-X components.

> 
> In any case I have marked this as "Deferred" pending acceptance of the
> driver change. If you think it can go in now then I'm open to discussion.

You can mark it as rejected and forget about it :-)

-- 
Regards,
Niklas Söderlund

Re: [PATCH 20/20] arm64: dts: renesas: salvator: use VC1 for CVBS

[Niklas Söderlund]

Hi Simon,

On 2017-08-30 09:36:37 +0200, Simon Horman wrote:
> On Fri, Aug 11, 2017 at 11:57:03AM +0200, Niklas S�derlund wrote:
> > In order to test Virtual Channels use VC1 for CVBS input from the
> > adv748x.
> > 
> > Signed-off-by: Niklas S�derlund <niklas.soderlund+renesas@ragnatech.se>
> > ---
> >  arch/arm64/boot/dts/renesas/salvator-common.dtsi | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/arch/arm64/boot/dts/renesas/salvator-common.dtsi b/arch/arm64/boot/dts/renesas/salvator-common.dtsi
> > index 7b67efcb1d22090a..8047fe1df065d63b 100644
> > --- a/arch/arm64/boot/dts/renesas/salvator-common.dtsi
> > +++ b/arch/arm64/boot/dts/renesas/salvator-common.dtsi
> > @@ -41,7 +41,7 @@
> >  	};
> >  
> >  	chosen {
> > -		bootargs = "ignore_loglevel rw root=/dev/nfs ip=dhcp";
> > +		bootargs = "ignore_loglevel rw root=/dev/nfs ip=dhcp adv748x.txbvc=1";
> >  		stdout-path = "serial0:115200n8";
> >  	};
> 
> Hi Niklas,
> 
> I'm somewhat surprised to see what appears to be a new module parameter.
> I'm not going to reject this but did you give consideration to doing this
> another way?

This is my fault when sending this series out it should be marked as RFC 
as it's stated in the cover-letter. This new module parameter is not 
intended to be unstreamed, not even the driver parts. It's only usage is 
to be able to easy test the multiplexed media pad using the onboard 
Salvator-X components.

> 
> In any case I have marked this as "Deferred" pending acceptance of the
> driver change. If you think it can go in now then I'm open to discussion.

You can mark it as rejected and forget about it :-)

-- 
Regards,
Niklas S�derlund

Re: [PATCH 20/20] arm64: dts: renesas: salvator: use VC1 for CVBS

[Simon Horman]

On Wed, Aug 30, 2017 at 10:08:24AM +0200, Niklas Söderlund wrote:
> Hi Simon,
> 
> On 2017-08-30 09:36:37 +0200, Simon Horman wrote:
> > On Fri, Aug 11, 2017 at 11:57:03AM +0200, Niklas Söderlund wrote:
> > > In order to test Virtual Channels use VC1 for CVBS input from the
> > > adv748x.
> > > 
> > > Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> > > ---
> > >  arch/arm64/boot/dts/renesas/salvator-common.dtsi | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/arch/arm64/boot/dts/renesas/salvator-common.dtsi b/arch/arm64/boot/dts/renesas/salvator-common.dtsi
> > > index 7b67efcb1d22090a..8047fe1df065d63b 100644
> > > --- a/arch/arm64/boot/dts/renesas/salvator-common.dtsi
> > > +++ b/arch/arm64/boot/dts/renesas/salvator-common.dtsi
> > > @@ -41,7 +41,7 @@
> > >  	};
> > >  
> > >  	chosen {
> > > -		bootargs = "ignore_loglevel rw root=/dev/nfs ip=dhcp";
> > > +		bootargs = "ignore_loglevel rw root=/dev/nfs ip=dhcp adv748x.txbvc=1";
> > >  		stdout-path = "serial0:115200n8";
> > >  	};
> > 
> > Hi Niklas,
> > 
> > I'm somewhat surprised to see what appears to be a new module parameter.
> > I'm not going to reject this but did you give consideration to doing this
> > another way?
> 
> This is my fault when sending this series out it should be marked as RFC 
> as it's stated in the cover-letter. This new module parameter is not 
> intended to be unstreamed, not even the driver parts. It's only usage is 
> to be able to easy test the multiplexed media pad using the onboard 
> Salvator-X components.
> 
> > 
> > In any case I have marked this as "Deferred" pending acceptance of the
> > driver change. If you think it can go in now then I'm open to discussion.
> 
> You can mark it as rejected and forget about it :-)

Thanks for the clarification, I marked it as RFC :)

Re: [PATCH 00/20] Add multiplexed media pads to support CSI-2 virtual channels

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 4769 bytes --]

Hi Niklas,

On Fri, Aug 11, 2017 at 11:56:43AM +0200, Niklas Söderlund wrote:
> This series is a RFC for how I think one could add CSI-2 virtual channel 
> support to the V4L2 framework. The problem is that there is no way to in 
> the media framework describe and control links between subdevices which 
> carry more then one video stream, for example a CSI-2 bus which can have 
> 4 virtual channels carrying different video streams.
> 
> This series adds a new pad flag which would indicate that a pad carries 
> multiplexed streams, adds a new s_stream() operation to the pad 
> operations structure which takes a new argument 'stream'. This new 
> s_stream() operation then is both pad and stream aware. It also extends 
> struct v4l2_mbus_frame_desc_entry with a new sub-struct to describe how 
> a CSI-2 link multiplexes virtual channels. I also include one 
> implementation based on Renesas R-Car which makes use of these patches 
> as I think they help with understanding but they have no impact on the 
> RFC feature itself.
> 
> The idea is that on both sides of the multiplexed media link there are 
> one multiplexer subdevice and one demultiplexer subdevice. These two 
> subdevices can't do any format conversions, there sole purpose is to 
> (de)multiplex the CSI-2 link. If there is hardware which can do both 
> CSI-2 multiplexing and format conversions they can be modeled as two 
> subdevices from the same device driver and using the still pending 
> incremental async mechanism to connect the external pads. The reason 
> there is no format conversion is important as the multiplexed pads can't 
> have a format in the current V4L2 model, get/set_fmt are not aware of 
> streams.
> 
>         +------------------+              +------------------+
>      +-------+  subdev 1   |              |  subdev 2   +-------+
>   +--+ Pad 1 |             |              |             | Pad 3 +---+
>      +--+----+   +---------+---+      +---+---------+   +----+--+
>         |        | Muxed pad A +------+ Muxed pad B |        |
>      +--+----+   +---------+---+      +---+---------+   +----+--+
>   +--+ Pad 2 |             |              |             | Pad 4 +---+
>      +-------+             |              |             +-------+
>         +------------------+              +------------------+
> 
> In the simple example above Pad 1 is routed to Pad 3 and Pad 2 to Pad 4, 
> and the video data for both of them travels the link between pad A and 
> B. One shortcoming of this RFC is that there currently are no way to 
> express to user-space which pad is routed to which stream of the 
> multiplexed link. But inside the kernel this is known and format 
> validation is done by comparing the format of Pad 1 to Pad 3 and Pad 2 
> to Pad 4 by the V4L2 framework. But it would be nice for the user to 
> also be able to get this information while setting up the MC graph in 
> user-space.

Thanks for your patches.

I guess I already have a different use-case for this, one you might
not have envisionned (on top of the Cadence CSI2-RX driver I've been
posting and that would need it at some point).

I have a CSI2-TX controller that I'm currently writing a driver
for. That controller takes 4 video streams in input, and aggregates
them on a single CSI-2 link. Those video streams use some kind of
parallel interfaces. So far, so good.

However, those parallel interfaces come with additional signals that
set the virtual channel and data types of the associated video
signal. Those signals are under control of the upstream video output
device, and can change at runtime.

The virtual channel are direcly mapped to the CSI-2 Virtual Channels,
so that part is completely transparent to the transmitter. However,
the video input datatype is a 0-7 value. Each data type value has a
separate set of registers where you need to set up the width and
height of the video data, and the corresponding CSI-2 Data Type.

You can use this to do some interleaving of either the virtual
channels or the data types, or both.

To make things (slightly) more complicated, the FIFO registers are
per-video input. Which means that while most of the (input)
configuration will be done per-datatype, we still have to know which
input stream is generating it, for example to optimise the FIFO fill
levels to the resolution...

Since the stream is multiplexed both using the virtual channels and
the data-types, I'm not sure representing it using only muxed pads
like you did would work.

And I don't really know what a good stop gap measure would be either.

Thanks!
Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [PATCH 00/20] Add multiplexed media pads to support CSI-2 virtual channels

[Todor Tomov]

Hello,

I'm trying to understand what is the current state of the multiple virtual
channel support in V4L2 and Media framework. This is the last activity
on this topic which I was able to find. Is anything new happened since
this RFC, is someone working on this or planing to work on?

Best regards,
Todor Tomov

On 11.08.2017 12:56, Niklas Söderlund wrote:
> Hi,
> 
> This series is a RFC for how I think one could add CSI-2 virtual channel 
> support to the V4L2 framework. The problem is that there is no way to in 
> the media framework describe and control links between subdevices which 
> carry more then one video stream, for example a CSI-2 bus which can have 
> 4 virtual channels carrying different video streams.
> 
> This series adds a new pad flag which would indicate that a pad carries 
> multiplexed streams, adds a new s_stream() operation to the pad 
> operations structure which takes a new argument 'stream'. This new 
> s_stream() operation then is both pad and stream aware. It also extends 
> struct v4l2_mbus_frame_desc_entry with a new sub-struct to describe how 
> a CSI-2 link multiplexes virtual channels. I also include one 
> implementation based on Renesas R-Car which makes use of these patches 
> as I think they help with understanding but they have no impact on the 
> RFC feature itself.
> 
> The idea is that on both sides of the multiplexed media link there are 
> one multiplexer subdevice and one demultiplexer subdevice. These two 
> subdevices can't do any format conversions, there sole purpose is to 
> (de)multiplex the CSI-2 link. If there is hardware which can do both 
> CSI-2 multiplexing and format conversions they can be modeled as two 
> subdevices from the same device driver and using the still pending 
> incremental async mechanism to connect the external pads. The reason 
> there is no format conversion is important as the multiplexed pads can't 
> have a format in the current V4L2 model, get/set_fmt are not aware of 
> streams.
> 
>         +------------------+              +------------------+
>      +-------+  subdev 1   |              |  subdev 2   +-------+
>   +--+ Pad 1 |             |              |             | Pad 3 +---+
>      +--+----+   +---------+---+      +---+---------+   +----+--+
>         |        | Muxed pad A +------+ Muxed pad B |        |
>      +--+----+   +---------+---+      +---+---------+   +----+--+
>   +--+ Pad 2 |             |              |             | Pad 4 +---+
>      +-------+             |              |             +-------+
>         +------------------+              +------------------+
> 
> In the simple example above Pad 1 is routed to Pad 3 and Pad 2 to Pad 4, 
> and the video data for both of them travels the link between pad A and 
> B. One shortcoming of this RFC is that there currently are no way to 
> express to user-space which pad is routed to which stream of the 
> multiplexed link. But inside the kernel this is known and format 
> validation is done by comparing the format of Pad 1 to Pad 3 and Pad 2 
> to Pad 4 by the V4L2 framework. But it would be nice for the user to 
> also be able to get this information while setting up the MC graph in 
> user-space.
> 
> Obviously there are things that are not perfect in this RFC, one is the 
> above mentioned lack of user-space visibility of that Pad 1 is in fact 
> routed to Pad 3. Others are lack of Documentation/ work and I'm sure 
> there are error path shortcuts which are not fully thought out. One big 
> question is also if the s_stream() operation added to ops structure 
> should be a compliment to the existing ones in video and audio ops or 
> aim to replace the one in video ops. I'm also unsure of the CSI2 flag of 
> struct v4l2_mbus_frame_desc_entry don't really belong in struct 
> v4l2_mbus_frame_desc. And I'm sure there are lots of other stuff that's 
> why this is a RFC...
> 
> A big thanks to Laurent and Sakari for being really nice and taking time 
> helping me grasp all the possibilities and issues with this problem, all 
> cred to them and all blame to me for misunderstanding there guidance :-)
> 
> This series based on the latest R-Car CSI-2 and VIN patches which can be 
> found at [1], but that is a dependency only for the driver specific
> implementation which acts as an example of implementation. For the V4L2 
> framework patches the media-tree is the base.
> 
> 1. https://git.ragnatech.se/linux#rcar-vin-elinux-v12
> 
> Niklas Söderlund (20):
>   media.h: add MEDIA_PAD_FL_MUXED flag
>   v4l2-subdev.h: add pad and stream aware s_stream
>   v4l2-subdev.h: add CSI-2 bus description to struct
>     v4l2_mbus_frame_desc_entry
>   v4l2-core: check that both pads in a link are muxed if one are
>   v4l2-core: verify all streams formats on multiplexed links
>   rcar-vin: use the pad and stream aware s_stream
>   rcar-csi2: declare sink pad as multiplexed
>   rcar-csi2: switch to pad and stream aware s_stream
>   rcar-csi2: figure out remote pad and stream which are starting
>   rcar-csi2: count usage for each source pad
>   rcar-csi2: when starting CSI-2 receiver use frame descriptor
>     information
>   rcar-csi2: only allow formats on source pads
>   rcar-csi2: implement get_frame_desc
>   adv748x: add module param for virtual channel
>   adv748x: declare source pad as multiplexed
>   adv748x: add translation from pixelcode to CSI-2 datatype
>   adv748x: implement get_frame_desc
>   adv748x: switch to pad and stream aware s_stream
>   adv748x: only allow formats on sink pads
>   arm64: dts: renesas: salvator: use VC1 for CVBS
> 
>  arch/arm64/boot/dts/renesas/salvator-common.dtsi |   2 +-
>  drivers/media/i2c/adv748x/adv748x-core.c         |  10 +
>  drivers/media/i2c/adv748x/adv748x-csi2.c         |  78 +++++++-
>  drivers/media/i2c/adv748x/adv748x.h              |   1 +
>  drivers/media/platform/rcar-vin/rcar-csi2.c      | 239 ++++++++++++++++-------
>  drivers/media/platform/rcar-vin/rcar-dma.c       |   6 +-
>  drivers/media/v4l2-core/v4l2-subdev.c            |  65 ++++++
>  include/media/v4l2-subdev.h                      |  16 ++
>  include/uapi/linux/media.h                       |   1 +
>  9 files changed, 341 insertions(+), 77 deletions(-)
> 

-- 
Best regards,
Todor Tomov

Re: [PATCH 00/20] Add multiplexed media pads to support CSI-2 virtual channels

[Niklas Söderlund]

Hi Todor,

On 2018-03-15 11:43:31 +0200, Todor Tomov wrote:
> Hello,
> 
> I'm trying to understand what is the current state of the multiple virtual
> channel support in V4L2 and Media framework. This is the last activity
> on this topic which I was able to find. Is anything new happened since
> this RFC, is someone working on this or planing to work on?

I'm currently working on this but right now I'm focusing on driver 
dependencies for my use-case, once that is done I will resume to push 
more for the multiplexed stream support. I randomly push my latest work 
to

git://git.ragnatech.se/linux v4l2/mux

But this is a unstable branch and contains some LOCAL patches to test my 
work on Renesas platforms. But if you want to checkout my current status 
that's the branch to check.

Out of curiosity what board or use-case are you interested in where 
multiplexed streams would be useful for you?

> 
> Best regards,
> Todor Tomov
> 
> On 11.08.2017 12:56, Niklas Söderlund wrote:
> > Hi,
> > 
> > This series is a RFC for how I think one could add CSI-2 virtual channel 
> > support to the V4L2 framework. The problem is that there is no way to in 
> > the media framework describe and control links between subdevices which 
> > carry more then one video stream, for example a CSI-2 bus which can have 
> > 4 virtual channels carrying different video streams.
> > 
> > This series adds a new pad flag which would indicate that a pad carries 
> > multiplexed streams, adds a new s_stream() operation to the pad 
> > operations structure which takes a new argument 'stream'. This new 
> > s_stream() operation then is both pad and stream aware. It also extends 
> > struct v4l2_mbus_frame_desc_entry with a new sub-struct to describe how 
> > a CSI-2 link multiplexes virtual channels. I also include one 
> > implementation based on Renesas R-Car which makes use of these patches 
> > as I think they help with understanding but they have no impact on the 
> > RFC feature itself.
> > 
> > The idea is that on both sides of the multiplexed media link there are 
> > one multiplexer subdevice and one demultiplexer subdevice. These two 
> > subdevices can't do any format conversions, there sole purpose is to 
> > (de)multiplex the CSI-2 link. If there is hardware which can do both 
> > CSI-2 multiplexing and format conversions they can be modeled as two 
> > subdevices from the same device driver and using the still pending 
> > incremental async mechanism to connect the external pads. The reason 
> > there is no format conversion is important as the multiplexed pads can't 
> > have a format in the current V4L2 model, get/set_fmt are not aware of 
> > streams.
> > 
> >         +------------------+              +------------------+
> >      +-------+  subdev 1   |              |  subdev 2   +-------+
> >   +--+ Pad 1 |             |              |             | Pad 3 +---+
> >      +--+----+   +---------+---+      +---+---------+   +----+--+
> >         |        | Muxed pad A +------+ Muxed pad B |        |
> >      +--+----+   +---------+---+      +---+---------+   +----+--+
> >   +--+ Pad 2 |             |              |             | Pad 4 +---+
> >      +-------+             |              |             +-------+
> >         +------------------+              +------------------+
> > 
> > In the simple example above Pad 1 is routed to Pad 3 and Pad 2 to Pad 4, 
> > and the video data for both of them travels the link between pad A and 
> > B. One shortcoming of this RFC is that there currently are no way to 
> > express to user-space which pad is routed to which stream of the 
> > multiplexed link. But inside the kernel this is known and format 
> > validation is done by comparing the format of Pad 1 to Pad 3 and Pad 2 
> > to Pad 4 by the V4L2 framework. But it would be nice for the user to 
> > also be able to get this information while setting up the MC graph in 
> > user-space.
> > 
> > Obviously there are things that are not perfect in this RFC, one is the 
> > above mentioned lack of user-space visibility of that Pad 1 is in fact 
> > routed to Pad 3. Others are lack of Documentation/ work and I'm sure 
> > there are error path shortcuts which are not fully thought out. One big 
> > question is also if the s_stream() operation added to ops structure 
> > should be a compliment to the existing ones in video and audio ops or 
> > aim to replace the one in video ops. I'm also unsure of the CSI2 flag of 
> > struct v4l2_mbus_frame_desc_entry don't really belong in struct 
> > v4l2_mbus_frame_desc. And I'm sure there are lots of other stuff that's 
> > why this is a RFC...
> > 
> > A big thanks to Laurent and Sakari for being really nice and taking time 
> > helping me grasp all the possibilities and issues with this problem, all 
> > cred to them and all blame to me for misunderstanding there guidance :-)
> > 
> > This series based on the latest R-Car CSI-2 and VIN patches which can be 
> > found at [1], but that is a dependency only for the driver specific
> > implementation which acts as an example of implementation. For the V4L2 
> > framework patches the media-tree is the base.
> > 
> > 1. https://git.ragnatech.se/linux#rcar-vin-elinux-v12
> > 
> > Niklas Söderlund (20):
> >   media.h: add MEDIA_PAD_FL_MUXED flag
> >   v4l2-subdev.h: add pad and stream aware s_stream
> >   v4l2-subdev.h: add CSI-2 bus description to struct
> >     v4l2_mbus_frame_desc_entry
> >   v4l2-core: check that both pads in a link are muxed if one are
> >   v4l2-core: verify all streams formats on multiplexed links
> >   rcar-vin: use the pad and stream aware s_stream
> >   rcar-csi2: declare sink pad as multiplexed
> >   rcar-csi2: switch to pad and stream aware s_stream
> >   rcar-csi2: figure out remote pad and stream which are starting
> >   rcar-csi2: count usage for each source pad
> >   rcar-csi2: when starting CSI-2 receiver use frame descriptor
> >     information
> >   rcar-csi2: only allow formats on source pads
> >   rcar-csi2: implement get_frame_desc
> >   adv748x: add module param for virtual channel
> >   adv748x: declare source pad as multiplexed
> >   adv748x: add translation from pixelcode to CSI-2 datatype
> >   adv748x: implement get_frame_desc
> >   adv748x: switch to pad and stream aware s_stream
> >   adv748x: only allow formats on sink pads
> >   arm64: dts: renesas: salvator: use VC1 for CVBS
> > 
> >  arch/arm64/boot/dts/renesas/salvator-common.dtsi |   2 +-
> >  drivers/media/i2c/adv748x/adv748x-core.c         |  10 +
> >  drivers/media/i2c/adv748x/adv748x-csi2.c         |  78 +++++++-
> >  drivers/media/i2c/adv748x/adv748x.h              |   1 +
> >  drivers/media/platform/rcar-vin/rcar-csi2.c      | 239 ++++++++++++++++-------
> >  drivers/media/platform/rcar-vin/rcar-dma.c       |   6 +-
> >  drivers/media/v4l2-core/v4l2-subdev.c            |  65 ++++++
> >  include/media/v4l2-subdev.h                      |  16 ++
> >  include/uapi/linux/media.h                       |   1 +
> >  9 files changed, 341 insertions(+), 77 deletions(-)
> > 
> 
> -- 
> Best regards,
> Todor Tomov

-- 
Regards,
Niklas Söderlund

Re: [PATCH 00/20] Add multiplexed media pads to support CSI-2 virtual channels

[Niklas Söderlund]

Hi Todor,

On 2018-03-15 11:43:31 +0200, Todor Tomov wrote:
> Hello,
> 
> I'm trying to understand what is the current state of the multiple virtual
> channel support in V4L2 and Media framework. This is the last activity
> on this topic which I was able to find. Is anything new happened since
> this RFC, is someone working on this or planing to work on?

I'm currently working on this but right now I'm focusing on driver 
dependencies for my use-case, once that is done I will resume to push 
more for the multiplexed stream support. I randomly push my latest work 
to

git://git.ragnatech.se/linux v4l2/mux

But this is a unstable branch and contains some LOCAL patches to test my 
work on Renesas platforms. But if you want to checkout my current status 
that's the branch to check.

Out of curiosity what board or use-case are you interested in where 
multiplexed streams would be useful for you?

> 
> Best regards,
> Todor Tomov
> 
> On 11.08.2017 12:56, Niklas S�derlund wrote:
> > Hi,
> > 
> > This series is a RFC for how I think one could add CSI-2 virtual channel 
> > support to the V4L2 framework. The problem is that there is no way to in 
> > the media framework describe and control links between subdevices which 
> > carry more then one video stream, for example a CSI-2 bus which can have 
> > 4 virtual channels carrying different video streams.
> > 
> > This series adds a new pad flag which would indicate that a pad carries 
> > multiplexed streams, adds a new s_stream() operation to the pad 
> > operations structure which takes a new argument 'stream'. This new 
> > s_stream() operation then is both pad and stream aware. It also extends 
> > struct v4l2_mbus_frame_desc_entry with a new sub-struct to describe how 
> > a CSI-2 link multiplexes virtual channels. I also include one 
> > implementation based on Renesas R-Car which makes use of these patches 
> > as I think they help with understanding but they have no impact on the 
> > RFC feature itself.
> > 
> > The idea is that on both sides of the multiplexed media link there are 
> > one multiplexer subdevice and one demultiplexer subdevice. These two 
> > subdevices can't do any format conversions, there sole purpose is to 
> > (de)multiplex the CSI-2 link. If there is hardware which can do both 
> > CSI-2 multiplexing and format conversions they can be modeled as two 
> > subdevices from the same device driver and using the still pending 
> > incremental async mechanism to connect the external pads. The reason 
> > there is no format conversion is important as the multiplexed pads can't 
> > have a format in the current V4L2 model, get/set_fmt are not aware of 
> > streams.
> > 
> >         +------------------+              +------------------+
> >      +-------+  subdev 1   |              |  subdev 2   +-------+
> >   +--+ Pad 1 |             |              |             | Pad 3 +---+
> >      +--+----+   +---------+---+      +---+---------+   +----+--+
> >         |        | Muxed pad A +------+ Muxed pad B |        |
> >      +--+----+   +---------+---+      +---+---------+   +----+--+
> >   +--+ Pad 2 |             |              |             | Pad 4 +---+
> >      +-------+             |              |             +-------+
> >         +------------------+              +------------------+
> > 
> > In the simple example above Pad 1 is routed to Pad 3 and Pad 2 to Pad 4, 
> > and the video data for both of them travels the link between pad A and 
> > B. One shortcoming of this RFC is that there currently are no way to 
> > express to user-space which pad is routed to which stream of the 
> > multiplexed link. But inside the kernel this is known and format 
> > validation is done by comparing the format of Pad 1 to Pad 3 and Pad 2 
> > to Pad 4 by the V4L2 framework. But it would be nice for the user to 
> > also be able to get this information while setting up the MC graph in 
> > user-space.
> > 
> > Obviously there are things that are not perfect in this RFC, one is the 
> > above mentioned lack of user-space visibility of that Pad 1 is in fact 
> > routed to Pad 3. Others are lack of Documentation/ work and I'm sure 
> > there are error path shortcuts which are not fully thought out. One big 
> > question is also if the s_stream() operation added to ops structure 
> > should be a compliment to the existing ones in video and audio ops or 
> > aim to replace the one in video ops. I'm also unsure of the CSI2 flag of 
> > struct v4l2_mbus_frame_desc_entry don't really belong in struct 
> > v4l2_mbus_frame_desc. And I'm sure there are lots of other stuff that's 
> > why this is a RFC...
> > 
> > A big thanks to Laurent and Sakari for being really nice and taking time 
> > helping me grasp all the possibilities and issues with this problem, all 
> > cred to them and all blame to me for misunderstanding there guidance :-)
> > 
> > This series based on the latest R-Car CSI-2 and VIN patches which can be 
> > found at [1], but that is a dependency only for the driver specific
> > implementation which acts as an example of implementation. For the V4L2 
> > framework patches the media-tree is the base.
> > 
> > 1. https://git.ragnatech.se/linux#rcar-vin-elinux-v12
> > 
> > Niklas S�derlund (20):
> >   media.h: add MEDIA_PAD_FL_MUXED flag
> >   v4l2-subdev.h: add pad and stream aware s_stream
> >   v4l2-subdev.h: add CSI-2 bus description to struct
> >     v4l2_mbus_frame_desc_entry
> >   v4l2-core: check that both pads in a link are muxed if one are
> >   v4l2-core: verify all streams formats on multiplexed links
> >   rcar-vin: use the pad and stream aware s_stream
> >   rcar-csi2: declare sink pad as multiplexed
> >   rcar-csi2: switch to pad and stream aware s_stream
> >   rcar-csi2: figure out remote pad and stream which are starting
> >   rcar-csi2: count usage for each source pad
> >   rcar-csi2: when starting CSI-2 receiver use frame descriptor
> >     information
> >   rcar-csi2: only allow formats on source pads
> >   rcar-csi2: implement get_frame_desc
> >   adv748x: add module param for virtual channel
> >   adv748x: declare source pad as multiplexed
> >   adv748x: add translation from pixelcode to CSI-2 datatype
> >   adv748x: implement get_frame_desc
> >   adv748x: switch to pad and stream aware s_stream
> >   adv748x: only allow formats on sink pads
> >   arm64: dts: renesas: salvator: use VC1 for CVBS
> > 
> >  arch/arm64/boot/dts/renesas/salvator-common.dtsi |   2 +-
> >  drivers/media/i2c/adv748x/adv748x-core.c         |  10 +
> >  drivers/media/i2c/adv748x/adv748x-csi2.c         |  78 +++++++-
> >  drivers/media/i2c/adv748x/adv748x.h              |   1 +
> >  drivers/media/platform/rcar-vin/rcar-csi2.c      | 239 ++++++++++++++++-------
> >  drivers/media/platform/rcar-vin/rcar-dma.c       |   6 +-
> >  drivers/media/v4l2-core/v4l2-subdev.c            |  65 ++++++
> >  include/media/v4l2-subdev.h                      |  16 ++
> >  include/uapi/linux/media.h                       |   1 +
> >  9 files changed, 341 insertions(+), 77 deletions(-)
> > 
> 
> -- 
> Best regards,
> Todor Tomov

-- 
Regards,
Niklas S�derlund

Re: [PATCH 00/20] Add multiplexed media pads to support CSI-2 virtual channels

[Todor Tomov]

Hi Niklas,

On 16.03.2018 01:16, Niklas Söderlund wrote:
> Hi Todor,
> 
> On 2018-03-15 11:43:31 +0200, Todor Tomov wrote:
>> Hello,
>>
>> I'm trying to understand what is the current state of the multiple virtual
>> channel support in V4L2 and Media framework. This is the last activity
>> on this topic which I was able to find. Is anything new happened since
>> this RFC, is someone working on this or planing to work on?
> 
> I'm currently working on this but right now I'm focusing on driver 
> dependencies for my use-case, once that is done I will resume to push 
> more for the multiplexed stream support. I randomly push my latest work 
> to
> 
> git://git.ragnatech.se/linux v4l2/mux
> 
> But this is a unstable branch and contains some LOCAL patches to test my 
> work on Renesas platforms. But if you want to checkout my current status 
> that's the branch to check.
> 
> Out of curiosity what board or use-case are you interested in where 
> multiplexed streams would be useful for you?

Good to hear that you are still working on this :)
I'm working on QComm SoCs and I may need multiple virtual channel
support however the usecases are not clear yet. I'll be keeping an
eye on any activity from you then ;) or when I have more details
about my usecases I'll try to evaluate how they fit with this RFC.

Thank you.
Best regards,
Todor Tomov

> 
>>
>> Best regards,
>> Todor Tomov
>>
>> On 11.08.2017 12:56, Niklas Söderlund wrote:
>>> Hi,
>>>
>>> This series is a RFC for how I think one could add CSI-2 virtual channel 
>>> support to the V4L2 framework. The problem is that there is no way to in 
>>> the media framework describe and control links between subdevices which 
>>> carry more then one video stream, for example a CSI-2 bus which can have 
>>> 4 virtual channels carrying different video streams.
>>>
>>> This series adds a new pad flag which would indicate that a pad carries 
>>> multiplexed streams, adds a new s_stream() operation to the pad 
>>> operations structure which takes a new argument 'stream'. This new 
>>> s_stream() operation then is both pad and stream aware. It also extends 
>>> struct v4l2_mbus_frame_desc_entry with a new sub-struct to describe how 
>>> a CSI-2 link multiplexes virtual channels. I also include one 
>>> implementation based on Renesas R-Car which makes use of these patches 
>>> as I think they help with understanding but they have no impact on the 
>>> RFC feature itself.
>>>
>>> The idea is that on both sides of the multiplexed media link there are 
>>> one multiplexer subdevice and one demultiplexer subdevice. These two 
>>> subdevices can't do any format conversions, there sole purpose is to 
>>> (de)multiplex the CSI-2 link. If there is hardware which can do both 
>>> CSI-2 multiplexing and format conversions they can be modeled as two 
>>> subdevices from the same device driver and using the still pending 
>>> incremental async mechanism to connect the external pads. The reason 
>>> there is no format conversion is important as the multiplexed pads can't 
>>> have a format in the current V4L2 model, get/set_fmt are not aware of 
>>> streams.
>>>
>>>         +------------------+              +------------------+
>>>      +-------+  subdev 1   |              |  subdev 2   +-------+
>>>   +--+ Pad 1 |             |              |             | Pad 3 +---+
>>>      +--+----+   +---------+---+      +---+---------+   +----+--+
>>>         |        | Muxed pad A +------+ Muxed pad B |        |
>>>      +--+----+   +---------+---+      +---+---------+   +----+--+
>>>   +--+ Pad 2 |             |              |             | Pad 4 +---+
>>>      +-------+             |              |             +-------+
>>>         +------------------+              +------------------+
>>>
>>> In the simple example above Pad 1 is routed to Pad 3 and Pad 2 to Pad 4, 
>>> and the video data for both of them travels the link between pad A and 
>>> B. One shortcoming of this RFC is that there currently are no way to 
>>> express to user-space which pad is routed to which stream of the 
>>> multiplexed link. But inside the kernel this is known and format 
>>> validation is done by comparing the format of Pad 1 to Pad 3 and Pad 2 
>>> to Pad 4 by the V4L2 framework. But it would be nice for the user to 
>>> also be able to get this information while setting up the MC graph in 
>>> user-space.
>>>
>>> Obviously there are things that are not perfect in this RFC, one is the 
>>> above mentioned lack of user-space visibility of that Pad 1 is in fact 
>>> routed to Pad 3. Others are lack of Documentation/ work and I'm sure 
>>> there are error path shortcuts which are not fully thought out. One big 
>>> question is also if the s_stream() operation added to ops structure 
>>> should be a compliment to the existing ones in video and audio ops or 
>>> aim to replace the one in video ops. I'm also unsure of the CSI2 flag of 
>>> struct v4l2_mbus_frame_desc_entry don't really belong in struct 
>>> v4l2_mbus_frame_desc. And I'm sure there are lots of other stuff that's 
>>> why this is a RFC...
>>>
>>> A big thanks to Laurent and Sakari for being really nice and taking time 
>>> helping me grasp all the possibilities and issues with this problem, all 
>>> cred to them and all blame to me for misunderstanding there guidance :-)
>>>
>>> This series based on the latest R-Car CSI-2 and VIN patches which can be 
>>> found at [1], but that is a dependency only for the driver specific
>>> implementation which acts as an example of implementation. For the V4L2 
>>> framework patches the media-tree is the base.
>>>
>>> 1. https://git.ragnatech.se/linux#rcar-vin-elinux-v12
>>>
>>> Niklas Söderlund (20):
>>>   media.h: add MEDIA_PAD_FL_MUXED flag
>>>   v4l2-subdev.h: add pad and stream aware s_stream
>>>   v4l2-subdev.h: add CSI-2 bus description to struct
>>>     v4l2_mbus_frame_desc_entry
>>>   v4l2-core: check that both pads in a link are muxed if one are
>>>   v4l2-core: verify all streams formats on multiplexed links
>>>   rcar-vin: use the pad and stream aware s_stream
>>>   rcar-csi2: declare sink pad as multiplexed
>>>   rcar-csi2: switch to pad and stream aware s_stream
>>>   rcar-csi2: figure out remote pad and stream which are starting
>>>   rcar-csi2: count usage for each source pad
>>>   rcar-csi2: when starting CSI-2 receiver use frame descriptor
>>>     information
>>>   rcar-csi2: only allow formats on source pads
>>>   rcar-csi2: implement get_frame_desc
>>>   adv748x: add module param for virtual channel
>>>   adv748x: declare source pad as multiplexed
>>>   adv748x: add translation from pixelcode to CSI-2 datatype
>>>   adv748x: implement get_frame_desc
>>>   adv748x: switch to pad and stream aware s_stream
>>>   adv748x: only allow formats on sink pads
>>>   arm64: dts: renesas: salvator: use VC1 for CVBS
>>>
>>>  arch/arm64/boot/dts/renesas/salvator-common.dtsi |   2 +-
>>>  drivers/media/i2c/adv748x/adv748x-core.c         |  10 +
>>>  drivers/media/i2c/adv748x/adv748x-csi2.c         |  78 +++++++-
>>>  drivers/media/i2c/adv748x/adv748x.h              |   1 +
>>>  drivers/media/platform/rcar-vin/rcar-csi2.c      | 239 ++++++++++++++++-------
>>>  drivers/media/platform/rcar-vin/rcar-dma.c       |   6 +-
>>>  drivers/media/v4l2-core/v4l2-subdev.c            |  65 ++++++
>>>  include/media/v4l2-subdev.h                      |  16 ++
>>>  include/uapi/linux/media.h                       |   1 +
>>>  9 files changed, 341 insertions(+), 77 deletions(-)
>>>
>>
>> -- 
>> Best regards,
>> Todor Tomov
> 

-- 
Best regards,
Todor Tomov