[PATCH 00/22] Basic i.MX IPUv3 capture support

[PATCH 00/22] Basic i.MX IPUv3 capture support

[Philipp Zabel]

Hi,

this series adds very basic IPUv3 capture support for i.MX5/6 SoCs via a SoC
global media controller device that collects all IPUv3 CSIs and their direct
sources from the device tree via of_graph bindings. The device is probed from
device tree using a capture-subsystem node similarly to the already existing
display-subsystem node for imx-drm.
Each subdevice is then allowed to add further still missing subdevices to the
asynchronous subdevice notifier waitlist until the graph is complete.

Each CSI subdevice gets assigned an ipu-capture video_device that corresponds
to one direct CSI -> SMFC -> IDMAC channel path.
This series does not contain IC support yet. I'd like to add the IC as another
subdevice with one sink pad (PRP) that can be connected to either CSI, and two
source pads (PRP VF and PRP ENC) that each would get their own video_device.

Also included are drivers for the video bus multiplexers in front of the CSIs
and for the DesignWare MIPI CSI-2 Host Controller and an example device tree
configuration for TC358743 on Nitrogen6X.
This is the output of media-ctl --print-dot:

digraph board {
	rankdir=TB
	n00000001 [label="{{<port0> 0} | IPU0 CSI0\n/dev/v4l-subdev0 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
	n00000001:port1 -> n00000004
	n00000004 [label="imx-ipuv3-capture.0\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
	n0000000a [label="{{<port0> 0} | IPU0 CSI1\n/dev/v4l-subdev1 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
	n0000000a:port1 -> n0000000d
	n0000000d [label="imx-ipuv3-capture.1\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
	n00000013 [label="{{<port0> 0} | IPU1 CSI0\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
	n00000013:port1 -> n00000016
	n00000016 [label="imx-ipuv3-capture.0\n/dev/video2", shape=box, style=filled, fillcolor=yellow]
	n0000001c [label="{{<port0> 0} | IPU1 CSI1\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
	n0000001c:port1 -> n0000001f
	n0000001f [label="imx-ipuv3-capture.1\n/dev/video3", shape=box, style=filled, fillcolor=yellow]
	n00000025 [label="{{<port0> 0 | <port1> 1} | mipi_ipu1_mux\n/dev/v4l-subdev4 | {<port2> 2}}", shape=Mrecord, style=filled, fillcolor=green]
	n00000025:port2 -> n00000001:port0
	n00000029 [label="{{<port0> 0 | <port1> 1} | mipi_ipu2_mux\n/dev/v4l-subdev5 | {<port2> 2}}", shape=Mrecord, style=filled, fillcolor=green]
	n00000029:port2 -> n0000001c:port0 [style=dashed]
	n0000002d [label="{{<port0> 0} | mipi-csi2\n/dev/v4l-subdev6 | {<port1> 1 | <port2> 2 | <port3> 3 | <port4> 4}}", shape=Mrecord, style=filled, fillcolor=green]
	n0000002d:port1 -> n00000025:port0
	n0000002d:port4 -> n00000029:port0 [style=dashed]
	n0000002d:port3 -> n00000013:port0 [style=dashed]
	n0000002d:port2 -> n0000000a:port0 [style=dashed]
	n00000033 [label="{{} | tc358743 1-000f\n/dev/v4l-subdev7 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
	n00000033:port0 -> n0000002d:port0
}

regards
Philipp

Philipp Zabel (21):
  [media] v4l2-async: move code out of v4l2_async_notifier_register into
    v4l2_async_test_nofity_all
  [media] v4l2-async: allow subdevices to add further subdevices to the
    notifier waiting list
  [media] v4l: of: add v4l2_of_subdev_registered
  [media] v4l2-subdev.h: add prepare_stream op
  [media] v4l2-async: add new subdevices to the tail of subdev_list
  [media] imx: Add i.MX SoC wide media device driver
  [media] imx-ipu: Add i.MX IPUv3 CSI subdevice driver
  [media] imx: Add i.MX IPUv3 capture driver
  [media] platform: add video-multiplexer subdevice driver
  [media] imx: Add i.MX MIPI CSI-2 subdevice driver
  [media] tc358743: put lanes in STOP state before starting streaming
  ARM: dts: imx6qdl: Add capture-subsystem node
  ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their
    connections
  ARM: dts: imx6qdl: Add MIPI CSI-2 D-PHY compatible and clocks
  ARM: dts: nitrogen6x: Add dtsi for BD_HDMI_MIPI HDMI to MIPI CSI-2
    receiver board
  gpu: ipuv3: add ipu_csi_set_downsize
  [media] imx-ipuv3-csi: support downsizing
  [media] add mux and video interface bridge entity functions
  [media] video-multiplexer: set entity function to mux
  [media] imx: Set i.MX MIPI CSI-2 entity function to bridge
  [media] tc358743: set entity function to video interface bridge

Sascha Hauer (1):
  [media] imx: Add IPUv3 media common code

 .../devicetree/bindings/media/fsl-imx-capture.txt  |   92 ++
 .../bindings/media/video-multiplexer.txt           |   59 ++
 Documentation/media/uapi/mediactl/media-types.rst  |   22 +
 arch/arm/boot/dts/imx6dl.dtsi                      |  187 ++++
 arch/arm/boot/dts/imx6q.dtsi                       |  123 +++
 .../boot/dts/imx6qdl-nitrogen6x-bd-hdmi-mipi.dtsi  |   73 ++
 arch/arm/boot/dts/imx6qdl.dtsi                     |   17 +-
 drivers/gpu/ipu-v3/ipu-csi.c                       |   16 +
 drivers/media/i2c/tc358743.c                       |   10 +
 drivers/media/platform/Kconfig                     |   10 +
 drivers/media/platform/Makefile                    |    3 +
 drivers/media/platform/imx/Kconfig                 |   33 +
 drivers/media/platform/imx/Makefile                |    5 +
 drivers/media/platform/imx/imx-ipu-capture.c       | 1085 ++++++++++++++++++++
 drivers/media/platform/imx/imx-ipu.c               |  321 ++++++
 drivers/media/platform/imx/imx-ipu.h               |   43 +
 drivers/media/platform/imx/imx-ipuv3-csi.c         |  555 ++++++++++
 drivers/media/platform/imx/imx-media.c             |  249 +++++
 drivers/media/platform/imx/imx-mipi-csi2.c         |  677 ++++++++++++
 drivers/media/platform/video-multiplexer.c         |  445 ++++++++
 drivers/media/v4l2-core/v4l2-async.c               |   96 +-
 drivers/media/v4l2-core/v4l2-of.c                  |   68 ++
 include/media/v4l2-async.h                         |   12 +
 include/media/v4l2-of.h                            |   12 +
 include/media/v4l2-subdev.h                        |    1 +
 include/uapi/linux/media.h                         |    6 +
 include/video/imx-ipu-v3.h                         |    1 +
 27 files changed, 4210 insertions(+), 11 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/media/fsl-imx-capture.txt
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 arch/arm/boot/dts/imx6qdl-nitrogen6x-bd-hdmi-mipi.dtsi
 create mode 100644 drivers/media/platform/imx/Kconfig
 create mode 100644 drivers/media/platform/imx/Makefile
 create mode 100644 drivers/media/platform/imx/imx-ipu-capture.c
 create mode 100644 drivers/media/platform/imx/imx-ipu.c
 create mode 100644 drivers/media/platform/imx/imx-ipu.h
 create mode 100644 drivers/media/platform/imx/imx-ipuv3-csi.c
 create mode 100644 drivers/media/platform/imx/imx-media.c
 create mode 100644 drivers/media/platform/imx/imx-mipi-csi2.c
 create mode 100644 drivers/media/platform/video-multiplexer.c

-- 
2.9.3

[PATCH 01/22] [media] v4l2-async: move code out of v4l2_async_notifier_register into v4l2_async_test_nofity_all

[Philipp Zabel]

This will be reused in the following patch to catch already registered,
newly added asynchronous subdevices from v4l2_async_register_subdev.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/media/v4l2-core/v4l2-async.c | 38 +++++++++++++++++++++---------------
 1 file changed, 22 insertions(+), 16 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-async.c b/drivers/media/v4l2-core/v4l2-async.c
index 5bada20..c4f1930 100644
--- a/drivers/media/v4l2-core/v4l2-async.c
+++ b/drivers/media/v4l2-core/v4l2-async.c
@@ -134,11 +134,31 @@ static void v4l2_async_cleanup(struct v4l2_subdev *sd)
 	sd->dev = NULL;
 }
 
+static int v4l2_async_test_notify_all(struct v4l2_async_notifier *notifier)
+{
+	struct v4l2_subdev *sd, *tmp;
+
+	list_for_each_entry_safe(sd, tmp, &subdev_list, async_list) {
+		struct v4l2_async_subdev *asd;
+		int ret;
+
+		asd = v4l2_async_belongs(notifier, sd);
+		if (!asd)
+			continue;
+
+		ret = v4l2_async_test_notify(notifier, sd, asd);
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
 int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
 				 struct v4l2_async_notifier *notifier)
 {
-	struct v4l2_subdev *sd, *tmp;
 	struct v4l2_async_subdev *asd;
+	int ret;
 	int i;
 
 	if (!notifier->num_subdevs || notifier->num_subdevs > V4L2_MAX_SUBDEVS)
@@ -171,23 +191,9 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
 	/* Keep also completed notifiers on the list */
 	list_add(&notifier->list, &notifier_list);
 
-	list_for_each_entry_safe(sd, tmp, &subdev_list, async_list) {
-		int ret;
-
-		asd = v4l2_async_belongs(notifier, sd);
-		if (!asd)
-			continue;
-
-		ret = v4l2_async_test_notify(notifier, sd, asd);
-		if (ret < 0) {
-			mutex_unlock(&list_lock);
-			return ret;
-		}
-	}
-
 	mutex_unlock(&list_lock);
 
-	return 0;
+	return ret;
 }
 EXPORT_SYMBOL(v4l2_async_notifier_register);
 
-- 
2.9.3

[PATCH 02/22] [media] v4l2-async: allow subdevices to add further subdevices to the notifier waiting list

[Philipp Zabel]

Currently the v4l2_async_notifier needs to be given a list of matches
for all expected subdevices on creation. When chaining subdevices that
are asynchronously probed via device tree, the bridge device that sets
up the notifier does not know the complete list of subdevices, as it
can only parse its own device tree node to obtain information about
the nearest neighbor subdevices.
To support indirectly connected subdevices, we need to support amending
the existing notifier waiting list with newly found neighbor subdevices
with each registered subdevice.

This can be achieved by adding new v42l_async_subdev matches to the
notifier waiting list during the v4l2_subdev registered callback, which
is called under the list lock from either v4l2_async_register_subdev or
v4l2_async_notifier_register. For this purpose a new exported function
__v4l2_async_notifier_add_subdev is added.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/media/v4l2-core/v4l2-async.c | 78 ++++++++++++++++++++++++++++++++++--
 include/media/v4l2-async.h           | 12 ++++++
 2 files changed, 86 insertions(+), 4 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-async.c b/drivers/media/v4l2-core/v4l2-async.c
index c4f1930..404eeea 100644
--- a/drivers/media/v4l2-core/v4l2-async.c
+++ b/drivers/media/v4l2-core/v4l2-async.c
@@ -109,6 +109,7 @@ static int v4l2_async_test_notify(struct v4l2_async_notifier *notifier,
 		if (ret < 0)
 			return ret;
 	}
+
 	/* Move from the global subdevice list to notifier's done */
 	list_move(&sd->async_list, &notifier->done);
 
@@ -158,7 +159,7 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
 				 struct v4l2_async_notifier *notifier)
 {
 	struct v4l2_async_subdev *asd;
-	int ret;
+	struct list_head *tail;
 	int i;
 
 	if (!notifier->num_subdevs || notifier->num_subdevs > V4L2_MAX_SUBDEVS)
@@ -191,17 +192,71 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
 	/* Keep also completed notifiers on the list */
 	list_add(&notifier->list, &notifier_list);
 
+	do {
+		int ret;
+
+		tail = notifier->waiting.prev;
+
+		ret = v4l2_async_test_notify_all(notifier);
+		if (ret < 0) {
+			mutex_unlock(&list_lock);
+			return ret;
+		}
+
+		/*
+		 * If entries were added to the notifier waiting list, check
+		 * again if the corresponding subdevices are already available.
+		 */
+	} while (tail != notifier->waiting.prev);
+
 	mutex_unlock(&list_lock);
 
-	return ret;
+	return 0;
 }
 EXPORT_SYMBOL(v4l2_async_notifier_register);
 
+int __v4l2_async_notifier_add_subdev(struct v4l2_async_notifier *notifier,
+				     struct v4l2_async_subdev *asd)
+{
+	struct v4l2_async_subdev *tmp_asd;
+
+	lockdep_assert_held(&list_lock);
+
+	if (asd->match_type != V4L2_ASYNC_MATCH_OF)
+		return -EINVAL;
+
+	/*
+	 * First check if the same notifier is already on the waiting or done
+	 * lists. This can happen if a subdevice with multiple outputs is added
+	 * by all its downstream subdevices.
+	 */
+	list_for_each_entry(tmp_asd, &notifier->waiting, list)
+		if (tmp_asd->match.of.node == asd->match.of.node)
+			return 0;
+	list_for_each_entry(tmp_asd, &notifier->done, list)
+		if (tmp_asd->match.of.node == asd->match.of.node)
+			return 0;
+
+	/*
+	 * Add the new async subdev to the notifier waiting list, so
+	 * v4l2_async_belongs may use it to compare against entries in
+	 * subdev_list.
+	 * In case the subdev matching asd has already been passed in the
+	 * subdev_list walk in v4l2_async_notifier_register, or if
+	 * we are called from v4l2_async_register_subdev, the subdev_list
+	 * will have to be walked again.
+	 */
+	list_add_tail(&asd->list, &notifier->waiting);
+
+	return 0;
+}
+
 void v4l2_async_notifier_unregister(struct v4l2_async_notifier *notifier)
 {
 	struct v4l2_subdev *sd, *tmp;
-	unsigned int notif_n_subdev = notifier->num_subdevs;
-	unsigned int n_subdev = min(notif_n_subdev, V4L2_MAX_SUBDEVS);
+	unsigned int notif_n_subdev = 0;
+	unsigned int n_subdev;
+	struct list_head *list;
 	struct device **dev;
 	int i = 0;
 
@@ -218,6 +273,10 @@ void v4l2_async_notifier_unregister(struct v4l2_async_notifier *notifier)
 
 	list_del(&notifier->list);
 
+	list_for_each(list, &notifier->done)
+		++notif_n_subdev;
+	n_subdev = min(notif_n_subdev, V4L2_MAX_SUBDEVS);
+
 	list_for_each_entry_safe(sd, tmp, &notifier->done, async_list) {
 		struct device *d;
 
@@ -294,8 +353,19 @@ int v4l2_async_register_subdev(struct v4l2_subdev *sd)
 	list_for_each_entry(notifier, &notifier_list, list) {
 		struct v4l2_async_subdev *asd = v4l2_async_belongs(notifier, sd);
 		if (asd) {
+			struct list_head *tail = notifier->waiting.prev;
 			int ret = v4l2_async_test_notify(notifier, sd, asd);
+
+			/*
+			 * If entries were added to the notifier waiting list,
+			 * check if the corresponding subdevices are already
+			 * available.
+			 */
+			if (tail != notifier->waiting.prev)
+				ret = v4l2_async_test_notify_all(notifier);
+
 			mutex_unlock(&list_lock);
+
 			return ret;
 		}
 	}
diff --git a/include/media/v4l2-async.h b/include/media/v4l2-async.h
index 8e2a236..e4e4b11 100644
--- a/include/media/v4l2-async.h
+++ b/include/media/v4l2-async.h
@@ -114,6 +114,18 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
 				 struct v4l2_async_notifier *notifier);
 
 /**
+ * __v4l2_async_notifier_add_subdev - adds a subdevice to the notifier waitlist
+ *
+ * @v4l2_notifier: notifier the calling subdev is bound to
+ * @asd: asynchronous subdev match
+ *
+ * To be called from inside a subdevices' registered_async callback to add
+ * additional subdevices to the notifier waiting list.
+ */
+int __v4l2_async_notifier_add_subdev(struct v4l2_async_notifier *notifier,
+				     struct v4l2_async_subdev *asd);
+
+/**
  * v4l2_async_notifier_unregister - unregisters a subdevice asynchronous notifier
  *
  * @notifier: pointer to &struct v4l2_async_notifier
-- 
2.9.3

[PATCH 03/22] [media] v4l: of: add v4l2_of_subdev_registered

[Philipp Zabel]

Provide a default registered callback for device tree probed subdevices
that use OF graph bindings to add still missing source subdevices to
the async notifier waiting list.
This is only necessary for subdevices that have input ports to which
other subdevices are connected that are not initially known to the
master/bridge device when it sets up the notifier.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/media/v4l2-core/v4l2-of.c | 68 +++++++++++++++++++++++++++++++++++++++
 include/media/v4l2-of.h           | 12 +++++++
 2 files changed, 80 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-of.c b/drivers/media/v4l2-core/v4l2-of.c
index 93b3368..fbdd6b4 100644
--- a/drivers/media/v4l2-core/v4l2-of.c
+++ b/drivers/media/v4l2-core/v4l2-of.c
@@ -19,6 +19,7 @@
 #include <linux/types.h>
 
 #include <media/v4l2-of.h>
+#include <media/v4l2-device.h>
 
 static int v4l2_of_parse_csi_bus(const struct device_node *node,
 				 struct v4l2_of_endpoint *endpoint)
@@ -314,3 +315,70 @@ void v4l2_of_put_link(struct v4l2_of_link *link)
 	of_node_put(link->remote_node);
 }
 EXPORT_SYMBOL(v4l2_of_put_link);
+
+struct v4l2_subdev *v4l2_find_subdev_by_node(struct v4l2_device *v4l2_dev,
+					     struct device_node *node)
+{
+	struct v4l2_subdev *sd;
+
+	list_for_each_entry(sd, &v4l2_dev->subdevs, list) {
+		if (sd->of_node == node)
+			return sd;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL(v4l2_find_subdev_by_node);
+
+/**
+ * v4l2_of_subdev_registered() - default OF probed subdev registered callback
+ * @subdev: subdevice with initialized entities
+ *
+ * Parse all OF graph endpoints corrensponding to the subdev's entity input pads
+ * and add the remote subdevs to the async subdev notifier.
+ */
+int v4l2_of_subdev_registered(struct v4l2_subdev *sd)
+{
+	struct device_node *ep;
+
+	for_each_endpoint_of_node(sd->of_node, ep) {
+		struct v4l2_of_link link;
+		struct media_entity *entity;
+		unsigned int pad;
+		int ret;
+
+		ret = v4l2_of_parse_link(ep, &link);
+		if (ret)
+			continue;
+
+		/*
+		 * Assume 1:1 correspondence between OF node and entity,
+		 * and between OF port numbers and pad indices.
+		 */
+		entity = &sd->entity;
+		pad = link.local_port;
+
+		if (pad >= entity->num_pads)
+			return -EINVAL;
+
+		/* Add source subdevs to async notifier */
+		if (entity->pads[pad].flags & MEDIA_PAD_FL_SINK) {
+			struct v4l2_async_subdev *asd;
+
+			asd = devm_kzalloc(sd->dev, sizeof(*asd), GFP_KERNEL);
+			if (!asd) {
+				v4l2_of_put_link(&link);
+				return -ENOMEM;
+			}
+
+			asd->match_type = V4L2_ASYNC_MATCH_OF;
+			asd->match.of.node = link.remote_node;
+
+			__v4l2_async_notifier_add_subdev(sd->notifier, asd);
+		}
+
+		v4l2_of_put_link(&link);
+	}
+
+	return 0;
+}
diff --git a/include/media/v4l2-of.h b/include/media/v4l2-of.h
index 4dc34b2..67d4f8b 100644
--- a/include/media/v4l2-of.h
+++ b/include/media/v4l2-of.h
@@ -22,6 +22,8 @@
 #include <media/v4l2-mediabus.h>
 
 struct device_node;
+struct v4l2_device;
+struct v4l2_subdev;
 
 /**
  * struct v4l2_of_bus_mipi_csi2 - MIPI CSI-2 bus data structure
@@ -95,6 +97,9 @@ void v4l2_of_free_endpoint(struct v4l2_of_endpoint *endpoint);
 int v4l2_of_parse_link(const struct device_node *node,
 		       struct v4l2_of_link *link);
 void v4l2_of_put_link(struct v4l2_of_link *link);
+int v4l2_of_subdev_registered(struct v4l2_subdev *sd);
+struct v4l2_subdev *v4l2_find_subdev_by_node(struct v4l2_device *v4l2_dev,
+					     struct device_node *node);
 #else /* CONFIG_OF */
 
 static inline int v4l2_of_parse_endpoint(const struct device_node *node,
@@ -123,6 +128,13 @@ static inline void v4l2_of_put_link(struct v4l2_of_link *link)
 {
 }
 
+#define v4l2_of_subdev_registered NULL
+
+struct v4l2_subdev *v4l2_find_subdev_by_node(struct v4l2_device *v4l2_dev,
+					     struct device_node *node)
+{
+	return NULL;
+}
 #endif /* CONFIG_OF */
 
 #endif /* _V4L2_OF_H */
-- 
2.9.3

[PATCH 04/22] [media] v4l2-subdev.h: add prepare_stream op

[Philipp Zabel]

In some cases, for example MIPI CSI-2 input on i.MX6, the sending and
receiving subdevice need to be prepared in lock-step before the actual
streaming can start. In the i.MX6 MIPI CSI-2 case, the sender needs to
put its MIPI CSI-2 transmitter lanes into stop state, and the receiver
needs to configure its D-PHY and detect the stop state on all active
lanes. Only then the sender can be enabled to stream data and the
receiver can lock its PLL to the clock lane.

This patch adds a prepare_stream(sd) callback that can be issued to all
v4l2_subdevs before calling s_stream(sd, 1).

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 include/media/v4l2-subdev.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index cf778c5..6502f43 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -395,6 +395,7 @@ struct v4l2_subdev_video_ops {
 	int (*g_tvnorms)(struct v4l2_subdev *sd, v4l2_std_id *std);
 	int (*g_tvnorms_output)(struct v4l2_subdev *sd, v4l2_std_id *std);
 	int (*g_input_status)(struct v4l2_subdev *sd, u32 *status);
+	int (*prepare_stream)(struct v4l2_subdev *sd);
 	int (*s_stream)(struct v4l2_subdev *sd, int enable);
 	int (*g_pixelaspect)(struct v4l2_subdev *sd, struct v4l2_fract *aspect);
 	int (*g_parm)(struct v4l2_subdev *sd, struct v4l2_streamparm *param);
-- 
2.9.3

[PATCH 05/22] [media] v4l2-async: add new subdevices to the tail of subdev_list

[Philipp Zabel]

That way the asynchronous notifier will pick them up in the order they
were registered.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/media/v4l2-core/v4l2-async.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-async.c b/drivers/media/v4l2-core/v4l2-async.c
index 404eeea..1a62ed7 100644
--- a/drivers/media/v4l2-core/v4l2-async.c
+++ b/drivers/media/v4l2-core/v4l2-async.c
@@ -371,7 +371,7 @@ int v4l2_async_register_subdev(struct v4l2_subdev *sd)
 	}
 
 	/* None matched, wait for hot-plugging */
-	list_add(&sd->async_list, &subdev_list);
+	list_add_tail(&sd->async_list, &subdev_list);
 
 	mutex_unlock(&list_lock);
 
-- 
2.9.3

[PATCH 06/22] [media] imx: Add i.MX SoC wide media device driver

[Philipp Zabel]

This driver registers a single, SoC wide media device, which all entities
in the media graph can be registered to. It is probed from device tree
using a capture-subsystem node, listing the IPUv3 input ports, similarly
to the already existing display-subsystem node that lists the IPUv3 output
ports.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 .../devicetree/bindings/media/fsl-imx-capture.txt  |  25 +++
 drivers/media/platform/Kconfig                     |   2 +
 drivers/media/platform/Makefile                    |   1 +
 drivers/media/platform/imx/Kconfig                 |   7 +
 drivers/media/platform/imx/Makefile                |   1 +
 drivers/media/platform/imx/imx-media.c             | 249 +++++++++++++++++++++
 6 files changed, 285 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/fsl-imx-capture.txt
 create mode 100644 drivers/media/platform/imx/Kconfig
 create mode 100644 drivers/media/platform/imx/Makefile
 create mode 100644 drivers/media/platform/imx/imx-media.c

diff --git a/Documentation/devicetree/bindings/media/fsl-imx-capture.txt b/Documentation/devicetree/bindings/media/fsl-imx-capture.txt
new file mode 100644
index 0000000..5805331
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/fsl-imx-capture.txt
@@ -0,0 +1,25 @@
+Freescale i.MX IPUv3 capture subsystem
+======================================
+
+The i.MX5/6 IPUv3 capture subsystem consists of one or two IPUs and all
+external subdevices connected to the IPU CSI input ports. On i.MX5 these
+are only external subdevices. On i.MX6, there are additional SoC internal
+multiplexers and a MIPI CSI-2 bridge connected to the CSI input ports via
+of_graph bindings.
+
+On i.MX6 variants with two IPUs, either a single capture subsystem node may be
+defined, containing all CSI ports, or two separate capture subsystem nodes may
+be defined, each containing the CSI ports of a single IPU. In the latter case
+there must be no external of_graph links between the two subsystems.
+
+Required properties:
+- compatible: should be "fsl,imx-capture-subsystem"
+- ports: should contain a list of phandles pointing to the capture interface
+  ports of IPU devices
+
+Example:
+
+capture-subsystem {
+        compatible = "fsl,imx-capture-subsystem";
+        ports = <&ipu1_csi0>, <&ipu1_csi1>, <&ipu2_csi0>, <&ipu2_csi1>;
+};
diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
index ce4a96f..105bf57 100644
--- a/drivers/media/platform/Kconfig
+++ b/drivers/media/platform/Kconfig
@@ -29,6 +29,8 @@ config VIDEO_VIA_CAMERA
 
 source "drivers/media/platform/davinci/Kconfig"
 
+source "drivers/media/platform/imx/Kconfig"
+
 source "drivers/media/platform/omap/Kconfig"
 
 source "drivers/media/platform/blackfin/Kconfig"
diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
index 40b18d1..f7f9008 100644
--- a/drivers/media/platform/Makefile
+++ b/drivers/media/platform/Makefile
@@ -51,6 +51,7 @@ obj-$(CONFIG_VIDEO_RENESAS_FCP) 	+= rcar-fcp.o
 obj-$(CONFIG_VIDEO_RENESAS_JPU) 	+= rcar_jpu.o
 obj-$(CONFIG_VIDEO_RENESAS_VSP1)	+= vsp1/
 
+obj-y	+= imx/
 obj-y	+= omap/
 
 obj-$(CONFIG_VIDEO_AM437X_VPFE)		+= am437x/
diff --git a/drivers/media/platform/imx/Kconfig b/drivers/media/platform/imx/Kconfig
new file mode 100644
index 0000000..3bd699c
--- /dev/null
+++ b/drivers/media/platform/imx/Kconfig
@@ -0,0 +1,7 @@
+config MEDIA_IMX
+	tristate "Multimedia Support for Freescale i.MX"
+	depends on MEDIA_CONTROLLER
+	default y if IMX_IPUV3_CORE
+	---help---
+	  This driver provides a SoC wide media controller device that all
+	  multimedia components in i.MX5 and i.MX6 SoCs can register with.
diff --git a/drivers/media/platform/imx/Makefile b/drivers/media/platform/imx/Makefile
new file mode 100644
index 0000000..74bed76
--- /dev/null
+++ b/drivers/media/platform/imx/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_MEDIA_IMX)			+= imx-media.o
diff --git a/drivers/media/platform/imx/imx-media.c b/drivers/media/platform/imx/imx-media.c
new file mode 100644
index 0000000..1f1fab4
--- /dev/null
+++ b/drivers/media/platform/imx/imx-media.c
@@ -0,0 +1,249 @@
+/*
+ * i.MX V4L2 Capture Driver
+ *
+ * Copyright (C) 2016, Pengutronix, Philipp Zabel <kernel@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/types.h>
+#include <media/media-device.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+
+#define IMX_MEDIA_MAX_PORTS	4
+
+struct imx_media {
+	struct media_device mdev;
+	struct v4l2_device v4l2_dev;
+	struct v4l2_async_notifier subdev_notifier;
+	struct v4l2_async_subdev subdevs[IMX_MEDIA_MAX_PORTS];
+};
+
+static int v4l2_of_create_pad_link(struct v4l2_subdev *sd,
+				   struct v4l2_of_link *link)
+{
+	struct v4l2_subdev *remote_sd, *src_sd, *sink_sd;
+	struct media_link *mlink;
+	int src_port, sink_port;
+
+	if (link->local_port >= sd->entity.num_pads)
+		return -EINVAL;
+
+	remote_sd = v4l2_find_subdev_by_node(sd->v4l2_dev, link->remote_node);
+	if (!remote_sd)
+		return 0;
+
+	if (sd->entity.pads[link->local_port].flags & MEDIA_PAD_FL_SINK) {
+		src_sd = remote_sd;
+		src_port = link->remote_port;
+		sink_sd = sd;
+		sink_port = link->local_port;
+	} else {
+		src_sd = sd;
+		src_port = link->local_port;
+		sink_sd = remote_sd;
+		sink_port = link->remote_port;
+	}
+
+	mlink = media_entity_find_link(&src_sd->entity.pads[src_port],
+				       &sink_sd->entity.pads[sink_port]);
+	if (mlink)
+		return 0;
+
+	return media_create_pad_link(&src_sd->entity, src_port,
+			&sink_sd->entity, sink_port, 0);
+}
+
+static int imx_media_complete(struct v4l2_async_notifier *notifier)
+{
+	struct imx_media *im = container_of(notifier, struct imx_media,
+					    subdev_notifier);
+	struct media_device *mdev = &im->mdev;
+	struct v4l2_subdev *sd;
+	struct device_node *ep;
+
+	mutex_lock(&mdev->graph_mutex);
+
+	/*
+	 * Link all subdevices
+	 */
+	list_for_each_entry(sd, &notifier->done, async_list) {
+		struct v4l2_of_link link;
+		int ret;
+
+		/*
+		 * The IPU port nodes 0 and 1 correspond to the CSI subdevices.
+		 * We can't use for_each_endpoint_of_node here, which would
+		 * iterate over all endpoints of the IPU, including output ones.
+		 */
+		if (strcmp(sd->of_node->name, "port") == 0) {
+			/* There should be one endpoint in the CSI port */
+			ep = of_get_next_child(sd->of_node, NULL);
+			if (!ep)
+				continue;
+
+			ret = v4l2_of_parse_link(ep, &link);
+			if (ret) {
+				of_node_put(ep);
+				continue;
+			}
+
+			/*
+			 * The IPU port id in the device tree does not
+			 * correspond to the CSI pad id. Always connect
+			 * the source to the CSI input pad (pad 0).
+			 */
+			link.local_port = 0;
+
+			v4l2_of_create_pad_link(sd, &link);
+
+			v4l2_of_put_link(&link);
+
+			continue;
+		}
+
+		/*
+		 * TODO: for now assume that all subdevices other than CSI have
+		 * a 1:1 relationship between device_node and v4l2_subdev and
+		 * between of_graph port number and media_entity pad index.
+		 */
+		for_each_endpoint_of_node(sd->of_node, ep) {
+			ret = v4l2_of_parse_link(ep, &link);
+			if (ret)
+				continue;
+
+			v4l2_of_create_pad_link(sd, &link);
+
+			v4l2_of_put_link(&link);
+		}
+	}
+
+	mutex_unlock(&mdev->graph_mutex);
+
+	return v4l2_device_register_subdev_nodes(&im->v4l2_dev);
+}
+
+static void imx_media_register_notifier(struct imx_media *im)
+{
+	struct v4l2_async_notifier *notifier = &im->subdev_notifier;
+	struct v4l2_async_subdev *asd = &im->subdevs[0];
+	struct device_node *port;
+	int i;
+
+	/*
+	 * ports correspond to the CSI subdevices that terminate the media
+	 * pipelines.
+	 */
+	for (i = 0; i < IMX_MEDIA_MAX_PORTS; i++) {
+		port = of_parse_phandle(im->mdev.dev->of_node, "ports", i);
+		if (!port)
+			break;
+
+		if (!of_device_is_available(port->parent)) {
+			of_node_put(port);
+			continue;
+		}
+
+		asd[i].match_type = V4L2_ASYNC_MATCH_OF;
+		asd[i].match.of.node = port;
+		of_node_put(port);
+	}
+	notifier->num_subdevs = i;
+	notifier->subdevs = devm_kcalloc(im->mdev.dev, i,
+					 sizeof(*notifier->subdevs),
+					 GFP_KERNEL);
+	for (i = 0; i < notifier->num_subdevs; i++)
+		notifier->subdevs[i] = &im->subdevs[i];
+
+	notifier->complete = imx_media_complete;
+
+	v4l2_async_notifier_register(&im->v4l2_dev, notifier);
+}
+
+static int imx_media_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct media_device *mdev;
+	struct imx_media *im;
+	int ret;
+
+	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(32));
+	if (ret)
+		return ret;
+
+	im = devm_kzalloc(dev, sizeof(*im), GFP_KERNEL);
+	if (!im)
+		return -ENOMEM;
+
+	mdev = &im->mdev;
+	mdev->dev = dev;
+	strlcpy(mdev->model, "i.MX IPUv3", sizeof(mdev->model));
+	media_device_init(mdev);
+
+	im->v4l2_dev.mdev = mdev;
+	ret = v4l2_device_register(dev, &im->v4l2_dev);
+	if (ret) {
+		dev_err(dev, "Failed to register v4l2 device: %d\n", ret);
+		return ret;
+	}
+
+	imx_media_register_notifier(im);
+
+	ret = media_device_register(mdev);
+	if (ret) {
+		dev_err(dev, "Failed to register media controller device: %d\n",
+			ret);
+		goto err;
+	}
+
+	platform_set_drvdata(pdev, mdev);
+
+	return 0;
+
+err:
+	media_device_cleanup(mdev);
+	return ret;
+}
+
+static int imx_media_remove(struct platform_device *pdev)
+{
+	struct media_device *mdev = platform_get_drvdata(pdev);
+	struct imx_media *im = container_of(mdev, struct imx_media, mdev);
+
+	media_device_unregister(mdev);
+	v4l2_device_unregister(&im->v4l2_dev);
+	media_device_cleanup(mdev);
+
+	return 0;
+}
+
+static const struct of_device_id imx_media_dt_ids[] = {
+	{ .compatible = "fsl,imx-capture-subsystem", },
+	{ },
+};
+MODULE_DEVICE_TABLE(of, imx_media_dt_ids);
+
+static struct platform_driver imx_media_driver = {
+	.probe = imx_media_probe,
+	.remove = imx_media_remove,
+	.driver = {
+		.name = "imx-media",
+		.of_match_table = imx_media_dt_ids,
+	},
+};
+module_platform_driver(imx_media_driver);
+
+MODULE_AUTHOR("Sascha Hauer <s.hauer@pengutronix.de>");
+MODULE_AUTHOR("Philipp Zabel <p.zabel@pengutronix.de>");
+MODULE_DESCRIPTION("i.MX SoC wide media device");
+MODULE_LICENSE("GPL");
-- 
2.9.3

[PATCH 07/22] [media] imx: Add IPUv3 media common code

[Philipp Zabel]

From: Sascha Hauer <s.hauer@pengutronix.de>

Add video4linux API routines common to drivers for units that
accept or provide video data via the i.MX IPU IDMAC channels,
such as capture, mem2mem scaler or deinterlacer drivers.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/media/platform/imx/Kconfig   |   3 +
 drivers/media/platform/imx/Makefile  |   1 +
 drivers/media/platform/imx/imx-ipu.c | 321 +++++++++++++++++++++++++++++++++++
 drivers/media/platform/imx/imx-ipu.h |  34 ++++
 4 files changed, 359 insertions(+)
 create mode 100644 drivers/media/platform/imx/imx-ipu.c
 create mode 100644 drivers/media/platform/imx/imx-ipu.h

diff --git a/drivers/media/platform/imx/Kconfig b/drivers/media/platform/imx/Kconfig
index 3bd699c..1662bb0b 100644
--- a/drivers/media/platform/imx/Kconfig
+++ b/drivers/media/platform/imx/Kconfig
@@ -5,3 +5,6 @@ config MEDIA_IMX
 	---help---
 	  This driver provides a SoC wide media controller device that all
 	  multimedia components in i.MX5 and i.MX6 SoCs can register with.
+
+config VIDEO_IMX_IPU_COMMON
+	tristate
diff --git a/drivers/media/platform/imx/Makefile b/drivers/media/platform/imx/Makefile
index 74bed76..0ba601a 100644
--- a/drivers/media/platform/imx/Makefile
+++ b/drivers/media/platform/imx/Makefile
@@ -1 +1,2 @@
 obj-$(CONFIG_MEDIA_IMX)			+= imx-media.o
+obj-$(CONFIG_VIDEO_IMX_IPU_COMMON)	+= imx-ipu.o
diff --git a/drivers/media/platform/imx/imx-ipu.c b/drivers/media/platform/imx/imx-ipu.c
new file mode 100644
index 0000000..da1deb0
--- /dev/null
+++ b/drivers/media/platform/imx/imx-ipu.c
@@ -0,0 +1,321 @@
+/*
+ * i.MX IPUv3 common v4l2 support
+ *
+ * Copyright (C) 2011 Pengutronix, Sascha Hauer <kernel@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+#include <linux/module.h>
+#include <media/v4l2-common.h>
+#include <media/v4l2-dev.h>
+#include <media/v4l2-ioctl.h>
+
+#include "imx-ipu.h"
+
+/*
+ * These formats are in order of preference: interleaved YUV first,
+ * because those are the most bandwidth efficient, followed by
+ * chroma-interleaved formats, and planar formats last.
+ * In each category, YUV 4:2:0 may be preferrable to 4:2:2 for bandwidth
+ * reasons, if the IDMAC channel supports double read/write reduction
+ * (all write channels, VDIC read channels).
+ */
+static struct ipu_fmt ipu_fmt_yuv[] = {
+	{
+		.fourcc = V4L2_PIX_FMT_YUYV,
+		.bytes_per_pixel = 2,
+	}, {
+		.fourcc = V4L2_PIX_FMT_UYVY,
+		.bytes_per_pixel = 2,
+	}, {
+		.fourcc = V4L2_PIX_FMT_NV12,
+		.bytes_per_pixel = 1,
+	}, {
+		.fourcc = V4L2_PIX_FMT_NV16,
+		.bytes_per_pixel = 1,
+	}, {
+		.fourcc = V4L2_PIX_FMT_YUV420,
+		.bytes_per_pixel = 1,
+	}, {
+		.fourcc = V4L2_PIX_FMT_YVU420,
+		.bytes_per_pixel = 1,
+	}, {
+		.fourcc = V4L2_PIX_FMT_YUV422P,
+		.bytes_per_pixel = 1,
+	},
+};
+
+static struct ipu_fmt ipu_fmt_rgb[] = {
+	{
+		.fourcc = V4L2_PIX_FMT_RGB32,
+		.bytes_per_pixel = 4,
+	}, {
+		.fourcc = V4L2_PIX_FMT_RGB24,
+		.bytes_per_pixel = 3,
+	}, {
+		.fourcc = V4L2_PIX_FMT_BGR24,
+		.bytes_per_pixel = 3,
+	}, {
+		.fourcc = V4L2_PIX_FMT_RGB565,
+		.bytes_per_pixel = 2,
+	},
+	{
+		.fourcc = V4L2_PIX_FMT_BGR32,
+		.bytes_per_pixel = 4,
+	},
+};
+
+struct ipu_fmt *ipu_find_fmt_yuv(unsigned int pixelformat)
+{
+	struct ipu_fmt *fmt;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(ipu_fmt_yuv); i++) {
+		fmt = &ipu_fmt_yuv[i];
+		if (fmt->fourcc == pixelformat)
+			return fmt;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(ipu_find_fmt_yuv);
+
+struct ipu_fmt *ipu_find_fmt_rgb(unsigned int pixelformat)
+{
+	struct ipu_fmt *fmt;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(ipu_fmt_rgb); i++) {
+		fmt = &ipu_fmt_rgb[i];
+		if (fmt->fourcc == pixelformat)
+			return fmt;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(ipu_find_fmt_rgb);
+
+static struct ipu_fmt *ipu_find_fmt(unsigned long pixelformat)
+{
+	struct ipu_fmt *fmt;
+
+	fmt = ipu_find_fmt_yuv(pixelformat);
+	if (fmt)
+		return fmt;
+	fmt = ipu_find_fmt_rgb(pixelformat);
+
+	return fmt;
+}
+EXPORT_SYMBOL_GPL(ipu_find_fmt);
+
+int ipu_try_fmt(struct file *file, void *fh,
+		struct v4l2_format *f)
+{
+	struct ipu_fmt *fmt;
+
+	v4l_bound_align_image(&f->fmt.pix.width, 8, 4096, 2,
+			      &f->fmt.pix.height, 2, 4096, 1, 0);
+
+	f->fmt.pix.field = V4L2_FIELD_NONE;
+
+	fmt = ipu_find_fmt(f->fmt.pix.pixelformat);
+	if (!fmt)
+		return -EINVAL;
+
+	f->fmt.pix.bytesperline = f->fmt.pix.width * fmt->bytes_per_pixel;
+	f->fmt.pix.sizeimage = f->fmt.pix.bytesperline * f->fmt.pix.height;
+	if (fmt->fourcc == V4L2_PIX_FMT_YUV420 ||
+	    fmt->fourcc == V4L2_PIX_FMT_YVU420 ||
+	    fmt->fourcc == V4L2_PIX_FMT_NV12)
+		f->fmt.pix.sizeimage = f->fmt.pix.sizeimage * 3 / 2;
+	else if (fmt->fourcc == V4L2_PIX_FMT_YUV422P ||
+		 fmt->fourcc == V4L2_PIX_FMT_NV16)
+		f->fmt.pix.sizeimage *= 2;
+
+	switch (f->fmt.pix.pixelformat) {
+	case V4L2_PIX_FMT_YUV420:
+	case V4L2_PIX_FMT_YVU420:
+	case V4L2_PIX_FMT_YUV422P:
+	case V4L2_PIX_FMT_NV12:
+	case V4L2_PIX_FMT_NV16:
+	case V4L2_PIX_FMT_UYVY:
+	case V4L2_PIX_FMT_YUYV:
+		if (f->fmt.pix.width <= 720 && f->fmt.pix.height <= 576)
+			f->fmt.pix.colorspace = V4L2_COLORSPACE_SMPTE170M;
+		else
+			f->fmt.pix.colorspace = V4L2_COLORSPACE_REC709;
+		break;
+	case V4L2_PIX_FMT_RGB32:
+	case V4L2_PIX_FMT_RGB24:
+	case V4L2_PIX_FMT_BGR24:
+	case V4L2_PIX_FMT_RGB565:
+	case V4L2_PIX_FMT_BGR32:
+		f->fmt.pix.colorspace = V4L2_COLORSPACE_SRGB;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(ipu_try_fmt);
+
+int ipu_try_fmt_rgb(struct file *file, void *fh,
+		struct v4l2_format *f)
+{
+	struct ipu_fmt *fmt;
+
+	fmt = ipu_find_fmt_rgb(f->fmt.pix.pixelformat);
+	if (!fmt)
+		return -EINVAL;
+
+	return ipu_try_fmt(file, fh, f);
+}
+EXPORT_SYMBOL_GPL(ipu_try_fmt_rgb);
+
+int ipu_try_fmt_yuv(struct file *file, void *fh,
+		struct v4l2_format *f)
+{
+	struct ipu_fmt *fmt;
+
+	fmt = ipu_find_fmt_yuv(f->fmt.pix.pixelformat);
+	if (!fmt)
+		return -EINVAL;
+
+	return ipu_try_fmt(file, fh, f);
+}
+EXPORT_SYMBOL_GPL(ipu_try_fmt_yuv);
+
+int ipu_enum_fmt_rgb(struct file *file, void *fh,
+		struct v4l2_fmtdesc *f)
+{
+	struct ipu_fmt *fmt;
+
+	if (f->index >= ARRAY_SIZE(ipu_fmt_rgb))
+		return -EINVAL;
+
+	fmt = &ipu_fmt_rgb[f->index];
+
+	f->pixelformat = fmt->fourcc;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(ipu_enum_fmt_rgb);
+
+int ipu_enum_fmt_yuv(struct file *file, void *fh,
+		struct v4l2_fmtdesc *f)
+{
+	struct ipu_fmt *fmt;
+
+	if (f->index >= ARRAY_SIZE(ipu_fmt_yuv))
+		return -EINVAL;
+
+	fmt = &ipu_fmt_yuv[f->index];
+
+	f->pixelformat = fmt->fourcc;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(ipu_enum_fmt_yuv);
+
+int ipu_enum_fmt(struct file *file, void *fh,
+		struct v4l2_fmtdesc *f)
+{
+	struct ipu_fmt *fmt;
+	int index = f->index;
+
+	if (index >= ARRAY_SIZE(ipu_fmt_yuv)) {
+		index -= ARRAY_SIZE(ipu_fmt_yuv);
+		if (index >= ARRAY_SIZE(ipu_fmt_rgb))
+			return -EINVAL;
+		fmt = &ipu_fmt_rgb[index];
+	} else {
+		fmt = &ipu_fmt_yuv[index];
+	}
+
+	f->pixelformat = fmt->fourcc;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(ipu_enum_fmt);
+
+int ipu_s_fmt(struct file *file, void *fh,
+		struct v4l2_format *f, struct v4l2_pix_format *pix)
+{
+	int ret;
+
+	ret = ipu_try_fmt(file, fh, f);
+	if (ret)
+		return ret;
+
+	*pix = f->fmt.pix;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(ipu_s_fmt);
+
+int ipu_s_fmt_rgb(struct file *file, void *fh,
+		struct v4l2_format *f, struct v4l2_pix_format *pix)
+{
+	struct ipu_fmt *fmt;
+
+	fmt = ipu_find_fmt_rgb(f->fmt.pix.pixelformat);
+	if (!fmt)
+		return -EINVAL;
+
+	return ipu_s_fmt(file, fh, f, pix);
+}
+EXPORT_SYMBOL_GPL(ipu_s_fmt_rgb);
+
+int ipu_s_fmt_yuv(struct file *file, void *fh,
+		struct v4l2_format *f, struct v4l2_pix_format *pix)
+{
+	struct ipu_fmt *fmt;
+
+	fmt = ipu_find_fmt_yuv(f->fmt.pix.pixelformat);
+	if (!fmt)
+		return -EINVAL;
+
+	return ipu_s_fmt(file, fh, f, pix);
+}
+EXPORT_SYMBOL_GPL(ipu_s_fmt_yuv);
+
+int ipu_g_fmt(struct v4l2_format *f, struct v4l2_pix_format *pix)
+{
+	f->fmt.pix = *pix;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(ipu_g_fmt);
+
+int ipu_enum_framesizes(struct file *file, void *fh,
+			struct v4l2_frmsizeenum *fsize)
+{
+	struct ipu_fmt *fmt;
+
+	if (fsize->index != 0)
+		return -EINVAL;
+
+	fmt = ipu_find_fmt(fsize->pixel_format);
+	if (!fmt)
+		return -EINVAL;
+
+	fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS;
+	fsize->stepwise.min_width = 1;
+	fsize->stepwise.min_height = 1;
+	fsize->stepwise.max_width = 4096;
+	fsize->stepwise.max_height = 4096;
+	fsize->stepwise.step_width = fsize->stepwise.step_height = 1;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(ipu_enum_framesizes);
+
+MODULE_LICENSE("GPL");
diff --git a/drivers/media/platform/imx/imx-ipu.h b/drivers/media/platform/imx/imx-ipu.h
new file mode 100644
index 0000000..7b344b6
--- /dev/null
+++ b/drivers/media/platform/imx/imx-ipu.h
@@ -0,0 +1,34 @@
+#ifndef __MEDIA_IMX_IPU_H
+#define __MEDIA_IMX_IPU_H
+#include <linux/videodev2.h>
+
+struct ipu_fmt {
+	u32 fourcc;
+	int bytes_per_pixel;
+};
+
+int ipu_enum_fmt(struct file *file, void *fh,
+		struct v4l2_fmtdesc *f);
+int ipu_enum_fmt_rgb(struct file *file, void *fh,
+		struct v4l2_fmtdesc *f);
+int ipu_enum_fmt_yuv(struct file *file, void *fh,
+		struct v4l2_fmtdesc *f);
+struct ipu_fmt *ipu_find_fmt_rgb(unsigned int pixelformat);
+struct ipu_fmt *ipu_find_fmt_yuv(unsigned int pixelformat);
+int ipu_try_fmt(struct file *file, void *fh,
+		struct v4l2_format *f);
+int ipu_try_fmt_rgb(struct file *file, void *fh,
+		struct v4l2_format *f);
+int ipu_try_fmt_yuv(struct file *file, void *fh,
+		struct v4l2_format *f);
+int ipu_s_fmt(struct file *file, void *fh,
+		struct v4l2_format *f, struct v4l2_pix_format *pix);
+int ipu_s_fmt_rgb(struct file *file, void *fh,
+		struct v4l2_format *f, struct v4l2_pix_format *pix);
+int ipu_s_fmt_yuv(struct file *file, void *fh,
+		struct v4l2_format *f, struct v4l2_pix_format *pix);
+int ipu_g_fmt(struct v4l2_format *f, struct v4l2_pix_format *pix);
+int ipu_enum_framesizes(struct file *file, void *fh,
+			struct v4l2_frmsizeenum *fsize);
+
+#endif /* __MEDIA_IMX_IPU_H */
-- 
2.9.3

[PATCH 08/22] [media] imx-ipu: Add i.MX IPUv3 CSI subdevice driver

[Philipp Zabel]

This adds a V4L2 subdevice driver for the two CMOS Sensor Interface (CSI)
modules contained in each IPUv3. These sample video data from the
parallel CSI0/1 pads or from the the MIPI CSI-2 bridge via IOMUXC video
bus multiplexers and write to IPU internal FIFOs to deliver data to
either the IDMAC or IC modules.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
Signed-off-by: Marc Kleine-Budde <mkl@pengutronix.de>
Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/media/platform/imx/Kconfig         |   7 +
 drivers/media/platform/imx/Makefile        |   1 +
 drivers/media/platform/imx/imx-ipuv3-csi.c | 522 +++++++++++++++++++++++++++++
 3 files changed, 530 insertions(+)
 create mode 100644 drivers/media/platform/imx/imx-ipuv3-csi.c

diff --git a/drivers/media/platform/imx/Kconfig b/drivers/media/platform/imx/Kconfig
index 1662bb0b..a88c4f7 100644
--- a/drivers/media/platform/imx/Kconfig
+++ b/drivers/media/platform/imx/Kconfig
@@ -8,3 +8,10 @@ config MEDIA_IMX
 
 config VIDEO_IMX_IPU_COMMON
 	tristate
+
+config VIDEO_IMX_IPU_CSI
+	tristate "i.MX5/6 CMOS Sensor Interface driver"
+	depends on VIDEO_DEV && IMX_IPUV3_CORE && MEDIA_IMX
+	select VIDEO_IMX_IPUV3
+	---help---
+	  This is a v4l2 subdevice driver for two CSI modules in each IPUv3.
diff --git a/drivers/media/platform/imx/Makefile b/drivers/media/platform/imx/Makefile
index 0ba601a..82a3616 100644
--- a/drivers/media/platform/imx/Makefile
+++ b/drivers/media/platform/imx/Makefile
@@ -1,2 +1,3 @@
 obj-$(CONFIG_MEDIA_IMX)			+= imx-media.o
 obj-$(CONFIG_VIDEO_IMX_IPU_COMMON)	+= imx-ipu.o
+obj-$(CONFIG_VIDEO_IMX_IPU_CSI)		+= imx-ipuv3-csi.o
diff --git a/drivers/media/platform/imx/imx-ipuv3-csi.c b/drivers/media/platform/imx/imx-ipuv3-csi.c
new file mode 100644
index 0000000..316175b
--- /dev/null
+++ b/drivers/media/platform/imx/imx-ipuv3-csi.c
@@ -0,0 +1,522 @@
+/*
+ * i.MX IPUv3 CSI V4L2 Subdevice Driver
+ *
+ * Copyright (C) 2016, Pengutronix, Philipp Zabel <kernel@pengutronix.de>
+ *
+ * Based on code
+ * Copyright (C) 2006, Sascha Hauer, Pengutronix
+ * Copyright (C) 2008, Guennadi Liakhovetski <kernel@pengutronix.de>
+ * Copyright (C) 2008, Paulius Zaleckas <paulius.zaleckas@teltonika.lt>
+ * Copyright (C) 2009, Darius Augulis <augulis.darius@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+#include <linux/clk.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/errno.h>
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/mm.h>
+#include <linux/module.h>
+#include <linux/moduleparam.h>
+#include <linux/mutex.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/time.h>
+#include <linux/version.h>
+#include <linux/videodev2.h>
+
+#include <media/v4l2-common.h>
+#include <media/v4l2-dev.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-mc.h>
+#include <media/v4l2-of.h>
+#include <media/videobuf2-dma-contig.h>
+
+#include <video/imx-ipu-v3.h>
+
+#include "imx-ipu.h"
+
+#define DRIVER_NAME "imx-ipuv3-csi"
+
+struct ipucsi {
+	struct v4l2_subdev		subdev;
+
+	struct device			*dev;
+	u32				id;
+
+	struct ipu_csi			*csi;
+	struct ipu_soc			*ipu;
+	struct v4l2_of_endpoint		endpoint;
+	enum ipu_csi_dest		csi_dest;
+
+	struct media_pad		subdev_pad[2];
+	struct v4l2_mbus_framefmt	format_mbus[2];
+	struct v4l2_fract		timeperframe[2];
+};
+
+static int ipu_csi_get_mbus_config(struct ipucsi *ipucsi,
+				   struct v4l2_mbus_config *config)
+{
+	struct v4l2_subdev *sd;
+	struct media_pad *pad;
+	int ret;
+
+	/*
+	 * Retrieve media bus configuration from the entity connected directly
+	 * to the CSI subdev sink pad.
+	 */
+	pad = media_entity_remote_pad(&ipucsi->subdev_pad[0]);
+	if (!pad) {
+		dev_err(ipucsi->dev,
+			"failed to retrieve mbus config from source entity\n");
+		return -ENODEV;
+	}
+	sd = media_entity_to_v4l2_subdev(pad->entity);
+	ret = v4l2_subdev_call(sd, video, g_mbus_config, config);
+	if (ret == -ENOIOCTLCMD) {
+		/* Fall back to static mbus configuration from device tree */
+		config->type = ipucsi->endpoint.bus_type;
+		config->flags = ipucsi->endpoint.bus.parallel.flags;
+		ret = 0;
+	}
+
+	return ret;
+}
+
+static struct v4l2_mbus_framefmt *
+__ipucsi_get_pad_format(struct v4l2_subdev *sd,
+			struct v4l2_subdev_pad_config *cfg,
+			unsigned int pad, u32 which)
+{
+	struct ipucsi *ipucsi = container_of(sd, struct ipucsi, subdev);
+
+	switch (which) {
+	case V4L2_SUBDEV_FORMAT_TRY:
+		return v4l2_subdev_get_try_format(sd, cfg, pad);
+	case V4L2_SUBDEV_FORMAT_ACTIVE:
+		return &ipucsi->format_mbus[pad ? 1 : 0];
+	default:
+		return NULL;
+	}
+}
+
+static int ipucsi_subdev_log_status(struct v4l2_subdev *subdev)
+{
+	struct ipucsi *ipucsi = container_of(subdev, struct ipucsi, subdev);
+
+	ipu_csi_dump(ipucsi->csi);
+
+	return 0;
+}
+
+static int ipucsi_subdev_get_format(struct v4l2_subdev *sd,
+		struct v4l2_subdev_pad_config *cfg,
+		struct v4l2_subdev_format *sdformat)
+{
+	sdformat->format = *__ipucsi_get_pad_format(sd, cfg, sdformat->pad,
+						    sdformat->which);
+	return 0;
+}
+
+static bool ipucsi_mbus_code_supported(const u32 mbus_code)
+{
+	static const u32 mbus_codes[] = {
+		MEDIA_BUS_FMT_Y8_1X8,
+		MEDIA_BUS_FMT_Y10_1X10,
+		MEDIA_BUS_FMT_Y12_1X12,
+		MEDIA_BUS_FMT_UYVY8_2X8,
+		MEDIA_BUS_FMT_YUYV8_2X8,
+		MEDIA_BUS_FMT_UYVY8_1X16,
+		MEDIA_BUS_FMT_YUYV8_1X16,
+	};
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(mbus_codes); i++) {
+		if (mbus_code == mbus_codes[i])
+			return true;
+	}
+
+	return false;
+}
+
+static int ipucsi_subdev_set_format(struct v4l2_subdev *sd,
+		struct v4l2_subdev_pad_config *cfg,
+		struct v4l2_subdev_format *sdformat)
+{
+	struct ipucsi *ipucsi = container_of(sd, struct ipucsi, subdev);
+	struct v4l2_mbus_framefmt *mbusformat;
+	unsigned int width, height;
+	bool supported;
+
+	supported = ipucsi_mbus_code_supported(sdformat->format.code);
+	if (!supported)
+		return -EINVAL;
+
+	if (sdformat->pad == 0) {
+		width = clamp_t(unsigned int, sdformat->format.width, 16, 8192);
+		height = clamp_t(unsigned int, sdformat->format.height, 16, 4096);
+	} else {
+		width = ipucsi->format_mbus[0].width;
+		height = ipucsi->format_mbus[0].height;
+	}
+
+	mbusformat = __ipucsi_get_pad_format(sd, cfg, sdformat->pad,
+					     sdformat->which);
+	mbusformat->width = width;
+	mbusformat->height = height;
+	mbusformat->code = sdformat->format.code;
+	mbusformat->field = sdformat->format.field;
+
+	if (mbusformat->field == V4L2_FIELD_SEQ_TB &&
+	    mbusformat->width == 720 && mbusformat->height == 480 &&
+	    ipucsi->endpoint.bus_type == V4L2_MBUS_BT656) {
+		/* We capture NTSC bottom field first */
+		mbusformat->field = V4L2_FIELD_SEQ_BT;
+	} else if (mbusformat->field == V4L2_FIELD_ANY) {
+		mbusformat->field = ipucsi->format_mbus[!sdformat->pad].field;
+	}
+
+	sdformat->format = *mbusformat;
+
+	return 0;
+}
+
+static int ipucsi_subdev_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct ipucsi *ipucsi = container_of(sd, struct ipucsi, subdev);
+
+	if (enable) {
+		struct v4l2_mbus_framefmt *fmt = ipucsi->format_mbus;
+		struct v4l2_mbus_config mbus_config;
+		struct v4l2_rect window;
+		bool mux_mct;
+		int ret;
+
+		ret = ipu_csi_get_mbus_config(ipucsi, &mbus_config);
+		if (ret)
+			return ret;
+
+		window.left = 0;
+		window.top = 0;
+		window.width = fmt[0].width;
+		window.height = fmt[0].height;
+		ipu_csi_set_window(ipucsi->csi, &window);
+
+		/* Is CSI data source MCT (MIPI)? */
+		mux_mct = (mbus_config.type == V4L2_MBUS_CSI2);
+
+		ipu_set_csi_src_mux(ipucsi->ipu, ipucsi->id, mux_mct);
+		if (mux_mct)
+			ipu_csi_set_mipi_datatype(ipucsi->csi, /*VC*/ 0,
+						  &fmt[0]);
+
+		ret = ipu_csi_init_interface(ipucsi->csi, &mbus_config,
+					     &fmt[0]);
+		if (ret) {
+			dev_err(ipucsi->dev, "Failed to initialize CSI\n");
+			return ret;
+		}
+
+		ipu_csi_set_dest(ipucsi->csi, ipucsi->csi_dest);
+
+		ipu_csi_enable(ipucsi->csi);
+	} else {
+		ipu_csi_disable(ipucsi->csi);
+	}
+
+	return 0;
+}
+
+static int ipucsi_subdev_g_frame_interval(struct v4l2_subdev *subdev,
+					  struct v4l2_subdev_frame_interval *fi)
+{
+	struct ipucsi *ipucsi = container_of(subdev, struct ipucsi, subdev);
+
+	if (fi->pad > 4)
+		return -EINVAL;
+
+	fi->interval = ipucsi->timeperframe[(fi->pad == 0) ? 0 : 1];
+
+	return 0;
+}
+
+/*
+ * struct ipucsi_skip_desc - CSI frame skipping descriptor
+ * @keep - number of frames kept per max_ratio frames
+ * @max_ratio - width of skip_smfc, written to MAX_RATIO bitfield
+ * @skip_smfc - skip pattern written to the SKIP_SMFC bitfield
+ */
+struct ipucsi_skip_desc {
+	u8 keep;
+	u8 max_ratio;
+	u8 skip_smfc;
+};
+
+static const struct ipucsi_skip_desc ipucsi_skip[12] = {
+	{ 1, 1, 0x00 }, /* Keep all frames */
+	{ 5, 6, 0x10 }, /* Skip every sixth frame */
+	{ 4, 5, 0x08 }, /* Skip every fifth frame */
+	{ 3, 4, 0x04 }, /* Skip every fourth frame */
+	{ 2, 3, 0x02 }, /* Skip every third frame */
+	{ 3, 5, 0x0a }, /* Skip frames 1 and 3 of every 5 */
+	{ 1, 2, 0x01 }, /* Skip every second frame */
+	{ 2, 5, 0x0b }, /* Keep frames 1 and 4 of every 5 */
+	{ 1, 3, 0x03 }, /* Keep one in three frames */
+	{ 1, 4, 0x07 }, /* Keep one in four frames */
+	{ 1, 5, 0x0f }, /* Keep one in five frames */
+	{ 1, 6, 0x1f }, /* Keep one in six frames */
+};
+
+static int ipucsi_subdev_s_frame_interval(struct v4l2_subdev *subdev,
+					  struct v4l2_subdev_frame_interval *fi)
+{
+	struct ipucsi *ipucsi = container_of(subdev, struct ipucsi, subdev);
+	const struct ipucsi_skip_desc *skip;
+	struct v4l2_fract *want, *in;
+	u32 min_err = UINT_MAX;
+	int i, best_i = 0;
+	u64 want_us;
+
+	if (fi->pad > 4)
+		return -EINVAL;
+
+	if (fi->pad == 0) {
+		/* Believe what we are told about the input frame rate */
+		ipucsi->timeperframe[0] = fi->interval;
+		/* Reset output frame rate to input frame rate */
+		ipucsi->timeperframe[1] = fi->interval;
+		return 0;
+	}
+
+	want = &fi->interval;
+	in = &ipucsi->timeperframe[0];
+
+	if (want->numerator == 0 || want->denominator == 0 ||
+	    in->denominator == 0) {
+		ipucsi->timeperframe[1] = ipucsi->timeperframe[0];
+		fi->interval = ipucsi->timeperframe[1];
+		return 0;
+	}
+
+	want_us = 1000000ULL * want->numerator;
+	do_div(want_us, want->denominator);
+
+	/* Find the reduction closest to the requested timeperframe */
+	for (i = 0; i < ARRAY_SIZE(ipucsi_skip); i++) {
+		u64 tmp;
+		u32 err;
+
+		skip = &ipucsi_skip[i];
+		tmp = 1000000ULL * in->numerator * skip->max_ratio;
+		do_div(tmp, in->denominator * skip->keep);
+		err = (tmp > want_us) ? (tmp - want_us) : (want_us - tmp);
+		if (err < min_err) {
+			min_err = err;
+			best_i = i;
+		}
+	}
+
+	skip = &ipucsi_skip[best_i];
+
+	ipu_csi_set_skip_smfc(ipucsi->csi, skip->skip_smfc, skip->max_ratio - 1,
+			      0);
+
+	fi->interval.numerator = in->numerator * skip->max_ratio;
+	fi->interval.denominator = in->denominator * skip->keep;
+	ipucsi->timeperframe[1] = fi->interval;
+
+	dev_dbg(ipucsi->dev, "skip: %d/%d -> %d/%d frames:%d pattern:0x%x\n",
+	       in->numerator, in->denominator,
+	       fi->interval.numerator, fi->interval.denominator,
+	       skip->max_ratio, skip->skip_smfc);
+
+	return 0;
+}
+
+static struct v4l2_subdev_core_ops ipucsi_subdev_core_ops = {
+	.log_status = ipucsi_subdev_log_status,
+};
+
+static struct v4l2_subdev_pad_ops ipucsi_subdev_pad_ops = {
+	.get_fmt = ipucsi_subdev_get_format,
+	.set_fmt = ipucsi_subdev_set_format,
+};
+
+static struct v4l2_subdev_video_ops ipucsi_subdev_video_ops = {
+	.s_stream = ipucsi_subdev_s_stream,
+	.g_frame_interval = ipucsi_subdev_g_frame_interval,
+	.s_frame_interval = ipucsi_subdev_s_frame_interval,
+};
+
+static const struct v4l2_subdev_ops ipucsi_subdev_ops = {
+	.core   = &ipucsi_subdev_core_ops,
+	.pad    = &ipucsi_subdev_pad_ops,
+	.video  = &ipucsi_subdev_video_ops,
+};
+
+static int ipu_csi_link_setup(struct media_entity *entity,
+			      const struct media_pad *local,
+			      const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct ipucsi *ipucsi = container_of(sd, struct ipucsi, subdev);
+
+	ipucsi->csi_dest = IPU_CSI_DEST_IDMAC;
+
+	return 0;
+}
+
+struct media_entity_operations ipucsi_entity_ops = {
+	.link_setup = ipu_csi_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static int ipu_csi_registered(struct v4l2_subdev *sd)
+{
+	struct ipucsi *ipucsi = container_of(sd, struct ipucsi, subdev);
+	struct device_node *rpp;
+
+	/*
+	 * Add source subdevice to asynchronous subdevice waiting list.
+	 */
+	rpp = of_graph_get_remote_port_parent(ipucsi->endpoint.base.local_node);
+	if (rpp) {
+		struct v4l2_async_subdev *asd;
+
+		asd = devm_kzalloc(sd->dev, sizeof(*asd), GFP_KERNEL);
+		if (!asd) {
+			of_node_put(rpp);
+			return -ENOMEM;
+		}
+
+		asd->match_type = V4L2_ASYNC_MATCH_OF;
+		asd->match.of.node = rpp;
+
+		__v4l2_async_notifier_add_subdev(sd->notifier, asd);
+	}
+
+	return 0;
+}
+
+struct v4l2_subdev_internal_ops ipu_csi_internal_ops = {
+	.registered = ipu_csi_registered,
+};
+
+static int ipucsi_subdev_init(struct ipucsi *ipucsi, struct device_node *node)
+{
+	struct device_node *endpoint;
+	int ret;
+
+	v4l2_subdev_init(&ipucsi->subdev, &ipucsi_subdev_ops);
+	ipucsi->subdev.dev = ipucsi->dev;
+
+	snprintf(ipucsi->subdev.name, sizeof(ipucsi->subdev.name), "%s%d %s%d",
+		 "IPU", ipu_get_num(ipucsi->ipu), "CSI", ipucsi->id);
+
+	endpoint = of_get_next_child(node, NULL);
+	if (endpoint)
+		v4l2_of_parse_endpoint(endpoint, &ipucsi->endpoint);
+
+	ipucsi->subdev.entity.ops = &ipucsi_entity_ops;
+
+	ipucsi->subdev.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	ipucsi->subdev.of_node = node;
+
+	ipucsi->subdev_pad[0].flags = MEDIA_PAD_FL_SINK;
+	ipucsi->subdev_pad[1].flags = MEDIA_PAD_FL_SOURCE;
+	ret = media_entity_pads_init(&ipucsi->subdev.entity, 2,
+				     ipucsi->subdev_pad);
+	if (ret < 0)
+		return ret;
+
+	ipucsi->subdev.internal_ops = &ipu_csi_internal_ops;
+	ret = v4l2_async_register_subdev(&ipucsi->subdev);
+	if (ret < 0) {
+		dev_err(ipucsi->dev, "Failed to register CSI subdev \"%s\": %d\n",
+			ipucsi->subdev.name, ret);
+	}
+
+	return ret;
+}
+
+static u64 camera_mask = DMA_BIT_MASK(32);
+
+static int ipucsi_probe(struct platform_device *pdev)
+{
+	struct ipu_client_platformdata *pdata = pdev->dev.platform_data;
+	struct ipu_soc *ipu = dev_get_drvdata(pdev->dev.parent);
+	struct ipucsi *ipucsi;
+	int ret;
+	struct device_node *node;
+
+	pdev->dev.dma_mask = &camera_mask;
+	pdev->dev.coherent_dma_mask = DMA_BIT_MASK(32);
+
+	if (!pdata)
+		return -EINVAL;
+
+	ipucsi = devm_kzalloc(&pdev->dev, sizeof(*ipucsi), GFP_KERNEL);
+	if (!ipucsi)
+		return -ENOMEM;
+
+	ipucsi->ipu = ipu;
+	ipucsi->id = pdata->csi;
+	ipucsi->dev = &pdev->dev;
+
+	node = of_graph_get_port_by_id(pdev->dev.parent->of_node, ipucsi->id);
+	if (!node) {
+		dev_err(&pdev->dev, "cannot find node port@%d\n", ipucsi->id);
+		return -ENODEV;
+	}
+
+	ipucsi->csi = ipu_csi_get(ipu, ipucsi->id);
+	if (IS_ERR(ipucsi->csi))
+		return PTR_ERR(ipucsi->csi);
+
+	ret = ipucsi_subdev_init(ipucsi, node);
+	if (ret)
+		return ret;
+
+	of_node_put(node);
+
+	platform_set_drvdata(pdev, ipucsi);
+
+	dev_info(&pdev->dev, "loaded\n");
+
+	return 0;
+}
+
+static struct platform_driver ipucsi_driver = {
+	.driver = {
+		.name = DRIVER_NAME,
+	},
+	.probe = ipucsi_probe,
+};
+
+static int __init ipucsi_init(void)
+{
+	return platform_driver_register(&ipucsi_driver);
+}
+
+static void __exit ipucsi_exit(void)
+{
+	return platform_driver_unregister(&ipucsi_driver);
+}
+
+subsys_initcall(ipucsi_init);
+module_exit(ipucsi_exit);
+
+MODULE_DESCRIPTION("i.MX IPUv3 capture interface driver");
+MODULE_AUTHOR("Sascha Hauer, Pengutronix");
+MODULE_AUTHOR("Philipp Zabel, Pengutronix");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:" DRIVER_NAME);
-- 
2.9.3

[PATCH 09/22] [media] imx: Add i.MX IPUv3 capture driver

[Philipp Zabel]

This driver uses the IDMAC module's double buffering feature to do the
processing of finished frames in the new frame acknowledge (NFACK)
interrupt handler while the next frame is already being captured. This
avoids a race condition between the end of frame interrupt and NFACK for
very short blanking intervals, but causes the driver to need at least
two buffers in flight. The last remaining frame will never be handed out
to userspace until a new one is queued.
It supports interlaced input and allows to translate between sequential
and interlaced field formats using the IDMAC scan order and interlace
offset parameters.
Currently the direct CSI -> SMFC -> IDMAC path is supported.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
Signed-off-by: Marc Kleine-Budde <mkl@pengutronix.de>
Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/media/platform/imx/Kconfig           |    9 +
 drivers/media/platform/imx/Makefile          |    1 +
 drivers/media/platform/imx/imx-ipu-capture.c | 1085 ++++++++++++++++++++++++++
 drivers/media/platform/imx/imx-ipu.h         |    9 +
 drivers/media/platform/imx/imx-ipuv3-csi.c   |   25 +
 5 files changed, 1129 insertions(+)
 create mode 100644 drivers/media/platform/imx/imx-ipu-capture.c

diff --git a/drivers/media/platform/imx/Kconfig b/drivers/media/platform/imx/Kconfig
index a88c4f7..69e8648 100644
--- a/drivers/media/platform/imx/Kconfig
+++ b/drivers/media/platform/imx/Kconfig
@@ -9,6 +9,15 @@ config MEDIA_IMX
 config VIDEO_IMX_IPU_COMMON
 	tristate
 
+config VIDEO_IMX_IPU_CAPTURE
+	tristate "i.MX5/6 Video Capture driver"
+	depends on IMX_IPUV3_CORE && VIDEO_V4L2_SUBDEV_API && MEDIA_IMX
+	select VIDEOBUF2_DMA_CONTIG
+	select VIDEO_IMX_IPU_COMMON
+	select VIDEO_IMX_IPUV3
+	help
+	  This is a v4l2 video capture driver for the IPUv3 on i.MX5/6.
+
 config VIDEO_IMX_IPU_CSI
 	tristate "i.MX5/6 CMOS Sensor Interface driver"
 	depends on VIDEO_DEV && IMX_IPUV3_CORE && MEDIA_IMX
diff --git a/drivers/media/platform/imx/Makefile b/drivers/media/platform/imx/Makefile
index 82a3616..919eaa1 100644
--- a/drivers/media/platform/imx/Makefile
+++ b/drivers/media/platform/imx/Makefile
@@ -1,3 +1,4 @@
 obj-$(CONFIG_MEDIA_IMX)			+= imx-media.o
 obj-$(CONFIG_VIDEO_IMX_IPU_COMMON)	+= imx-ipu.o
+obj-$(CONFIG_VIDEO_IMX_IPU_CAPTURE)	+= imx-ipu-capture.o
 obj-$(CONFIG_VIDEO_IMX_IPU_CSI)		+= imx-ipuv3-csi.o
diff --git a/drivers/media/platform/imx/imx-ipu-capture.c b/drivers/media/platform/imx/imx-ipu-capture.c
new file mode 100644
index 0000000..7d760b0
--- /dev/null
+++ b/drivers/media/platform/imx/imx-ipu-capture.c
@@ -0,0 +1,1085 @@
+/*
+ * i.MX IPUv3 V4L2 Capture Driver
+ *
+ * Copyright (C) 2016, Pengutronix, Philipp Zabel <kernel@pengutronix.de>
+ *
+ * Based on code
+ * Copyright (C) 2006, Pengutronix, Sascha Hauer <kernel@pengutronix.de>
+ * Copyright (C) 2008, Guennadi Liakhovetski <kernel@pengutronix.de>
+ * Copyright (C) 2008, Paulius Zaleckas <paulius.zaleckas@teltonika.lt>
+ * Copyright (C) 2009, Darius Augulis <augulis.darius@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+#include <linux/dma-mapping.h>
+#include <linux/interrupt.h>
+#include <linux/videodev2.h>
+#include <linux/device.h>
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <linux/mutex.h>
+#include <linux/slab.h>
+#include <linux/time.h>
+
+#include <video/imx-ipu-v3.h>
+#include "imx-ipu.h"
+
+#include <linux/of_graph.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-common.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-dev.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-mc.h>
+#include <media/v4l2-of.h>
+
+#define DRIVER_NAME "imx-ipuv3-capture"
+
+/* buffer for one video frame */
+struct ipu_capture_buffer {
+	struct vb2_v4l2_buffer		vb;
+	struct list_head		queue;
+};
+
+struct ipu_capture {
+	struct video_device		vdev;
+
+	struct device			*dev;
+	struct v4l2_fh			fh;
+	struct vb2_queue		vb2_vidq;
+	struct media_pad		pad;
+	struct media_pipeline		pipe;
+	struct v4l2_format		format;
+
+	struct v4l2_subdev		*csi_sd;
+	struct ipu_smfc			*smfc;
+	struct ipuv3_channel		*ipuch;
+	struct ipu_soc			*ipu;
+	int				id;
+
+	spinlock_t			lock;
+	struct mutex			mutex;
+
+	/* The currently active buffer, set by NFACK and cleared by EOF interrupt */
+	struct ipu_capture_buffer		*active;
+	struct list_head		capture;
+	int				ilo;
+	int				sequence;
+
+	int				done_count;
+	int				skip_count;
+};
+
+
+static struct ipu_capture_buffer *to_ipu_capture_buffer(struct vb2_buffer *vb)
+{
+	return container_of(vb, struct ipu_capture_buffer, vb.vb2_buf);
+}
+
+static inline void ipu_capture_set_inactive_buffer(struct ipu_capture *priv,
+						   struct vb2_buffer *vb)
+{
+	int bufptr = !ipu_idmac_get_current_buffer(priv->ipuch);
+	dma_addr_t eba = vb2_dma_contig_plane_dma_addr(vb, 0);
+
+	if (priv->ilo < 0)
+		eba -= priv->ilo;
+
+	ipu_cpmem_set_buffer(priv->ipuch, bufptr, eba);
+	ipu_idmac_select_buffer(priv->ipuch, bufptr);
+}
+
+static irqreturn_t ipu_capture_new_frame_handler(int irq, void *context)
+{
+	struct ipu_capture *priv = context;
+	struct ipu_capture_buffer *buf;
+	struct vb2_v4l2_buffer *vb;
+	unsigned long flags;
+
+	/* The IDMAC just started to write pixel data into the current buffer */
+
+	spin_lock_irqsave(&priv->lock, flags);
+
+	/*
+	 * If there is a previously active frame, mark it as done to hand it off
+	 * to userspace. Or, if there are no further frames queued, hold on to it.
+	 */
+	if (priv->active) {
+		vb = &priv->active->vb;
+		buf = to_ipu_capture_buffer(&vb->vb2_buf);
+
+		if (vb2_is_streaming(vb->vb2_buf.vb2_queue) &&
+		    list_is_singular(&priv->capture)) {
+			pr_debug("%s: reusing 0x%08x\n", __func__,
+				vb2_dma_contig_plane_dma_addr(&vb->vb2_buf, 0));
+			/* DEBUG: check if buf == EBA(active) */
+		} else {
+			/* Otherwise, mark buffer as finished */
+			list_del_init(&buf->queue);
+
+			vb2_buffer_done(&vb->vb2_buf, VB2_BUF_STATE_DONE);
+			priv->done_count++;
+		}
+	}
+
+	if (list_empty(&priv->capture))
+		goto out;
+
+	priv->active = list_first_entry(&priv->capture,
+					   struct ipu_capture_buffer, queue);
+	vb = &priv->active->vb;
+	vb->vb2_buf.timestamp = ktime_get_ns();
+	vb->field = priv->format.fmt.pix.field;
+	vb->sequence = priv->sequence++;
+
+	/*
+	 * Point the inactive buffer address to the next queued buffer,
+	 * if available. Otherwise, prepare to reuse the currently active
+	 * buffer, unless ipu_capture_buf_queue gets called in time.
+	 */
+	if (!list_is_singular(&priv->capture)) {
+		buf = list_entry(priv->capture.next->next,
+				 struct ipu_capture_buffer, queue);
+		vb = &buf->vb;
+	}
+	ipu_capture_set_inactive_buffer(priv, &vb->vb2_buf);
+out:
+	spin_unlock_irqrestore(&priv->lock, flags);
+
+	return IRQ_HANDLED;
+}
+
+static int ipu_capture_get_resources(struct ipu_capture *priv)
+{
+	struct device *dev = priv->dev;
+	int channel = priv->id * 2;
+	int ret;
+
+	priv->ipuch = ipu_idmac_get(priv->ipu, channel);
+	if (IS_ERR(priv->ipuch)) {
+		ret = PTR_ERR(priv->ipuch);
+		dev_err(dev, "Failed to get IDMAC channel %d: %d\n", channel,
+			ret);
+		priv->ipuch = NULL;
+		return ret;
+	}
+
+	priv->smfc = ipu_smfc_get(priv->ipu, channel);
+	if (!priv->smfc) {
+		dev_err(dev, "Failed to get SMFC channel %d\n", channel);
+		ret = -EBUSY;
+		goto err;
+	}
+
+	return 0;
+err:
+	ipu_idmac_put(priv->ipuch);
+	return ret;
+}
+
+static void ipu_capture_put_resources(struct ipu_capture *priv)
+{
+	if (priv->ipuch) {
+		ipu_idmac_put(priv->ipuch);
+		priv->ipuch = NULL;
+	}
+
+	if (priv->smfc) {
+		ipu_smfc_put(priv->smfc);
+		priv->smfc = NULL;
+	}
+}
+
+/*
+ *  Videobuf operations
+ */
+static int ipu_capture_queue_setup(struct vb2_queue *vq, unsigned int *count,
+				   unsigned int *num_planes, unsigned int sizes[],
+				   struct device *alloc_devs[])
+{
+	struct ipu_capture *priv = vb2_get_drv_priv(vq);
+
+	priv->sequence = 0;
+
+	if (!*count)
+		*count = 32;
+	*num_planes = 1;
+	sizes[0] = priv->format.fmt.pix.sizeimage;
+
+	return 0;
+}
+
+static int ipu_capture_buf_prepare(struct vb2_buffer *vb)
+{
+	struct ipu_capture *priv = vb2_get_drv_priv(vb->vb2_queue);
+	struct v4l2_pix_format *pix = &priv->format.fmt.pix;
+
+	if (vb2_plane_size(vb, 0) < pix->sizeimage)
+		return -ENOBUFS;
+
+	vb2_set_plane_payload(vb, 0, pix->sizeimage);
+
+	return 0;
+}
+
+static void ipu_capture_buf_queue(struct vb2_buffer *vb)
+{
+	struct vb2_queue *vq = vb->vb2_queue;
+	struct ipu_capture *priv = vb2_get_drv_priv(vq);
+	struct ipu_capture_buffer *buf = to_ipu_capture_buffer(vb);
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->lock, flags);
+
+	/*
+	 * If there is no next buffer queued, point the inactive buffer
+	 * address to the incoming buffer
+	 */
+	if (vb2_is_streaming(vb->vb2_queue) &&
+	    list_is_singular(&priv->capture))
+		ipu_capture_set_inactive_buffer(priv, vb);
+
+	list_add_tail(&buf->queue, &priv->capture);
+
+	spin_unlock_irqrestore(&priv->lock, flags);
+}
+
+static void ipu_capture_buf_cleanup(struct vb2_buffer *vb)
+{
+	struct vb2_queue *vq = vb->vb2_queue;
+	struct ipu_capture *priv = vb2_get_drv_priv(vq);
+	struct ipu_capture_buffer *buf = to_ipu_capture_buffer(vb);
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->lock, flags);
+
+	if (priv->active == buf)
+		priv->active = NULL;
+
+	if (!list_empty(&buf->queue))
+		list_del_init(&buf->queue);
+
+	spin_unlock_irqrestore(&priv->lock, flags);
+
+	ipu_capture_put_resources(priv);
+}
+
+static int ipu_capture_buf_init(struct vb2_buffer *vb)
+{
+	struct ipu_capture_buffer *buf = to_ipu_capture_buffer(vb);
+
+	/* This is for locking debugging only */
+	INIT_LIST_HEAD(&buf->queue);
+
+	return 0;
+}
+
+int v4l2_media_subdev_prepare_stream(struct media_entity *entity)
+{
+	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_entity_graph graph;
+	struct media_entity *first;
+	struct v4l2_subdev *sd;
+	int ret;
+
+	first = entity;
+
+	mutex_lock(&mdev->graph_mutex);
+
+	ret = media_entity_graph_walk_init(&graph, mdev);
+	if (ret < 0) {
+		mutex_unlock(&mdev->graph_mutex);
+		return ret;
+	}
+
+	media_entity_graph_walk_start(&graph, entity);
+
+	while (!ret && (entity = media_entity_graph_walk_next(&graph))) {
+		if (is_media_entity_v4l2_subdev(entity)) {
+			sd = media_entity_to_v4l2_subdev(entity);
+			ret = v4l2_subdev_call(sd, video, prepare_stream);
+			if (ret == -ENOIOCTLCMD)
+				ret = 0;
+		}
+	}
+
+	if (!ret)
+		goto out;
+
+	media_entity_graph_walk_start(&graph, entity);
+
+	while ((entity = media_entity_graph_walk_next(&graph)) && first != entity) {
+		if (is_media_entity_v4l2_subdev(entity)) {
+			sd = media_entity_to_v4l2_subdev(entity);
+			v4l2_subdev_call(sd, video, s_stream, 0);
+		}
+	}
+
+out:
+	media_entity_graph_walk_cleanup(&graph);
+	mutex_unlock(&mdev->graph_mutex);
+	return ret;
+}
+
+
+int v4l2_media_subdev_s_stream(struct media_entity *entity, int enable)
+{
+	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_entity_graph graph;
+	struct media_entity *first;
+	struct v4l2_subdev *sd;
+	int ret;
+
+	first = entity;
+
+	mutex_lock(&mdev->graph_mutex);
+
+	ret = media_entity_graph_walk_init(&graph, mdev);
+	if (ret < 0) {
+		mutex_unlock(&mdev->graph_mutex);
+		return ret;
+	}
+
+	media_entity_graph_walk_start(&graph, entity);
+	if (!enable) {
+		first = NULL;
+		goto disable;
+	}
+
+	while (!ret && (entity = media_entity_graph_walk_next(&graph))) {
+		if (is_media_entity_v4l2_subdev(entity)) {
+			sd = media_entity_to_v4l2_subdev(entity);
+			ret = v4l2_subdev_call(sd, video, s_stream, 1);
+			if (ret == -ENOIOCTLCMD)
+				ret = 0;
+		}
+	}
+
+	if (!ret)
+		goto out;
+
+	media_entity_graph_walk_start(&graph, first);
+
+disable:
+	while ((entity = media_entity_graph_walk_next(&graph)) && first != entity) {
+		if (is_media_entity_v4l2_subdev(entity)) {
+			sd = media_entity_to_v4l2_subdev(entity);
+			v4l2_subdev_call(sd, video, s_stream, 0);
+		}
+	}
+
+out:
+	media_entity_graph_walk_cleanup(&graph);
+	mutex_unlock(&mdev->graph_mutex);
+	return ret;
+}
+
+static int ipu_capture_start_streaming(struct vb2_queue *vq, unsigned int count)
+{
+	struct ipu_capture *priv = vb2_get_drv_priv(vq);
+	struct v4l2_subdev *csi_sd = priv->csi_sd;
+	u32 width = priv->format.fmt.pix.width;
+	u32 height = priv->format.fmt.pix.height;
+	struct device *dev = priv->dev;
+	int burstsize;
+	struct ipu_capture_buffer *buf;
+	int nfack_irq;
+	int ret;
+	const char *irq_name[2] = { "CSI0", "CSI1" };
+	bool raw;
+
+	ret = ipu_capture_get_resources(priv);
+	if (ret < 0) {
+		dev_err(dev, "Failed to get resources: %d\n", ret);
+		goto err_dequeue;
+	}
+
+	ipu_cpmem_zero(priv->ipuch);
+
+	nfack_irq = ipu_idmac_channel_irq(priv->ipu, priv->ipuch,
+					  IPU_IRQ_NFACK);
+	ret = request_threaded_irq(nfack_irq, NULL,
+				   ipu_capture_new_frame_handler, IRQF_ONESHOT,
+				   irq_name[priv->id], priv);
+	if (ret) {
+		dev_err(dev, "Failed to request NFACK interrupt: %d\n", nfack_irq);
+		goto put_resources;
+	}
+
+	dev_dbg(dev, "width: %d height: %d, %.4s\n",
+		width, height, (char *)&priv->format.fmt.pix.pixelformat);
+
+	ipu_cpmem_set_resolution(priv->ipuch, width, height);
+
+	raw = false;
+
+	if (raw && priv->smfc) {
+		/*
+		 * raw formats. We can only pass them through to memory
+		 */
+		u32 fourcc = priv->format.fmt.pix.pixelformat;
+		int bytes;
+
+		switch (fourcc) {
+		case V4L2_PIX_FMT_GREY:
+			bytes = 1;
+			break;
+		case V4L2_PIX_FMT_Y10:
+		case V4L2_PIX_FMT_Y16:
+		case V4L2_PIX_FMT_UYVY:
+		case V4L2_PIX_FMT_YUYV:
+			bytes = 2;
+			break;
+		}
+
+		ipu_cpmem_set_stride(priv->ipuch, width * bytes);
+		ipu_cpmem_set_format_passthrough(priv->ipuch, bytes * 8);
+		/*
+		 * According to table 37-727 (SMFC Burst Size), burstsize should
+		 * be set to NBP[6:4] for PFS == 6. Unfortunately, with a 16-bit
+		 * bus any value below 4 doesn't produce proper images.
+		 */
+		burstsize = (64 / bytes) >> 3;
+	} else {
+		/*
+		 * formats we understand, we can write it in any format not requiring
+		 * colorspace conversion.
+		 */
+		u32 fourcc = priv->format.fmt.pix.pixelformat;
+
+		switch (fourcc) {
+		case V4L2_PIX_FMT_RGB32:
+			ipu_cpmem_set_stride(priv->ipuch, width * 4);
+			ipu_cpmem_set_fmt(priv->ipuch, fourcc);
+			break;
+		case V4L2_PIX_FMT_UYVY:
+		case V4L2_PIX_FMT_YUYV:
+			ipu_cpmem_set_stride(priv->ipuch, width * 2);
+			ipu_cpmem_set_yuv_interleaved(priv->ipuch, fourcc);
+			break;
+		case V4L2_PIX_FMT_YUV420:
+		case V4L2_PIX_FMT_YVU420:
+		case V4L2_PIX_FMT_NV12:
+		case V4L2_PIX_FMT_YUV422P:
+			ipu_cpmem_set_stride(priv->ipuch, width);
+			ipu_cpmem_set_fmt(priv->ipuch, fourcc);
+			ipu_cpmem_set_yuv_planar(priv->ipuch, fourcc,
+						 width, height);
+			burstsize = 16;
+			break;
+		default:
+			dev_err(dev, "invalid color format: %4.4s\n",
+				(char *)&fourcc);
+			ret = -EINVAL;
+			goto free_irq;
+		}
+	}
+
+	if (priv->ilo)
+		ipu_cpmem_interlaced_scan(priv->ipuch, priv->ilo);
+
+	if (priv->smfc) {
+		/*
+		 * Set the channel for the direct CSI-->memory via SMFC
+		 * use-case to very high priority, by enabling the watermark
+		 * signal in the SMFC, enabling WM in the channel, and setting
+		 * the channel priority to high.
+		 *
+		 * Refer to the iMx6 rev. D TRM Table 36-8: Calculated priority
+		 * value.
+		 *
+		 * The WM's are set very low by intention here to ensure that
+		 * the SMFC FIFOs do not overflow.
+		 */
+		ipu_smfc_set_watermark(priv->smfc, 2, 1);
+		ipu_idmac_enable_watermark(priv->ipuch, true);
+		ipu_cpmem_set_high_priority(priv->ipuch);
+
+		/* Superfluous due to call to ipu_cpmem_zero above */
+		ipu_cpmem_set_axi_id(priv->ipuch, 0);
+
+		ipu_smfc_set_burstsize(priv->smfc, burstsize - 1);
+		ipu_smfc_map_channel(priv->smfc, priv->id * 2, 0);
+	}
+
+	/* Set the media pipeline to streaming state */
+	ret = media_entity_pipeline_start(&csi_sd->entity, &priv->pipe);
+	if (ret) {
+		dev_err(dev, "Failed to start external media pipeline\n");
+		goto stop_pipe;
+	}
+
+	ipu_idmac_set_double_buffer(priv->ipuch, 1);
+
+	if (list_empty(&priv->capture)) {
+		dev_err(dev, "No capture buffers\n");
+		ret = -ENOMEM;
+		goto stop_pipe;
+	}
+
+	priv->active = NULL;
+
+	/* Point the inactive buffer address to the first buffer */
+	buf = list_first_entry(&priv->capture, struct ipu_capture_buffer, queue);
+	ipu_capture_set_inactive_buffer(priv, &buf->vb.vb2_buf);
+
+	ipu_idmac_enable_channel(priv->ipuch);
+
+	if (priv->smfc)
+		ipu_smfc_enable(priv->smfc);
+
+	ret = v4l2_media_subdev_prepare_stream(&csi_sd->entity);
+	if (ret) {
+		dev_err(dev, "Failed to prepare streaming\n");
+		goto stop_pipe;
+	}
+
+	ret = v4l2_media_subdev_s_stream(&priv->vdev.entity, 1);
+	if (ret) {
+		dev_err(dev, "Failed to start streaming\n");
+		goto stop_pipe;
+	}
+
+	return 0;
+
+stop_pipe:
+	media_entity_pipeline_stop(&csi_sd->entity);
+free_irq:
+	free_irq(nfack_irq, priv);
+put_resources:
+	ipu_capture_put_resources(priv);
+err_dequeue:
+	while (!list_empty(&vq->queued_list)) {
+		struct vb2_v4l2_buffer *buf;
+
+		buf = to_vb2_v4l2_buffer(list_first_entry(&vq->queued_list,
+							  struct vb2_buffer,
+							  queued_entry));
+		list_del(&buf->vb2_buf.queued_entry);
+		vb2_buffer_done(&buf->vb2_buf, VB2_BUF_STATE_QUEUED);
+	}
+	return ret;
+}
+
+static void ipu_capture_stop_streaming(struct vb2_queue *vq)
+{
+	struct ipu_capture *priv = vb2_get_drv_priv(vq);
+	unsigned long flags;
+	int nfack_irq = ipu_idmac_channel_irq(priv->ipu, priv->ipuch,
+					      IPU_IRQ_NFACK);
+
+	free_irq(nfack_irq, priv);
+
+	v4l2_media_subdev_s_stream(&priv->vdev.entity, 0);
+	ipu_idmac_disable_channel(priv->ipuch);
+	if (priv->smfc)
+		ipu_smfc_disable(priv->smfc);
+
+	spin_lock_irqsave(&priv->lock, flags);
+	while (!list_empty(&priv->capture)) {
+		struct ipu_capture_buffer *buf = list_entry(priv->capture.next,
+						 struct ipu_capture_buffer, queue);
+		list_del_init(priv->capture.next);
+		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
+	}
+	spin_unlock_irqrestore(&priv->lock, flags);
+
+	media_entity_pipeline_stop(&priv->csi_sd->entity);
+
+	ipu_capture_put_resources(priv);
+}
+
+static void ipu_capture_lock(struct vb2_queue *vq)
+{
+	struct ipu_capture *priv = vb2_get_drv_priv(vq);
+
+	mutex_lock(&priv->mutex);
+}
+
+static void ipu_capture_unlock(struct vb2_queue *vq)
+{
+	struct ipu_capture *priv = vb2_get_drv_priv(vq);
+
+	mutex_unlock(&priv->mutex);
+}
+
+static struct vb2_ops ipu_capture_vb2_ops = {
+	.queue_setup		= ipu_capture_queue_setup,
+	.buf_prepare		= ipu_capture_buf_prepare,
+	.buf_queue		= ipu_capture_buf_queue,
+	.buf_cleanup		= ipu_capture_buf_cleanup,
+	.buf_init		= ipu_capture_buf_init,
+	.start_streaming	= ipu_capture_start_streaming,
+	.stop_streaming		= ipu_capture_stop_streaming,
+	.wait_prepare		= ipu_capture_unlock,
+	.wait_finish		= ipu_capture_lock,
+};
+
+static int ipu_capture_querycap(struct file *file, void *priv,
+					struct v4l2_capability *cap)
+{
+	strlcpy(cap->driver, "imx-ipuv3-capture", sizeof(cap->driver));
+	/* cap->name is set by the friendly caller:-> */
+	strlcpy(cap->card, "imx-ipuv3-csi", sizeof(cap->card));
+	strlcpy(cap->bus_info, "platform:imx-ipuv3-capture", sizeof(cap->bus_info));
+	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
+
+	return 0;
+}
+
+static int ipu_capture_try_fmt(struct file *file, void *fh,
+			       struct v4l2_format *f)
+{
+	struct ipu_capture *priv = video_drvdata(file);
+	struct v4l2_subdev_format sd_fmt;
+	struct ipu_fmt *fmt = NULL;
+	enum v4l2_field in, out;
+	int bytes_per_pixel;
+	int ret;
+
+	sd_fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+	sd_fmt.pad = 1;
+	ret = v4l2_subdev_call(priv->csi_sd, pad, get_fmt, NULL, &sd_fmt);
+	if (ret)
+		return ret;
+
+	in = sd_fmt.format.field;
+	out = f->fmt.pix.field;
+
+	switch (sd_fmt.format.code) {
+	case MEDIA_BUS_FMT_FIXED:
+		fmt = ipu_find_fmt_rgb(f->fmt.pix.pixelformat);
+		if (!fmt)
+			return -EINVAL;
+		bytes_per_pixel = fmt->bytes_per_pixel;
+		break;
+	case MEDIA_BUS_FMT_UYVY8_2X8:
+	case MEDIA_BUS_FMT_YUYV8_2X8:
+		fmt = ipu_find_fmt_yuv(f->fmt.pix.pixelformat);
+		if (!fmt)
+			return -EINVAL;
+		bytes_per_pixel = fmt->bytes_per_pixel;
+		break;
+	case MEDIA_BUS_FMT_Y8_1X8:
+		f->fmt.pix.pixelformat = V4L2_PIX_FMT_GREY;
+		bytes_per_pixel = 1;
+		break;
+	case MEDIA_BUS_FMT_UYVY8_1X16:
+		f->fmt.pix.pixelformat = V4L2_PIX_FMT_UYVY;
+		bytes_per_pixel = 2;
+		break;
+	case MEDIA_BUS_FMT_YUYV8_1X16:
+		f->fmt.pix.pixelformat = V4L2_PIX_FMT_YUYV;
+		bytes_per_pixel = 2;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	v4l_bound_align_image(&f->fmt.pix.width, 128,
+			      sd_fmt.format.width, 3,
+			      &f->fmt.pix.height, 128,
+			      sd_fmt.format.height, 1, 0);
+
+	f->fmt.pix.bytesperline = f->fmt.pix.width * bytes_per_pixel;
+	f->fmt.pix.sizeimage = f->fmt.pix.bytesperline * f->fmt.pix.height;
+	if (fmt) {
+		if (fmt->fourcc == V4L2_PIX_FMT_YUV420 ||
+		    fmt->fourcc == V4L2_PIX_FMT_YVU420 ||
+		    fmt->fourcc == V4L2_PIX_FMT_NV12)
+			f->fmt.pix.sizeimage = f->fmt.pix.sizeimage * 3 / 2;
+		else if (fmt->fourcc == V4L2_PIX_FMT_YUV422P)
+			f->fmt.pix.sizeimage *= 2;
+	}
+
+	if ((in == V4L2_FIELD_SEQ_TB && out == V4L2_FIELD_INTERLACED_TB) ||
+	    (in == V4L2_FIELD_INTERLACED_TB && out == V4L2_FIELD_SEQ_TB) ||
+	    (in == V4L2_FIELD_SEQ_BT && out == V4L2_FIELD_INTERLACED_BT) ||
+	    (in == V4L2_FIELD_INTERLACED_BT && out == V4L2_FIELD_SEQ_BT)) {
+		/*
+		 * IDMAC scan order can be used for translation between
+		 * interlaced and sequential field formats.
+		 */
+	} else if (out == V4L2_FIELD_NONE || out == V4L2_FIELD_INTERLACED) {
+		/*
+		 * If userspace requests progressive or interlaced frames,
+		 * interlace sequential fields as closest approximation.
+		 */
+		if (in == V4L2_FIELD_SEQ_TB)
+			out = V4L2_FIELD_INTERLACED_TB;
+		else if (in == V4L2_FIELD_SEQ_BT)
+			out = V4L2_FIELD_INTERLACED_BT;
+		else
+			out = in;
+	} else {
+		/* Translation impossible or userspace doesn't care */
+		out = in;
+	}
+	f->fmt.pix.field = out;
+
+	return 0;
+}
+
+static int ipu_capture_s_fmt(struct file *file, void *fh,
+		struct v4l2_format *f)
+{
+	struct ipu_capture *priv = video_drvdata(file);
+	struct v4l2_subdev_format sd_fmt;
+	enum v4l2_field in, out;
+	int ret;
+
+	sd_fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+	sd_fmt.pad = 1;
+	ret = v4l2_subdev_call(priv->csi_sd, pad, get_fmt, NULL, &sd_fmt);
+	if (ret)
+		return ret;
+
+	ret = ipu_capture_try_fmt(file, fh, f);
+	if (ret)
+		return ret;
+
+	priv->format = *f;
+
+	/*
+	 * Set IDMAC scan order interlace offset (ILO) for translation between
+	 * interlaced and sequential field formats.
+	 */
+	in = sd_fmt.format.field;
+	out = f->fmt.pix.field;
+	if ((in == V4L2_FIELD_SEQ_TB && out == V4L2_FIELD_INTERLACED_TB) ||
+	    (in == V4L2_FIELD_INTERLACED_TB && out == V4L2_FIELD_SEQ_TB))
+		priv->ilo = f->fmt.pix.bytesperline;
+	else if ((in == V4L2_FIELD_SEQ_BT && out == V4L2_FIELD_INTERLACED_BT) ||
+		 (in == V4L2_FIELD_INTERLACED_BT && out == V4L2_FIELD_SEQ_BT))
+		priv->ilo = -f->fmt.pix.bytesperline;
+	else
+		priv->ilo = 0;
+
+	return 0;
+}
+
+static int ipu_capture_g_fmt(struct file *file, void *fh, struct v4l2_format *f)
+{
+	struct ipu_capture *priv = video_drvdata(file);
+
+	if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	*f = priv->format;
+
+	return 0;
+}
+
+static int ipu_capture_enum_fmt(struct file *file, void *fh,
+				struct v4l2_fmtdesc *f)
+{
+	struct ipu_capture *priv = video_drvdata(file);
+	struct v4l2_subdev_format sd_fmt;
+	u32 fourcc;
+	int ret;
+
+	sd_fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+	sd_fmt.pad = 1;
+	ret = v4l2_subdev_call(priv->csi_sd, pad, get_fmt, NULL, &sd_fmt);
+	if (ret)
+		return ret;
+
+	switch (sd_fmt.format.code) {
+	case V4L2_PIX_FMT_RGB32:
+		return ipu_enum_fmt_rgb(file, priv, f);
+	case MEDIA_BUS_FMT_UYVY8_2X8:
+	case MEDIA_BUS_FMT_YUYV8_2X8:
+		return ipu_enum_fmt_yuv(file, priv, f);
+	case MEDIA_BUS_FMT_Y8_1X8:
+		fourcc = V4L2_PIX_FMT_GREY;
+		break;
+	case MEDIA_BUS_FMT_UYVY8_1X16:
+		fourcc = V4L2_PIX_FMT_UYVY;
+		break;
+	case MEDIA_BUS_FMT_YUYV8_1X16:
+		fourcc = V4L2_PIX_FMT_YUYV;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	if (f->index)
+		return -EINVAL;
+
+	f->pixelformat = fourcc;
+
+	return 0;
+}
+
+static int ipu_capture_link_setup(struct media_entity *entity,
+				  const struct media_pad *local,
+				  const struct media_pad *remote, u32 flags)
+{
+	struct video_device *vdev = media_entity_to_video_device(entity);
+	struct ipu_capture *priv = container_of(vdev, struct ipu_capture, vdev);
+
+	if (priv->smfc)
+		ipu_smfc_map_channel(priv->smfc, priv->id * 2, 0);
+
+	return 0;
+}
+
+struct media_entity_operations ipu_capture_entity_ops = {
+	.link_setup = ipu_capture_link_setup,
+};
+
+static int ipu_capture_open(struct file *file)
+{
+	struct ipu_capture *priv = video_drvdata(file);
+	int ret;
+
+	ret = v4l2_pipeline_pm_use(&priv->vdev.entity, 1);
+	if (ret)
+		return ret;
+
+	mutex_lock(&priv->mutex);
+	ret = v4l2_fh_open(file);
+	mutex_unlock(&priv->mutex);
+
+	return ret;
+}
+
+static int ipu_capture_release(struct file *file)
+{
+	struct ipu_capture *priv = video_drvdata(file);
+
+	v4l2_pipeline_pm_use(&priv->vdev.entity, 0);
+
+	if (v4l2_fh_is_singular_file(file))
+		vb2_fop_release(file);
+	else
+		v4l2_fh_release(file);
+
+	return 0;
+}
+
+static const struct v4l2_file_operations ipu_capture_fops = {
+	.owner		= THIS_MODULE,
+	.open		= ipu_capture_open,
+	.release	= ipu_capture_release,
+	.unlocked_ioctl	= video_ioctl2,
+	.mmap		= vb2_fop_mmap,
+	.poll		= vb2_fop_poll,
+};
+
+static int ipu_capture_g_parm(struct file *file, void *fh,
+			 struct v4l2_streamparm *sp)
+{
+	struct ipu_capture *priv = video_drvdata(file);
+	struct v4l2_subdev *sd = priv->csi_sd;
+	struct v4l2_subdev_frame_interval fi;
+
+	if (sp->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	sp->parm.capture.capability = V4L2_CAP_TIMEPERFRAME;
+	v4l2_subdev_call(sd, video, g_frame_interval, &fi);
+	sp->parm.capture.timeperframe = fi.interval;
+
+	return 0;
+}
+
+static int ipu_capture_s_parm(struct file *file, void *fh,
+			 struct v4l2_streamparm *sp)
+{
+	struct ipu_capture *priv = video_drvdata(file);
+	struct v4l2_subdev *sd = priv->csi_sd;
+	struct v4l2_subdev_frame_interval fi;
+
+	if (sp->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	fi.pad = 1;
+	fi.interval = sp->parm.capture.timeperframe;
+	v4l2_subdev_call(sd, video, s_frame_interval, &fi);
+	v4l2_subdev_call(sd, video, g_frame_interval, &fi);
+	sp->parm.capture.timeperframe = fi.interval;
+
+	return 0;
+}
+
+static int ipu_capture_enum_framesizes(struct file *file, void *fh,
+				       struct v4l2_frmsizeenum *fsize)
+{
+	struct ipu_capture *priv = video_drvdata(file);
+	struct v4l2_subdev_format sd_fmt;
+	struct ipu_fmt *fmt = NULL;
+	int ret;
+
+	if (fsize->index != 0)
+		return -EINVAL;
+
+	sd_fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+	sd_fmt.pad = 1;
+	ret = v4l2_subdev_call(priv->csi_sd, pad, get_fmt, NULL, &sd_fmt);
+	if (ret)
+		return ret;
+
+	switch (sd_fmt.format.code) {
+	case V4L2_PIX_FMT_RGB32:
+		fmt = ipu_find_fmt_rgb(fsize->pixel_format);
+		if (!fmt)
+			return -EINVAL;
+		break;
+	case MEDIA_BUS_FMT_UYVY8_2X8:
+	case MEDIA_BUS_FMT_YUYV8_2X8:
+		fmt = ipu_find_fmt_yuv(fsize->pixel_format);
+		if (!fmt)
+			return -EINVAL;
+		break;
+	case MEDIA_BUS_FMT_Y8_1X8:
+		if (fsize->pixel_format != V4L2_PIX_FMT_GREY)
+			return -EINVAL;
+		break;
+	case MEDIA_BUS_FMT_Y10_1X10:
+		if (fsize->pixel_format != V4L2_PIX_FMT_Y10)
+			return -EINVAL;
+		break;
+	case MEDIA_BUS_FMT_Y12_1X12:
+		if (fsize->pixel_format != V4L2_PIX_FMT_Y16)
+			return -EINVAL;
+		break;
+	case MEDIA_BUS_FMT_UYVY8_1X16:
+		if (fsize->pixel_format != V4L2_PIX_FMT_UYVY)
+			return -EINVAL;
+		break;
+	case MEDIA_BUS_FMT_YUYV8_1X16:
+		if (fsize->pixel_format != V4L2_PIX_FMT_YUYV)
+			return -EINVAL;
+		break;
+	}
+
+	fsize->type = V4L2_FRMSIZE_TYPE_DISCRETE;
+	fsize->discrete.width = sd_fmt.format.width;
+	fsize->discrete.height = sd_fmt.format.height;
+
+	return 0;
+}
+
+static const struct v4l2_ioctl_ops ipu_capture_ioctl_ops = {
+	.vidioc_querycap		= ipu_capture_querycap,
+
+	.vidioc_enum_fmt_vid_cap	= ipu_capture_enum_fmt,
+	.vidioc_try_fmt_vid_cap		= ipu_capture_try_fmt,
+	.vidioc_s_fmt_vid_cap		= ipu_capture_s_fmt,
+	.vidioc_g_fmt_vid_cap		= ipu_capture_g_fmt,
+
+	.vidioc_create_bufs		= vb2_ioctl_create_bufs,
+	.vidioc_reqbufs			= vb2_ioctl_reqbufs,
+	.vidioc_querybuf		= vb2_ioctl_querybuf,
+
+	.vidioc_qbuf			= vb2_ioctl_qbuf,
+	.vidioc_dqbuf			= vb2_ioctl_dqbuf,
+	.vidioc_expbuf			= vb2_ioctl_expbuf,
+
+	.vidioc_streamon		= vb2_ioctl_streamon,
+	.vidioc_streamoff		= vb2_ioctl_streamoff,
+
+	.vidioc_g_parm			= ipu_capture_g_parm,
+	.vidioc_s_parm			= ipu_capture_s_parm,
+
+	.vidioc_enum_framesizes		= ipu_capture_enum_framesizes,
+};
+
+static int ipu_capture_vb2_init(struct ipu_capture *priv)
+{
+	struct vb2_queue *q = &priv->vb2_vidq;
+
+	q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	q->io_modes = VB2_MMAP | VB2_DMABUF;
+	q->drv_priv = priv;
+	q->ops = &ipu_capture_vb2_ops;
+	q->mem_ops = &vb2_dma_contig_memops;
+	q->buf_struct_size = sizeof(struct ipu_capture_buffer);
+	q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+	q->dev = priv->dev;
+
+	return vb2_queue_init(q);
+}
+
+struct ipu_capture *ipu_capture_create(struct device *dev, struct ipu_soc *ipu,
+				       int id, struct v4l2_subdev *sd,
+				       int pad_index)
+{
+	struct video_device *vdev;
+	struct ipu_capture *priv;
+	struct media_link *link;
+	int ret;
+
+	priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return ERR_PTR(-ENOMEM);
+
+	priv->dev = dev;
+	priv->ipu = ipu;
+	priv->csi_sd = sd;
+
+	INIT_LIST_HEAD(&priv->capture);
+	spin_lock_init(&priv->lock);
+	mutex_init(&priv->mutex);
+
+	ret = ipu_capture_vb2_init(priv);
+	if (ret < 0)
+		return ERR_PTR(ret);
+
+	vdev = &priv->vdev;
+
+	snprintf(vdev->name, sizeof(vdev->name), DRIVER_NAME ".%d", id);
+	vdev->release	= video_device_release_empty;
+	vdev->fops	= &ipu_capture_fops;
+	vdev->ioctl_ops	= &ipu_capture_ioctl_ops;
+	vdev->v4l2_dev	= sd->v4l2_dev;
+	vdev->minor	= -1;
+	vdev->release	= video_device_release_empty;
+	vdev->lock	= &priv->mutex;
+	vdev->queue	= &priv->vb2_vidq;
+
+	video_set_drvdata(vdev, priv);
+
+	priv->pad.flags = MEDIA_PAD_FL_SINK;
+	vdev->entity.ops = &ipu_capture_entity_ops;
+
+	ret = media_entity_pads_init(&vdev->entity, 1, &priv->pad);
+	if (ret < 0)
+		return ERR_PTR(ret);
+
+	ret = video_register_device(&priv->vdev, VFL_TYPE_GRABBER, -1);
+	if (ret)
+		media_entity_cleanup(&vdev->entity);
+
+	ret = media_create_pad_link(&sd->entity, pad_index,
+				    &vdev->entity, 0, 0);
+	if (ret < 0) {
+		v4l2_err(sd->v4l2_dev,
+			 "failed to create link for '%s':%d: %d\n",
+			 sd->entity.name, pad_index, ret);
+		return ERR_PTR(ret);
+	}
+
+	link = media_entity_find_link(&sd->entity.pads[pad_index],
+				      &vdev->entity.pads[0]);
+	media_entity_setup_link(link, MEDIA_LNK_FL_ENABLED);
+
+	return priv;
+}
+EXPORT_SYMBOL_GPL(ipu_capture_create);
+
+void ipu_capture_destroy(struct ipu_capture *priv)
+{
+	video_unregister_device(&priv->vdev);
+	media_entity_cleanup(&priv->vdev.entity);
+	ipu_capture_put_resources(priv);
+
+	kfree(priv);
+}
+EXPORT_SYMBOL_GPL(ipu_capture_destroy);
diff --git a/drivers/media/platform/imx/imx-ipu.h b/drivers/media/platform/imx/imx-ipu.h
index 7b344b6..3690915 100644
--- a/drivers/media/platform/imx/imx-ipu.h
+++ b/drivers/media/platform/imx/imx-ipu.h
@@ -31,4 +31,13 @@ int ipu_g_fmt(struct v4l2_format *f, struct v4l2_pix_format *pix);
 int ipu_enum_framesizes(struct file *file, void *fh,
 			struct v4l2_frmsizeenum *fsize);
 
+struct device;
+struct ipu_soc;
+struct v4l2_subdev;
+
+struct ipu_capture *ipu_capture_create(struct device *dev, struct ipu_soc *ipu,
+				       int id, struct v4l2_subdev *sd,
+				       int pad_index);
+void ipu_capture_destroy(struct ipu_capture *capture);
+
 #endif /* __MEDIA_IMX_IPU_H */
diff --git a/drivers/media/platform/imx/imx-ipuv3-csi.c b/drivers/media/platform/imx/imx-ipuv3-csi.c
index 316175b..699460e6 100644
--- a/drivers/media/platform/imx/imx-ipuv3-csi.c
+++ b/drivers/media/platform/imx/imx-ipuv3-csi.c
@@ -61,6 +61,8 @@ struct ipucsi {
 	struct media_pad		subdev_pad[2];
 	struct v4l2_mbus_framefmt	format_mbus[2];
 	struct v4l2_fract		timeperframe[2];
+
+	struct ipu_capture		*capture;
 };
 
 static int ipu_csi_get_mbus_config(struct ipucsi *ipucsi,
@@ -384,6 +386,7 @@ static int ipu_csi_registered(struct v4l2_subdev *sd)
 {
 	struct ipucsi *ipucsi = container_of(sd, struct ipucsi, subdev);
 	struct device_node *rpp;
+	int ret;
 
 	/*
 	 * Add source subdevice to asynchronous subdevice waiting list.
@@ -404,11 +407,33 @@ static int ipu_csi_registered(struct v4l2_subdev *sd)
 		__v4l2_async_notifier_add_subdev(sd->notifier, asd);
 	}
 
+	/*
+	 * Create an ipu_capture instance per CSI.
+	 */
+	ipucsi->capture = ipu_capture_create(ipucsi->dev, ipucsi->ipu,
+					     ipucsi->id, sd, 1);
+	if (IS_ERR(ipucsi->capture)) {
+		ret = PTR_ERR(ipucsi->capture);
+		ipucsi->capture = NULL;
+		v4l2_err(sd->v4l2_dev, "Failed to create capture device for %s: %d\n",
+			 sd->name, ret);
+		return ret;
+	}
+
 	return 0;
 }
 
+static void ipu_csi_unregistered(struct v4l2_subdev *sd)
+{
+	struct ipucsi *ipucsi = container_of(sd, struct ipucsi, subdev);
+
+	if (ipucsi->capture)
+		ipu_capture_destroy(ipucsi->capture);
+}
+
 struct v4l2_subdev_internal_ops ipu_csi_internal_ops = {
 	.registered = ipu_csi_registered,
+	.unregistered = ipu_csi_unregistered,
 };
 
 static int ipucsi_subdev_init(struct ipucsi *ipucsi, struct device_node *node)
-- 
2.9.3

[PATCH 10/22] [media] platform: add video-multiplexer subdevice driver

[Philipp Zabel]

This driver can handle SoC internal and external video bus multiplexers,
controlled either by register bit fields or by a GPIO. The subdevice
passes through frame interval and mbus configuration of the active input
to the output side.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 .../bindings/media/video-multiplexer.txt           |  59 +++
 drivers/media/platform/Kconfig                     |   8 +
 drivers/media/platform/Makefile                    |   2 +
 drivers/media/platform/video-multiplexer.c         | 444 +++++++++++++++++++++
 4 files changed, 513 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 drivers/media/platform/video-multiplexer.c

diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
new file mode 100644
index 0000000..9d133d9
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
@@ -0,0 +1,59 @@
+Video Multiplexer
+=================
+
+Video multiplexers allow to select between multiple input ports. Video received
+on the active input port is passed through to the output port. Muxes described
+by this binding may be controlled by a syscon register bitfield or by a GPIO.
+
+Required properties:
+- compatible : should be "video-multiplexer"
+- reg: should be register base of the register containing the control bitfield
+- bit-mask: bitmask of the control bitfield in the control register
+- bit-shift: bit offset of the control bitfield in the control register
+- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
+  may be given to switch between two inputs
+- #address-cells: should be <1>
+- #size-cells: should be <0>
+- port@*: at least three port nodes containing endpoints connecting to the
+  source and sink devices according to of_graph bindings. The last port is
+  the output port, all others are inputs.
+
+Example:
+
+syscon {
+	compatible = "syscon", "simple-mfd";
+
+	mux {
+		compatible = "video-multiplexer";
+		/* Single bit (1 << 19) in syscon register 0x04: */
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <19>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port@0 {
+			reg = <0>;
+
+			mux_in0: endpoint {
+				remote-endpoint = <&video_source0_out>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			mux_in1: endpoint {
+				remote-endpoint = <&video_source1_out>;
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			mux_out: endpoint {
+				remote-endpoint = <&capture_interface_in>;
+			};
+		};
+	};
+};
diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
index 105bf57..92680f6 100644
--- a/drivers/media/platform/Kconfig
+++ b/drivers/media/platform/Kconfig
@@ -76,6 +76,14 @@ config VIDEO_M32R_AR_M64278
 	  To compile this driver as a module, choose M here: the
 	  module will be called arv.
 
+config VIDEO_MULTIPLEXER
+	tristate "Video Multiplexer"
+	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
+	help
+	  This driver provides support for SoC internal N:1 video bus
+	  multiplexers controlled by register bitfields as well as external
+	  2:1 video multiplexers controlled by a single GPIO.
+
 config VIDEO_OMAP3
 	tristate "OMAP 3 Camera support"
 	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
index f7f9008..a251755a 100644
--- a/drivers/media/platform/Makefile
+++ b/drivers/media/platform/Makefile
@@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
 
 obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
 
+obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
+
 obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
 obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
 obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
diff --git a/drivers/media/platform/video-multiplexer.c b/drivers/media/platform/video-multiplexer.c
new file mode 100644
index 0000000..e9137ba
--- /dev/null
+++ b/drivers/media/platform/video-multiplexer.c
@@ -0,0 +1,444 @@
+/*
+ * video stream multiplexer controlled via gpio or syscon
+ *
+ * Copyright (C) 2013 Pengutronix, Sascha Hauer <kernel@pengutronix.de>
+ * Copyright (C) 2016 Pengutronix, Philipp Zabel <kernel@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/err.h>
+#include <linux/gpio/consumer.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+
+struct vidsw {
+	struct v4l2_subdev subdev;
+	unsigned int num_pads;
+	struct media_pad *pads;
+	struct v4l2_mbus_framefmt *format_mbus;
+	struct v4l2_fract timeperframe;
+	struct v4l2_of_endpoint *endpoint;
+	struct regmap_field *field;
+	struct gpio_desc *gpio;
+	int active;
+};
+
+static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
+{
+	return container_of(sd, struct vidsw, subdev);
+}
+
+static void vidsw_set_active(struct vidsw *vidsw, int active)
+{
+	vidsw->active = active;
+	if (active < 0)
+		return;
+
+	dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
+
+	if (vidsw->field)
+		regmap_field_write(vidsw->field, active);
+	else if (vidsw->gpio)
+		gpiod_set_value(vidsw->gpio, active);
+}
+
+static int vidsw_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	/* We have no limitations on enabling or disabling our output link */
+	if (local->index == vidsw->num_pads - 1)
+		return 0;
+
+	dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
+		if (local->index == vidsw->active) {
+			dev_dbg(sd->dev, "going inactive\n");
+			vidsw->active = -1;
+		}
+		return 0;
+	}
+
+	if (vidsw->active >= 0) {
+		struct media_pad *pad;
+
+		if (vidsw->active == local->index)
+			return 0;
+
+		pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+		if (pad) {
+			struct media_link *link;
+			int ret;
+
+			link = media_entity_find_link(pad,
+						&vidsw->pads[vidsw->active]);
+			if (link) {
+				ret = __media_entity_setup_link(link, 0);
+				if (ret)
+					return ret;
+			}
+		}
+	}
+
+	vidsw_set_active(vidsw, local->index);
+
+	return 0;
+}
+
+static struct media_entity_operations vidsw_ops = {
+	.link_setup = vidsw_link_setup,
+};
+
+static bool vidsw_endpoint_disabled(struct device_node *ep)
+{
+	struct device_node *rpp;
+
+	if (!of_device_is_available(ep))
+		return true;
+
+	rpp = of_graph_get_remote_port_parent(ep);
+	if (!rpp)
+		return true;
+
+	return !of_device_is_available(rpp);
+}
+
+static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
+{
+	struct device_node *ep;
+	u32 portno;
+	int numports;
+	int ret;
+	int i;
+	bool active_link = false;
+
+	numports = vidsw->num_pads;
+
+	for (i = 0; i < numports - 1; i++)
+		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
+	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
+
+	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
+				     vidsw->pads);
+	if (ret < 0)
+		return ret;
+
+	vidsw->subdev.entity.ops = &vidsw_ops;
+
+	for_each_endpoint_of_node(node, ep) {
+		struct v4l2_of_endpoint endpoint;
+
+		v4l2_of_parse_endpoint(ep, &endpoint);
+
+		portno = endpoint.base.port;
+		if (portno >= numports - 1)
+			continue;
+
+		if (vidsw_endpoint_disabled(ep)) {
+			dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
+			continue;
+		}
+
+		vidsw->endpoint[portno] = endpoint;
+
+		if (portno == vidsw->active)
+			active_link = true;
+	}
+
+	for (portno = 0; portno < numports - 1; portno++) {
+		if (!vidsw->endpoint[portno].base.local_node)
+			continue;
+
+		/* If the active input is not connected, use another */
+		if (!active_link) {
+			vidsw_set_active(vidsw, portno);
+			active_link = true;
+		}
+	}
+
+	return v4l2_async_register_subdev(&vidsw->subdev);
+}
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct media_pad *pad;
+	int ret;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "no configuration for inactive mux\n");
+		return -EINVAL;
+	}
+
+	/*
+	 * Retrieve media bus configuration from the entity connected to the
+	 * active input
+	 */
+	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+	if (pad) {
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
+		if (ret == -ENOIOCTLCMD)
+			pad = NULL;
+		else if (ret < 0) {
+			dev_err(sd->dev, "failed to get source configuration\n");
+			return ret;
+		}
+	}
+	if (!pad) {
+		/* Mirror the input side on the output side */
+		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
+		if (cfg->type == V4L2_MBUS_PARALLEL ||
+		    cfg->type == V4L2_MBUS_BT656)
+			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
+	}
+
+	return 0;
+}
+
+static int vidsw_g_frame_interval(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_frame_interval *fi)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	fi->interval = vidsw->timeperframe;
+
+	return 0;
+}
+
+static int vidsw_s_frame_interval(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_frame_interval *fi)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	vidsw->timeperframe = fi->interval;
+
+	return 0;
+}
+
+static const struct v4l2_subdev_video_ops vidsw_subdev_video_ops = {
+	.g_mbus_config = vidsw_g_mbus_config,
+	.g_frame_interval = vidsw_g_frame_interval,
+	.s_frame_interval = vidsw_s_frame_interval,
+};
+
+static struct v4l2_mbus_framefmt *
+__vidsw_get_pad_format(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       unsigned int pad, u32 which)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	switch (which) {
+	case V4L2_SUBDEV_FORMAT_TRY:
+		return v4l2_subdev_get_try_format(sd, cfg, pad);
+	case V4L2_SUBDEV_FORMAT_ACTIVE:
+		return &vidsw->format_mbus[pad];
+	default:
+		return NULL;
+	}
+}
+
+static int vidsw_get_format(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	sdformat->format = *__vidsw_get_pad_format(sd, cfg, sdformat->pad,
+						   sdformat->which);
+	return 0;
+}
+
+static int vidsw_set_format(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct v4l2_mbus_framefmt *mbusformat;
+
+	if (sdformat->pad >= vidsw->num_pads)
+		return -EINVAL;
+
+	mbusformat = __vidsw_get_pad_format(sd, cfg, sdformat->pad,
+					    sdformat->which);
+	if (!mbusformat)
+		return -EINVAL;
+
+	/* Output pad mirrors active input pad, no limitations on input pads */
+	if (sdformat->pad == (vidsw->num_pads - 1) && vidsw->active >= 0)
+		sdformat->format = vidsw->format_mbus[vidsw->active];
+
+	*mbusformat = sdformat->format;
+
+	return 0;
+}
+
+static struct v4l2_subdev_pad_ops vidsw_pad_ops = {
+	.get_fmt = vidsw_get_format,
+	.set_fmt = vidsw_set_format,
+};
+
+static struct v4l2_subdev_ops vidsw_subdev_ops = {
+	.pad = &vidsw_pad_ops,
+	.video = &vidsw_subdev_video_ops,
+};
+
+static struct v4l2_subdev_internal_ops vidsw_internal_ops = {
+	.registered = v4l2_of_subdev_registered,
+};
+
+static int of_get_reg_field(struct device_node *node, struct reg_field *field)
+{
+	u32 bit_mask;
+	int ret;
+
+	ret = of_property_read_u32(node, "reg", &field->reg);
+	if (ret < 0)
+		return ret;
+
+	ret = of_property_read_u32(node, "bit-mask", &bit_mask);
+	if (ret < 0)
+		return ret;
+
+	ret = of_property_read_u32(node, "bit-shift", &field->lsb);
+	if (ret < 0)
+		return ret;
+
+	field->msb = field->lsb + fls(bit_mask) - 1;
+
+	return 0;
+}
+
+static int vidsw_probe(struct platform_device *pdev)
+{
+	struct device_node *np = pdev->dev.of_node;
+	struct of_endpoint endpoint;
+	struct device_node *ep;
+	struct reg_field field;
+	struct vidsw *vidsw;
+	struct regmap *map;
+	unsigned int num_pads;
+	int ret;
+
+	vidsw = devm_kzalloc(&pdev->dev, sizeof(*vidsw), GFP_KERNEL);
+	if (!vidsw)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, vidsw);
+
+	v4l2_subdev_init(&vidsw->subdev, &vidsw_subdev_ops);
+	vidsw->subdev.internal_ops = &vidsw_internal_ops;
+	snprintf(vidsw->subdev.name, sizeof(vidsw->subdev.name), "%s",
+			np->name);
+	vidsw->subdev.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	vidsw->subdev.dev = &pdev->dev;
+
+	/*
+	 * The largest numbered port is the output port. It determines
+	 * total number of pads
+	 */
+	for_each_endpoint_of_node(np, ep) {
+		of_graph_parse_endpoint(ep, &endpoint);
+		num_pads = max(num_pads, endpoint.port + 1);
+	}
+
+	num_pads = of_get_child_count(np);
+	if (num_pads < 2) {
+		dev_err(&pdev->dev, "Not enough ports %d\n", num_pads);
+		return -EINVAL;
+	}
+
+	ret = of_get_reg_field(np, &field);
+	if (ret == 0) {
+		map = syscon_node_to_regmap(np->parent);
+		if (!map) {
+			dev_err(&pdev->dev, "Failed to get syscon register map\n");
+			return PTR_ERR(map);
+		}
+
+		vidsw->field = devm_regmap_field_alloc(&pdev->dev, map, field);
+		if (IS_ERR(vidsw->field)) {
+			dev_err(&pdev->dev, "Failed to allocate regmap field\n");
+			return PTR_ERR(vidsw->field);
+		}
+
+		regmap_field_read(vidsw->field, &vidsw->active);
+	} else {
+		if (num_pads > 3) {
+			dev_err(&pdev->dev, "Too many ports %d\n", num_pads);
+			return -EINVAL;
+		}
+
+		vidsw->gpio = devm_gpiod_get(&pdev->dev, NULL, GPIOD_OUT_LOW);
+		if (IS_ERR(vidsw->gpio)) {
+			dev_warn(&pdev->dev,
+				 "could not request control gpio: %d\n", ret);
+			vidsw->gpio = NULL;
+		}
+
+		vidsw->active = gpiod_get_value(vidsw->gpio) ? 1 : 0;
+	}
+
+	vidsw->num_pads = num_pads;
+	vidsw->pads = devm_kzalloc(&pdev->dev, sizeof(*vidsw->pads) * num_pads,
+			GFP_KERNEL);
+	vidsw->format_mbus = devm_kzalloc(&pdev->dev,
+			sizeof(*vidsw->format_mbus) * num_pads, GFP_KERNEL);
+	vidsw->endpoint = devm_kzalloc(&pdev->dev,
+			sizeof(*vidsw->endpoint) * (num_pads - 1), GFP_KERNEL);
+
+	ret = vidsw_async_init(vidsw, np);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int vidsw_remove(struct platform_device *pdev)
+{
+	struct vidsw *vidsw = platform_get_drvdata(pdev);
+
+	v4l2_async_register_subdev(&vidsw->subdev);
+
+	return 0;
+}
+
+static const struct of_device_id vidsw_dt_ids[] = {
+	{ .compatible = "video-multiplexer", },
+	{ /* sentinel */ }
+};
+
+static struct platform_driver vidsw_driver = {
+	.probe		= vidsw_probe,
+	.remove		= vidsw_remove,
+	.driver		= {
+		.of_match_table = vidsw_dt_ids,
+		.name = "video-multiplexer",
+	},
+};
+
+module_platform_driver(vidsw_driver);
+
+MODULE_DESCRIPTION("video stream multiplexer");
+MODULE_AUTHOR("Sascha Hauer, Pengutronix");
+MODULE_AUTHOR("Philipp Zabel, Pengutronix");
+MODULE_LICENSE("GPL");
-- 
2.9.3

[PATCH 11/22] [media] imx: Add i.MX MIPI CSI-2 subdevice driver

[Philipp Zabel]

Add a v4l2 subdevice driver for the Synopsys DesignWare MIPI CSI-2 host
controller on i.MX6. Here its output is connected to the CSI2IPU gasket,
which distributes the time division multiplexed virtual channels to the
four IPU CSI inputs, directly or through video bus multiplexers.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 .../devicetree/bindings/media/fsl-imx-capture.txt  |  67 ++
 drivers/media/platform/imx/Kconfig                 |   7 +
 drivers/media/platform/imx/Makefile                |   1 +
 drivers/media/platform/imx/imx-mipi-csi2.c         | 676 +++++++++++++++++++++
 4 files changed, 751 insertions(+)
 create mode 100644 drivers/media/platform/imx/imx-mipi-csi2.c

diff --git a/Documentation/devicetree/bindings/media/fsl-imx-capture.txt b/Documentation/devicetree/bindings/media/fsl-imx-capture.txt
index 5805331..b5ef101 100644
--- a/Documentation/devicetree/bindings/media/fsl-imx-capture.txt
+++ b/Documentation/devicetree/bindings/media/fsl-imx-capture.txt
@@ -23,3 +23,70 @@ capture-subsystem {
         compatible = "fsl,imx-capture-subsystem";
         ports = <&ipu1_csi0>, <&ipu1_csi1>, <&ipu2_csi0>, <&ipu2_csi1>;
 };
+
+i.MX MIPI CSI-2 Host Controller
+===============================
+
+The i.MX6 contains an implementation of a Synopsys DesignWare MIPI CSI-2 host
+controller combined with a CSI2IPU gasket that distributes the virtual channels
+to up to four IPUv3 CSIs.
+
+Required properties:
+- compatible: should be "fsl,imx6q-mipi-csi2", "snps,dw-mipi-csi2"
+- reg: should be register base and length as documented in the SoC
+  reference manual
+- clocks: should contain the pclk, cfg, ref, and pixel clocks, in the
+  order determined by the clock-names property.
+- clock-names: should be "pclk", "cfg", "ref", "pixel"
+- address-cells: should be 1
+- size-cells: should be 0
+- port@*: five port nodes using of_graph bindings, one input port and four
+  output ports corresponding to the virtual channels.
+
+Example:
+
+mipi_csi: mipi@021dc000 {
+	compatible = "fsl,imx6q-mipi-csi2", "snps,dw-mipi-csi2";
+	reg = <0x021dc000 0x4000>;
+	clocks = <&clks IMX6QDL_CLK_HSI_TX>, <&clks IMX6QDL_CLK_HSI_TX>,
+		 <&clks IMX6QDL_CLK_HSI_TX>, <&clks IMX6QDL_CLK_EIM_PODF>;
+	clock-names = "pclk", "cfg", "ref", "pixel";
+	#address-cells = <1>;
+	#size-cells = <0>;
+
+	port@0 {
+		reg = <0>;
+	};
+
+	port@1 {
+		reg = <1>;
+
+		vc0: endpoint {
+			remote-endpoint = <&csi0_in>;
+		};
+	};
+
+	port@2 {
+		reg = <2>;
+
+		vc1: endpoint {
+			remote-endpoint = <&csi1_in>;
+		};
+	};
+
+	port@3 {
+		reg = <3>;
+
+		vc2: endpoint {
+			remote-endpoint = <&csi2_in>;
+		};
+	};
+
+	port@4 {
+		reg = <4>;
+
+		vc3: endpoint {
+			remote-endpoint = <&csi3_in>;
+		};
+	};
+};
diff --git a/drivers/media/platform/imx/Kconfig b/drivers/media/platform/imx/Kconfig
index 69e8648..db04e64 100644
--- a/drivers/media/platform/imx/Kconfig
+++ b/drivers/media/platform/imx/Kconfig
@@ -6,6 +6,13 @@ config MEDIA_IMX
 	  This driver provides a SoC wide media controller device that all
 	  multimedia components in i.MX5 and i.MX6 SoCs can register with.
 
+config MEDIA_IMX_MIPI_CSI2
+	tristate "i.MX MIPI CSI-2 Host Controller"
+	depends on VIDEO_V4L2_SUBDEV_API
+	help
+	  This driver provides support for the MIPI CSI-2 Host Controller that
+	  is connected to the IPU CSI input multiplexers on i.MX6 SoCs.
+
 config VIDEO_IMX_IPU_COMMON
 	tristate
 
diff --git a/drivers/media/platform/imx/Makefile b/drivers/media/platform/imx/Makefile
index 919eaa1..22d86fd 100644
--- a/drivers/media/platform/imx/Makefile
+++ b/drivers/media/platform/imx/Makefile
@@ -1,4 +1,5 @@
 obj-$(CONFIG_MEDIA_IMX)			+= imx-media.o
+obj-$(CONFIG_MEDIA_IMX_MIPI_CSI2)	+= imx-mipi-csi2.o
 obj-$(CONFIG_VIDEO_IMX_IPU_COMMON)	+= imx-ipu.o
 obj-$(CONFIG_VIDEO_IMX_IPU_CAPTURE)	+= imx-ipu-capture.o
 obj-$(CONFIG_VIDEO_IMX_IPU_CSI)		+= imx-ipuv3-csi.o
diff --git a/drivers/media/platform/imx/imx-mipi-csi2.c b/drivers/media/platform/imx/imx-mipi-csi2.c
new file mode 100644
index 0000000..0613187
--- /dev/null
+++ b/drivers/media/platform/imx/imx-mipi-csi2.c
@@ -0,0 +1,676 @@
+/*
+ * i.MX MIPI CSI-2 Host Controller driver
+ *
+ * Copyright (C) 2016, Pengutronix, Philipp Zabel <kernel@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+
+#define MIPI_CSI2_VERSION	0x00
+#define MIPI_CSI2_N_LANES	0x04
+#define MIPI_CSI2_PHY_SHUTDOWNZ	0x08
+#define MIPI_CSI2_DPHY_RSTZ	0x0c
+#define MIPI_CSI2_CSI2_RESETN	0x10
+#define MIPI_CSI2_PHY_STATE	0x14
+#define MIPI_CSI2_DATA_IDS_1	0x18
+#define MIPI_CSI2_DATA_IDS_2	0x1c
+#define MIPI_CSI2_ERR1		0x20
+#define MIPI_CSI2_ERR2		0x24
+#define MIPI_CSI2_MASK1		0x28
+#define MIPI_CSI2_MASK2		0x2c
+#define MIPI_CSI2_PHY_TST_CTRL0	0x30
+#define MIPI_CSI2_PHY_TST_CTRL1	0x34
+
+#define CSI2IPU_SW_RST		0xf00
+
+#define PHY_RXULPSESC_0		BIT(0)
+#define PHY_RXULPSESC_1		BIT(1)
+#define PHY_RXULPSESC_2		BIT(2)
+#define PHY_RXULPSESC_3		BIT(3)
+#define PHY_STOPSTATEDATA_0	BIT(4)
+#define PHY_STOPSTATEDATA_1	BIT(5)
+#define PHY_STOPSTATEDATA_2	BIT(6)
+#define PHY_STOPSTATEDATA_3	BIT(7)
+#define PHY_RXCLKACTIVEHS	BIT(8)
+#define PHY_RXULPSCLKNOT	BIT(9)
+#define PHY_STOPSTATECLK	BIT(10)
+#define PHY_BYPASS_2ECC_TST	BIT(11)
+
+#define PHY_TESTCLR		BIT(0)
+#define PHY_TESTCLK		BIT(1)
+
+#define PHY_TESTEN		BIT(16)
+
+#define MIPI_CSI2_VERSION_IMX6	0x3130302a
+
+#define CSI2IPU_RGB444_FM	BIT(3)
+#define CSI2IPU_YUV422_8BIT_FM	BIT(2)
+#define CSI2IPU_CLK_SEL		BIT(1)
+
+#define MIPI_CSI2_PADS 5
+
+struct mipi_csi2 {
+	struct v4l2_subdev subdev;
+	struct media_pad pads[MIPI_CSI2_PADS];
+	struct v4l2_mbus_framefmt format;
+	struct v4l2_fract timeperframe;
+	struct device *dev;
+	spinlock_t lock;
+
+	struct clk *pclk;
+	struct clk *cfg_clk;
+	struct clk *ref_clk;
+	struct clk *pixel_clk;
+	void __iomem *regs;
+
+	int max_lanes;
+	int lanes;
+	bool enabled;
+	bool streaming;
+};
+
+struct dphy_pll_testdin_map {
+	u16 max_mbps;
+	u16 testdin;
+};
+
+/* The table is based on 27MHz DPHY pll reference clock. */
+static const struct dphy_pll_testdin_map dptdin_map[] = {
+	{ 90, 0x00}, {100, 0x20}, {110, 0x40}, {125, 0x02},
+	{140, 0x22}, {150, 0x42}, {160, 0x04}, {180, 0x24},
+	{200, 0x44}, {210, 0x06}, {240, 0x26}, {250, 0x46},
+	{270, 0x08}, {300, 0x28}, {330, 0x48}, {360, 0x2a},
+	{400, 0x4a}, {450, 0x0c}, {500, 0x2c}, {550, 0x0e},
+	{600, 0x2e}, {650, 0x10}, {700, 0x30}, {750, 0x12},
+	{800, 0x32}, {850, 0x14}, {900, 0x34}, {950, 0x54},
+	{1000, 0x74}
+};
+
+static int max_mbps_to_testdin(unsigned int max_mbps)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dptdin_map); i++)
+		if (dptdin_map[i].max_mbps == max_mbps)
+			return dptdin_map[i].testdin;
+
+	return -EINVAL;
+}
+
+static void mipi_csi2_dphy_write(struct mipi_csi2 *csi2, u8 addr, u8 data)
+{
+	/* Clear PHY test interface */
+	writel(PHY_TESTCLR, csi2->regs + MIPI_CSI2_PHY_TST_CTRL0);
+	writel(0x00000, csi2->regs + MIPI_CSI2_PHY_TST_CTRL1);
+	writel(0, csi2->regs + MIPI_CSI2_PHY_TST_CTRL0);
+
+	/* Raise test interface strobe signal */
+	writel(PHY_TESTCLK, csi2->regs + MIPI_CSI2_PHY_TST_CTRL0);
+
+	/* Configure address write on falling edge and lower strobe signal */
+	writel(PHY_TESTEN | addr, csi2->regs + MIPI_CSI2_PHY_TST_CTRL1);
+	writel(0, csi2->regs + MIPI_CSI2_PHY_TST_CTRL0);
+
+	/* Configure data write on rising edge and raise strobe signal */
+	writel(data, csi2->regs + MIPI_CSI2_PHY_TST_CTRL1);
+	writel(PHY_TESTCLK, csi2->regs + MIPI_CSI2_PHY_TST_CTRL0);
+
+	/* Clear strobe signal */
+	writel(0, csi2->regs + MIPI_CSI2_PHY_TST_CTRL0);
+}
+
+static int mipi_csi2_reset(struct mipi_csi2 *csi2)
+{
+	const u8 addr = 0x44;
+	u8 data = max_mbps_to_testdin(850);
+
+	/* Assert CSI-2 Host Controller and D-PHY resets */
+	writel(0, csi2->regs + MIPI_CSI2_PHY_SHUTDOWNZ);
+	writel(0, csi2->regs + MIPI_CSI2_DPHY_RSTZ);
+	writel(0, csi2->regs + MIPI_CSI2_CSI2_RESETN);
+
+	mipi_csi2_dphy_write(csi2, addr, data);
+
+	/* Deassert resets */
+	writel(1, csi2->regs + MIPI_CSI2_PHY_SHUTDOWNZ);
+	writel(1, csi2->regs + MIPI_CSI2_DPHY_RSTZ);
+	writel(1, csi2->regs + MIPI_CSI2_CSI2_RESETN);
+
+	return 0;
+}
+
+static int mipi_csi2_get_fmt(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_pad_config *cfg,
+			     struct v4l2_subdev_format *sdformat)
+{
+	struct mipi_csi2 *csi2 = container_of(sd, struct mipi_csi2, subdev);
+
+	dev_dbg(csi2->dev, "%s\n", __func__);
+
+	if (sdformat->pad > 4)
+		return -EINVAL;
+
+	sdformat->format = csi2->format;
+
+	return 0;
+}
+
+static unsigned int mipi_csi2_format_bpp(u32 mbus_code)
+{
+	switch (mbus_code) {
+	case MEDIA_BUS_FMT_RGB888_1X24:
+		return 24;
+	case MEDIA_BUS_FMT_UYVY8_1X16:
+	case MEDIA_BUS_FMT_YUYV8_1X16:
+		return 16;
+	default:
+		return 0;
+	}
+}
+
+static unsigned int mipi_csi2_lanes_needed(struct mipi_csi2 *csi2,
+					   struct v4l2_mbus_framefmt *format,
+					   struct v4l2_fract *timeperframe)
+{
+	unsigned int bps_per_lane = 594000000U; /* FIXME */
+	unsigned int bpp = mipi_csi2_format_bpp(format->code);
+	unsigned int bps;
+
+	bps = format->width * format->height * bpp;
+	bps = mult_frac(bps, timeperframe->denominator, timeperframe->numerator);
+	return DIV_ROUND_UP(bps, bps_per_lane);
+}
+
+static int mipi_csi2_set_fmt(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_pad_config *cfg,
+			     struct v4l2_subdev_format *sdformat)
+{
+	struct mipi_csi2 *csi2 = container_of(sd, struct mipi_csi2, subdev);
+	unsigned long flags;
+	u32 val;
+
+	dev_dbg(csi2->dev, "%s\n", __func__);
+
+	if (sdformat->pad > 4)
+		return -EINVAL;
+
+	switch (sdformat->format.code) {
+	case MEDIA_BUS_FMT_RGB888_1X24:
+	case MEDIA_BUS_FMT_UYVY8_1X16:
+	case MEDIA_BUS_FMT_YUYV8_1X16:
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	csi2->format = sdformat->format;
+	csi2->lanes = mipi_csi2_lanes_needed(csi2, &csi2->format,
+					     &csi2->timeperframe);
+
+	spin_lock_irqsave(&csi2->lock, flags);
+	val = readl(csi2->regs + CSI2IPU_SW_RST);
+	switch (csi2->format.code) {
+	case MEDIA_BUS_FMT_UYVY8_1X16:
+		val &= ~CSI2IPU_YUV422_8BIT_FM;
+		break;
+	case MEDIA_BUS_FMT_YUYV8_1X16:
+		val |= CSI2IPU_YUV422_8BIT_FM;
+		break;
+	}
+	writel(val, csi2->regs + CSI2IPU_SW_RST);
+	spin_unlock_irqrestore(&csi2->lock, flags);
+
+	return 0;
+}
+
+static int mipi_csi2_g_frame_interval(struct v4l2_subdev *sd,
+				      struct v4l2_subdev_frame_interval *fi)
+{
+	struct mipi_csi2 *csi2 = container_of(sd, struct mipi_csi2, subdev);
+
+	dev_dbg(csi2->dev, "%s\n", __func__);
+
+	fi->interval = csi2->timeperframe;
+
+	return 0;
+}
+
+static int mipi_csi2_s_frame_interval(struct v4l2_subdev *sd,
+				      struct v4l2_subdev_frame_interval *fi)
+{
+	struct mipi_csi2 *csi2 = container_of(sd, struct mipi_csi2, subdev);
+
+	dev_dbg(csi2->dev, "%s(%d, %d/%d)\n", __func__, fi->pad,
+		fi->interval.numerator, fi->interval.denominator);
+
+	if (fi->pad > 4)
+		return -EINVAL;
+
+	if (fi->pad > 0) {
+		fi->interval = csi2->timeperframe;
+
+		return 0;
+	}
+
+	csi2->timeperframe = fi->interval;
+	csi2->lanes = mipi_csi2_lanes_needed(csi2, &csi2->format,
+					     &csi2->timeperframe);
+
+	return 0;
+}
+
+
+static const struct v4l2_subdev_pad_ops mipi_csi2_pad_ops = {
+	.get_fmt = mipi_csi2_get_fmt,
+	.set_fmt = mipi_csi2_set_fmt,
+};
+
+static int mipi_csi2_g_mbus_config(struct v4l2_subdev *sd,
+			    struct v4l2_mbus_config *cfg)
+{
+	cfg->type = V4L2_MBUS_CSI2;
+	cfg->flags = V4L2_MBUS_CSI2_1_LANE | V4L2_MBUS_CSI2_2_LANE |
+		     V4L2_MBUS_CSI2_3_LANE | V4L2_MBUS_CSI2_4_LANE |
+		     V4L2_MBUS_CSI2_CHANNEL_0 | V4L2_MBUS_CSI2_CHANNEL_1 |
+		     V4L2_MBUS_CSI2_CHANNEL_2 | V4L2_MBUS_CSI2_CHANNEL_3 |
+		     V4L2_MBUS_CSI2_CONTINUOUS_CLOCK |
+		     V4L2_MBUS_CSI2_NONCONTINUOUS_CLOCK;
+
+	return 0;
+}
+
+static int mipi_csi2_enable(struct mipi_csi2 *csi2)
+{
+	if (csi2->enabled)
+		return -EBUSY;
+
+	clk_prepare_enable(csi2->pclk);
+	clk_prepare_enable(csi2->cfg_clk);
+	clk_prepare_enable(csi2->ref_clk);
+	csi2->enabled = true;
+
+	return 0;
+}
+
+static int mipi_csi2_disable(struct mipi_csi2 *csi2)
+{
+	if (!csi2->enabled)
+		return 0;
+
+	clk_disable_unprepare(csi2->ref_clk);
+	clk_disable_unprepare(csi2->cfg_clk);
+	clk_disable_unprepare(csi2->pclk);
+	csi2->enabled = false;
+
+	return 0;
+}
+
+static int mipi_csi2_wait(struct mipi_csi2 *csi2, u8 reg, u32 mask, u32 val)
+{
+	unsigned long timeout = jiffies + msecs_to_jiffies(100);
+	u32 state;
+
+	dev_dbg(csi2->dev, "wait for register 0x%02x & 0x%08x to be 0x%08x\n",
+		reg, mask, val);
+
+	for (;;) {
+		state = readl(csi2->regs + reg);
+		dev_dbg(csi2->dev, "register 0x%02x == 0x%08x\n", reg, state);
+		if ((state & mask) == val)
+			return 0;
+		if (time_after(jiffies, timeout))
+			return -ETIMEDOUT;
+		usleep_range(500, 2000);
+	}
+}
+
+static inline unsigned int mipi_csi2_stop_state(unsigned int lanes)
+{
+	switch (lanes) {
+	case 1:
+		return PHY_STOPSTATEDATA_0 |
+		       PHY_STOPSTATECLK;
+	case 2:
+		return PHY_STOPSTATEDATA_0 | PHY_STOPSTATEDATA_1 |
+		       PHY_STOPSTATECLK;
+	case 3:
+		return PHY_STOPSTATEDATA_0 | PHY_STOPSTATEDATA_1 |
+		       PHY_STOPSTATEDATA_2 |
+		       PHY_STOPSTATECLK;
+	case 4:
+		return PHY_STOPSTATEDATA_0 | PHY_STOPSTATEDATA_1 |
+		       PHY_STOPSTATEDATA_2 | PHY_STOPSTATEDATA_3 |
+		       PHY_STOPSTATECLK;
+	default:
+		return 0;
+	}
+}
+
+/**
+ * mipi_csi2_prepare_stream - prepare for streaming
+ *
+ * The sender should be configured to put all lanes in PL-11 state (STOPSTATE)
+ * in its prepare_stream(), which must be called prior to this function.
+ * Afterwards, the sender shall start sending data in its s_stream().
+ */
+static int mipi_csi2_prepare_stream(struct v4l2_subdev *sd)
+{
+	struct mipi_csi2 *csi2 = container_of(sd, struct mipi_csi2, subdev);
+	unsigned int stop_state;
+	int ret;
+
+	if (csi2->lanes > csi2->max_lanes) {
+		dev_err(csi2->dev, "%d lanes needed for format, %d supported\n",
+			csi2->lanes, csi2->max_lanes);
+		return -EINVAL;
+	}
+
+	stop_state = mipi_csi2_stop_state(csi2->lanes);
+	if (!stop_state)
+		return -EINVAL;
+
+	if (csi2->enabled) {
+		dev_warn(csi2->dev, "Already enabled, disabling\n");
+		mipi_csi2_disable(csi2);
+		usleep_range(1000, 2000);
+	}
+
+	/* Enable clocks */
+	mipi_csi2_enable(csi2);
+
+	/* Set number of lanes */
+	writel(csi2->lanes - 1, csi2->regs + MIPI_CSI2_N_LANES);
+
+	/* Reset D-PHY and host controller */
+	mipi_csi2_reset(csi2);
+
+	/* Wait for STOP state on all used lanes */
+	ret = mipi_csi2_wait(csi2, MIPI_CSI2_PHY_STATE,
+			     stop_state, stop_state);
+	if (ret < 0) {
+		dev_err(csi2->dev,
+			"Timeout waiting for lanes to enter STOP state\n");
+	}
+
+	return ret;
+}
+
+/**
+ * mipi_csi2_s_stream - start streaming
+ *
+ * When enabling streaming, the active lanes must have been detected to be in
+ * stop state in prepare_stream(), and the sender should now be sending data.
+ */
+static int mipi_csi2_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct mipi_csi2 *csi2 = container_of(sd, struct mipi_csi2, subdev);
+	int ret;
+
+	if (enable) {
+		/* Wait for PHY lock */
+		ret = mipi_csi2_wait(csi2, MIPI_CSI2_PHY_STATE,
+				     PHY_RXCLKACTIVEHS, PHY_RXCLKACTIVEHS);
+		if (ret < 0) {
+			dev_err(csi2->dev, "Timeout waiting for PHY lock\n");
+			return ret;
+		}
+
+		/* Wait for error free transmission */
+		ret = mipi_csi2_wait(csi2, MIPI_CSI2_ERR1, ~0, 0);
+		if (ret < 0)
+			dev_err(csi2->dev, "Timeout waiting for ERR1 to clear\n");
+
+		clk_prepare_enable(csi2->pixel_clk);
+		csi2->streaming = true;
+	} else {
+		if (csi2->streaming) {
+			clk_disable_unprepare(csi2->pixel_clk);
+			csi2->streaming = false;
+		}
+		mipi_csi2_disable(csi2);
+	}
+
+	return 0;
+}
+
+#ifdef CONFIG_VIDEO_ADV_DEBUG
+static int mipi_csi2_log_status(struct v4l2_subdev *sd)
+{
+	struct mipi_csi2 *csi2 = container_of(sd, struct mipi_csi2, subdev);
+	u32 val;
+	int i;
+
+	v4l2_info(sd, "-----MIPI CSI-2 Host status-----\n");
+	for (i = MIPI_CSI2_VERSION; i < MIPI_CSI2_PHY_TST_CTRL0; i += 4) {
+		val = readl(csi2->regs + i);
+		v4l2_info(sd, "%02x: %08x\n", i, val);
+	}
+
+	v4l2_info(sd, "-----CSI2IPU gasket status------\n");
+	val = readl(csi2->regs + CSI2IPU_SW_RST);
+	v4l2_info(sd, "f00: %08x\n", val);
+
+	return 0;
+}
+
+static int mipi_csi2_g_register(struct v4l2_subdev *sd,
+				struct v4l2_dbg_register *reg)
+{
+	struct mipi_csi2 *csi2 = container_of(sd, struct mipi_csi2, subdev);
+
+	if ((reg->reg % 4) ||
+	    (reg->reg > MIPI_CSI2_PHY_TST_CTRL1 && reg->reg != CSI2IPU_SW_RST))
+		return -EINVAL;
+
+	reg->val = readl(csi2->regs + reg->reg);
+	reg->size = 4;
+
+	return 0;
+}
+
+static int mipi_csi2_s_register(struct v4l2_subdev *sd,
+				const struct v4l2_dbg_register *reg)
+{
+	struct mipi_csi2 *csi2 = container_of(sd, struct mipi_csi2, subdev);
+
+	if ((reg->reg % 4) ||
+	    (reg->reg == MIPI_CSI2_VERSION) ||
+	    (reg->reg > MIPI_CSI2_PHY_TST_CTRL1 && reg->reg != CSI2IPU_SW_RST))
+		return -EINVAL;
+
+	writel(reg->val, csi2->regs + reg->reg);
+
+	return 0;
+}
+#endif
+
+static const struct v4l2_subdev_core_ops mipi_csi2_core_ops = {
+#ifdef CONFIG_VIDEO_ADV_DEBUG
+	.log_status = mipi_csi2_log_status,
+	.g_register = mipi_csi2_g_register,
+	.s_register = mipi_csi2_s_register,
+#endif
+};
+
+static const struct v4l2_subdev_video_ops mipi_csi2_video_ops = {
+	.g_frame_interval = mipi_csi2_g_frame_interval,
+	.s_frame_interval = mipi_csi2_s_frame_interval,
+	.g_mbus_config = mipi_csi2_g_mbus_config,
+	.prepare_stream = mipi_csi2_prepare_stream,
+	.s_stream = mipi_csi2_s_stream,
+};
+
+static const struct v4l2_subdev_internal_ops mipi_csi2_internal_ops = {
+	.registered = v4l2_of_subdev_registered,
+};
+
+static struct v4l2_subdev_ops mipi_csi2_subdev_ops = {
+	.core = &mipi_csi2_core_ops,
+	.pad = &mipi_csi2_pad_ops,
+	.video = &mipi_csi2_video_ops,
+};
+
+static int mipi_csi2_link_setup(struct media_entity *entity,
+				const struct media_pad *local,
+				const struct media_pad *remote, u32 flags)
+{
+	return 0;
+}
+
+static struct media_entity_operations mipi_csi2_entity_ops = {
+	.link_setup = mipi_csi2_link_setup,
+};
+
+static int mipi_csi2_probe(struct platform_device *pdev)
+{
+	struct resource *res;
+	struct mipi_csi2 *csi2;
+	struct device_node *ep;
+	unsigned int version;
+	int ret;
+	u32 val;
+
+	csi2 = devm_kzalloc(&pdev->dev, sizeof(*csi2), GFP_KERNEL);
+	if (!csi2)
+		return -ENOMEM;
+
+	spin_lock_init(&csi2->lock);
+
+	csi2->dev = &pdev->dev;
+	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+	csi2->regs = devm_ioremap_resource(csi2->dev, res);
+	if (IS_ERR(csi2->regs))
+		return PTR_ERR(csi2->regs);
+
+	csi2->pclk = devm_clk_get(csi2->dev, "pclk");
+	if (IS_ERR(csi2->pclk))
+		return PTR_ERR(csi2->pclk);
+
+	csi2->cfg_clk = devm_clk_get(csi2->dev, "cfg");
+	if (IS_ERR(csi2->cfg_clk))
+		return PTR_ERR(csi2->cfg_clk);
+
+	csi2->ref_clk = devm_clk_get(csi2->dev, "ref");
+	if (IS_ERR(csi2->ref_clk))
+		return PTR_ERR(csi2->ref_clk);
+
+	csi2->pixel_clk = devm_clk_get(csi2->dev, "pixel");
+	if (IS_ERR(csi2->pixel_clk))
+		return PTR_ERR(csi2->pixel_clk);
+
+	platform_set_drvdata(pdev, csi2);
+
+	v4l2_subdev_init(&csi2->subdev, &mipi_csi2_subdev_ops);
+	csi2->subdev.internal_ops = &mipi_csi2_internal_ops;
+	snprintf(csi2->subdev.name, sizeof(csi2->subdev.name), "mipi-csi2");
+	csi2->subdev.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	csi2->subdev.dev = &pdev->dev;
+
+	csi2->pads[0].flags = MEDIA_PAD_FL_SINK;
+	csi2->pads[1].flags = MEDIA_PAD_FL_SOURCE;
+	csi2->pads[2].flags = MEDIA_PAD_FL_SOURCE;
+	csi2->pads[3].flags = MEDIA_PAD_FL_SOURCE;
+	csi2->pads[4].flags = MEDIA_PAD_FL_SOURCE;
+	ret = media_entity_pads_init(&csi2->subdev.entity, MIPI_CSI2_PADS,
+				csi2->pads);
+	if (ret < 0)
+		return ret;
+
+	csi2->subdev.entity.ops = &mipi_csi2_entity_ops;
+
+	clk_prepare_enable(csi2->pclk);
+
+	version = readl(csi2->regs + MIPI_CSI2_VERSION);
+	if (version != MIPI_CSI2_VERSION_IMX6) {
+		dev_err(csi2->dev, "Unknown version: 0x%x\n", version);
+		return -ENODEV;
+	}
+
+	/* Maximal number of data lanes */
+	csi2->max_lanes = readl(csi2->regs + MIPI_CSI2_N_LANES) + 1;
+
+	ret = v4l2_async_register_subdev(&csi2->subdev);
+	if (ret < 0)
+		return ret;
+
+	for_each_endpoint_of_node(pdev->dev.of_node, ep) {
+		struct v4l2_of_endpoint endpoint;
+
+		v4l2_of_parse_endpoint(ep, &endpoint);
+		if (endpoint.base.port != 0)
+			continue;
+
+		if (endpoint.bus_type != V4L2_MBUS_CSI2) {
+			dev_err(csi2->dev, "missing MIPI CSI-2 endpoint\n");
+			return -EINVAL;
+		}
+
+		if (csi2->max_lanes < endpoint.bus.mipi_csi2.num_data_lanes) {
+			dev_warn(csi2->dev, "only %d data lanes supported\n",
+				 csi2->max_lanes);
+		}
+
+		if (endpoint.bus.mipi_csi2.num_data_lanes &&
+		    endpoint.bus.mipi_csi2.num_data_lanes < csi2->max_lanes) {
+			csi2->max_lanes = endpoint.bus.mipi_csi2.num_data_lanes;
+			dev_dbg(csi2->dev, "only using %d data lanes\n",
+				csi2->max_lanes);
+		}
+	}
+
+	csi2->timeperframe.numerator = 1;
+	csi2->timeperframe.denominator = 60;
+	csi2->lanes = csi2->max_lanes;
+
+	/*
+	 * According to the i.MX6 Reference Manual, MIPI CSI-2 has to use
+	 * the 'non-gated' (noncontinuous) clock mode, still the CLK_SEL
+	 * bit in CSI2IPU_SW_RST needs to be cleared (marked as 'gated')
+	 */
+	val = readl(csi2->regs + CSI2IPU_SW_RST);
+	val &= ~CSI2IPU_CLK_SEL;
+	writel(val, csi2->regs + CSI2IPU_SW_RST);
+
+	return 0;
+}
+
+static int mipi_csi2_remove(struct platform_device *pdev)
+{
+	return 0;
+}
+
+static const struct of_device_id mipi_csi2_dt_ids[] = {
+	{ .compatible = "fsl,imx6q-mipi-csi2", },
+	{ }
+};
+MODULE_DEVICE_TABLE(of, mipi_csi2_dt_ids);
+
+static struct platform_driver mipi_csi2_driver = {
+	.probe = mipi_csi2_probe,
+	.remove = mipi_csi2_remove,
+	.driver = {
+		.of_match_table = mipi_csi2_dt_ids,
+		.name = "mipi-csi2",
+		.owner = THIS_MODULE,
+	},
+};
+
+module_platform_driver(mipi_csi2_driver);
+
+MODULE_DESCRIPTION("i.MX MIPI CSI-2 Host Controller driver");
+MODULE_AUTHOR("Philipp Zabel, Pengutronix");
+MODULE_LICENSE("GPL");
-- 
2.9.3

[PATCH 12/22] [media] tc358743: put lanes in STOP state before starting streaming

[Philipp Zabel]

Without calling tc358743_set_csi from the new prepare_stream callback
(or calling tc358743_s_dv_timings or tc358743_set_fmt from userspace
after stopping the stream), the i.MX6 MIPI CSI2 input fails waiting
for lanes to enter STOP state when streaming is started again.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/media/i2c/tc358743.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/media/i2c/tc358743.c b/drivers/media/i2c/tc358743.c
index 1e3a0dd2..dfa45d2 100644
--- a/drivers/media/i2c/tc358743.c
+++ b/drivers/media/i2c/tc358743.c
@@ -1463,6 +1463,14 @@ static int tc358743_g_mbus_config(struct v4l2_subdev *sd,
 	return 0;
 }
 
+static int tc358743_prepare_stream(struct v4l2_subdev *sd)
+{
+	/* Put all lanes in PL-11 state (STOPSTATE) */
+	tc358743_set_csi(sd);
+
+	return 0;
+}
+
 static int tc358743_s_stream(struct v4l2_subdev *sd, int enable)
 {
 	enable_stream(sd, enable);
@@ -1637,6 +1645,7 @@ static const struct v4l2_subdev_video_ops tc358743_video_ops = {
 	.g_dv_timings = tc358743_g_dv_timings,
 	.query_dv_timings = tc358743_query_dv_timings,
 	.g_mbus_config = tc358743_g_mbus_config,
+	.prepare_stream = tc358743_prepare_stream,
 	.s_stream = tc358743_s_stream,
 };
 
-- 
2.9.3

[PATCH 13/22] ARM: dts: imx6qdl: Add capture-subsystem node

[Philipp Zabel]

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 arch/arm/boot/dts/imx6dl.dtsi | 5 +++++
 arch/arm/boot/dts/imx6q.dtsi  | 5 +++++
 2 files changed, 10 insertions(+)

diff --git a/arch/arm/boot/dts/imx6dl.dtsi b/arch/arm/boot/dts/imx6dl.dtsi
index 9a4c22c..3c817de 100644
--- a/arch/arm/boot/dts/imx6dl.dtsi
+++ b/arch/arm/boot/dts/imx6dl.dtsi
@@ -100,6 +100,11 @@
 		};
 	};
 
+	capture-subsystem {
+		compatible = "fsl,imx-capture-subsystem";
+		ports = <&ipu1_csi0>, <&ipu1_csi1>;
+	};
+
 	display-subsystem {
 		compatible = "fsl,imx-display-subsystem";
 		ports = <&ipu1_di0>, <&ipu1_di1>;
diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index c30c836..0c87a69 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi
@@ -198,6 +198,11 @@
 		};
 	};
 
+	capture-subsystem {
+		compatible = "fsl,imx-capture-subsystem";
+		ports = <&ipu1_csi0>, <&ipu1_csi1>, <&ipu2_csi0>, <&ipu2_csi1>;
+	};
+
 	display-subsystem {
 		compatible = "fsl,imx-display-subsystem";
 		ports = <&ipu1_di0>, <&ipu1_di1>, <&ipu2_di0>, <&ipu2_di1>;
-- 
2.9.3

[PATCH 14/22] ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their connections

[Philipp Zabel]

This patch adds the device tree graph connecting the input multiplexers
to the IPU CSIs and the MIPI-CSI2 gasket on i.MX6. The MIPI_IPU
multiplexers are added as children of the iomuxc-gpr syscon device node.
On i.MX6Q/D two two-input multiplexers in front of IPU1 CSI0 and IPU2
CSI1 allow to select between CSI0/1 parallel input pads and the MIPI
CSI-2 virtual channels 0/3.
On i.MX6DL/S two five-input multiplexers in front of IPU1 CSI0 and IPU1
CSI1 allow to select between CSI0/1 parallel input pads and any of the
four MIPI CSI-2 virtual channels.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 arch/arm/boot/dts/imx6dl.dtsi  | 182 +++++++++++++++++++++++++++++++++++++++++
 arch/arm/boot/dts/imx6q.dtsi   | 118 ++++++++++++++++++++++++++
 arch/arm/boot/dts/imx6qdl.dtsi |  10 ++-
 3 files changed, 309 insertions(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/imx6dl.dtsi b/arch/arm/boot/dts/imx6dl.dtsi
index 3c817de..7ed4efd6f 100644
--- a/arch/arm/boot/dts/imx6dl.dtsi
+++ b/arch/arm/boot/dts/imx6dl.dtsi
@@ -133,6 +133,188 @@
 		      "di0", "di1";
 };
 
+&gpr {
+	ipu_csi0_mux {
+		compatible = "video-multiplexer";
+		reg = <0x34>;
+		bit-mask = <0x7>;
+		bit-shift = <0>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port@0 {
+			reg = <0>;
+
+			ipu_csi0_mux_from_mipi_csi0: endpoint {
+				remote-endpoint = <&mipi_csi0_to_ipu_csi0_mux>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			ipu_csi0_mux_from_mipi_csi1: endpoint {
+				remote-endpoint = <&mipi_csi1_to_ipu_csi0_mux>;
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			ipu_csi0_mux_from_mipi_csi2: endpoint {
+				remote-endpoint = <&mipi_csi2_to_ipu_csi0_mux>;
+			};
+		};
+
+		port@3 {
+			reg = <3>;
+
+			ipu_csi0_mux_from_mipi_csi3: endpoint {
+				remote-endpoint = <&mipi_csi3_to_ipu_csi0_mux>;
+			};
+		};
+
+		csi0: port@4 {
+			reg = <4>;
+		};
+
+		port@5 {
+			reg = <5>;
+
+			ipu_csi0_mux_to_ipu1_csi0: endpoint {
+				remote-endpoint = <&ipu1_csi0_from_ipu_csi0_mux>;
+			};
+		};
+	};
+
+	ipu_csi1_mux {
+		compatible = "video-multiplexer";
+		reg = <0x34>;
+		bit-mask = <0x7>;
+		bit-shift = <3>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port@0 {
+			reg = <0>;
+
+			ipu_csi1_mux_from_mipi_csi0: endpoint {
+				remote-endpoint = <&mipi_csi0_to_ipu_csi1_mux>;
+			};
+		};
+
+		port@1 {
+			reg = <1>;
+
+			ipu_csi1_mux_from_mipi_csi1: endpoint {
+				remote-endpoint = <&mipi_csi1_to_ipu_csi1_mux>;
+			};
+		};
+
+		port@2 {
+			reg = <2>;
+
+			ipu_csi1_mux_from_mipi_csi2: endpoint {
+				remote-endpoint = <&mipi_csi2_to_ipu_csi1_mux>;
+			};
+		};
+
+		port@3 {
+			reg = <3>;
+
+			ipu_csi1_mux_from_mipi_csi3: endpoint {
+				remote-endpoint = <&mipi_csi3_to_ipu_csi1_mux>;
+			};
+		};
+
+		csi1: port@4 {
+			reg = <4>;
+		};
+
+		port@5 {
+			reg = <5>;
+
+			ipu_csi1_mux_to_ipu1_csi1: endpoint {
+				remote-endpoint = <&ipu1_csi1_from_ipu_csi1_mux>;
+			};
+		};
+	};
+};
+
+&ipu1_csi0 {
+	ipu1_csi0_from_ipu_csi0_mux: endpoint {
+		remote-endpoint = <&ipu_csi0_mux_to_ipu1_csi0>;
+	};
+};
+
+&ipu1_csi1 {
+	ipu1_csi1_from_ipu_csi1_mux: endpoint {
+		remote-endpoint = <&ipu_csi1_mux_to_ipu1_csi1>;
+	};
+};
+
+&mipi_csi {
+	port@0 {
+		reg = <0>;
+	};
+
+	port@1 {
+		reg = <1>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_csi0_to_ipu_csi0_mux: endpoint@0 {
+			remote-endpoint = <&ipu_csi0_mux_from_mipi_csi0>;
+		};
+
+		mipi_csi0_to_ipu_csi1_mux: endpoint@1 {
+			remote-endpoint = <&ipu_csi1_mux_from_mipi_csi0>;
+		};
+	};
+
+	port@2 {
+		reg = <2>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_csi1_to_ipu_csi0_mux: endpoint@0 {
+			remote-endpoint = <&ipu_csi0_mux_from_mipi_csi1>;
+		};
+
+		mipi_csi1_to_ipu_csi1_mux: endpoint@1 {
+			remote-endpoint = <&ipu_csi1_mux_from_mipi_csi1>;
+		};
+	};
+
+	port@3 {
+		reg = <3>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_csi2_to_ipu_csi0_mux: endpoint@0 {
+			remote-endpoint = <&ipu_csi0_mux_from_mipi_csi2>;
+		};
+
+		mipi_csi2_to_ipu_csi1_mux: endpoint@1 {
+			remote-endpoint = <&ipu_csi1_mux_from_mipi_csi2>;
+		};
+	};
+
+	port@4 {
+		reg = <4>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_csi3_to_ipu_csi0_mux: endpoint@0 {
+			remote-endpoint = <&ipu_csi0_mux_from_mipi_csi3>;
+		};
+
+		mipi_csi3_to_ipu_csi1_mux: endpoint@1 {
+			remote-endpoint = <&ipu_csi1_mux_from_mipi_csi3>;
+		};
+	};
+};
+
 &vpu {
 	compatible = "fsl,imx6dl-vpu", "cnm,coda960";
 };
diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index 0c87a69..675723b 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi
@@ -143,10 +143,18 @@
 
 			ipu2_csi0: port@0 {
 				reg = <0>;
+
+				ipu2_csi0_from_csi2ipu: endpoint {
+					remote-endpoint = <&csi2ipu_to_ipu2_csi0>;
+				};
 			};
 
 			ipu2_csi1: port@1 {
 				reg = <1>;
+
+				ipu2_csi1_from_mipi_ipu2_mux: endpoint {
+					remote-endpoint = <&mipi_ipu2_mux_to_ipu2_csi1>;
+				};
 			};
 
 			ipu2_di0: port@2 {
@@ -234,6 +242,78 @@
 	};
 };
 
+&gpr {
+	mipi_ipu1_mux {
+		compatible = "video-multiplexer";
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <19>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port@0 {
+			reg = <0>;
+
+			mipi_ipu1_mux_from_mipi_csi0: endpoint {
+				remote-endpoint = <&mipi_csi0_to_mipi_ipu1_mux>;
+			};
+		};
+
+		csi0: port@1 {
+			reg = <1>;
+		};
+
+		port@2 {
+			reg = <2>;
+
+			mipi_ipu1_mux_to_ipu1_csi0: endpoint {
+				remote-endpoint = <&ipu1_csi0_from_mipi_ipu1_mux>;
+			};
+		};
+	};
+
+	mipi_ipu2_mux {
+		compatible = "video-multiplexer";
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <20>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port@0 {
+			reg = <0>;
+
+			mipi_ipu2_mux_from_mipi_csi3: endpoint {
+				remote-endpoint = <&mipi_csi3_to_mipi_ipu2_mux>;
+			};
+		};
+
+		csi1: port@1 {
+			reg = <1>;
+		};
+
+		port@2 {
+			reg = <2>;
+
+			mipi_ipu2_mux_to_ipu2_csi1: endpoint {
+				remote-endpoint = <&ipu2_csi1_from_mipi_ipu2_mux>;
+			};
+		};
+	};
+};
+
+&ipu1_csi1 {
+	ipu1_csi1_from_mipi_csi1: endpoint {
+		remote-endpoint = <&mipi_csi1_to_ipu1_csi1>;
+	};
+};
+
+&ipu2_csi0 {
+	ipu2_csi0_from_mipi_csi2: endpoint {
+		remote-endpoint = <&mipi_csi2_to_ipu2_csi0>;
+	};
+};
+
 &ldb {
 	clocks = <&clks IMX6QDL_CLK_LDB_DI0_SEL>, <&clks IMX6QDL_CLK_LDB_DI1_SEL>,
 		 <&clks IMX6QDL_CLK_IPU1_DI0_SEL>, <&clks IMX6QDL_CLK_IPU1_DI1_SEL>,
@@ -280,6 +360,44 @@
 	};
 };
 
+&mipi_csi {
+	port@0 {
+		reg = <0>;
+	};
+
+	port@1 {
+		reg = <1>;
+
+		mipi_csi0_to_mipi_ipu1_mux: endpoint {
+			remote-endpoint = <&mipi_ipu1_mux_from_mipi_csi0>;
+		};
+	};
+
+	port@2 {
+		reg = <2>;
+
+		mipi_csi1_to_ipu1_csi1: endpoint {
+			remote-endpoint = <&ipu1_csi1_from_mipi_csi1>;
+		};
+	};
+
+	port@3 {
+		reg = <3>;
+
+		mipi_csi2_to_ipu2_csi0: endpoint {
+			remote-endpoint = <&ipu2_csi0_from_mipi_csi2>;
+		};
+	};
+
+	port@4 {
+		reg = <4>;
+
+		mipi_csi3_to_mipi_ipu2_mux: endpoint {
+			remote-endpoint = <&mipi_ipu2_mux_from_mipi_csi3>;
+		};
+	};
+};
+
 &mipi_dsi {
 	ports {
 		port@2 {
diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index b13b0b2..cd325bd 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -798,8 +798,10 @@
 			};
 
 			gpr: iomuxc-gpr@020e0000 {
-				compatible = "fsl,imx6q-iomuxc-gpr", "syscon";
+				compatible = "fsl,imx6q-iomuxc-gpr", "syscon", "simple-mfd";
 				reg = <0x020e0000 0x38>;
+				#address-cells = <1>;
+				#size-cells = <0>;
 			};
 
 			iomuxc: iomuxc@020e0000 {
@@ -1122,6 +1124,8 @@
 
 			mipi_csi: mipi@021dc000 {
 				reg = <0x021dc000 0x4000>;
+				#address-cells = <1>;
+				#size-cells = <0>;
 			};
 
 			mipi_dsi: mipi@021e0000 {
@@ -1221,6 +1225,10 @@
 
 			ipu1_csi0: port@0 {
 				reg = <0>;
+
+				ipu1_csi0_from_mipi_ipu1_mux: endpoint {
+					remote-endpoint = <&mipi_ipu1_mux_to_ipu1_csi0>;
+				};
 			};
 
 			ipu1_csi1: port@1 {
-- 
2.9.3

[PATCH 15/22] ARM: dts: imx6qdl: Add MIPI CSI-2 D-PHY compatible and clocks

[Philipp Zabel]

>From the data sheets it is not quite clear what the clock inputs should
be named, but freescale code calls them "dphy_clk" (would that be per?)
and "pixel_clk" and connects them to the mipi_core_cfg and emi_podf
clocks, respectively.  The mipi_core_cfg control is called hsi_tx
currently, but it really gates a whole lot of other clocks, too.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index cd325bd..2be6de4 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -1123,9 +1123,16 @@
 			};
 
 			mipi_csi: mipi@021dc000 {
+				compatible = "fsl,imx6q-mipi-csi2", "dw-mipi-csi2";
 				reg = <0x021dc000 0x4000>;
+				clocks = <&clks IMX6QDL_CLK_HSI_TX>,	/* mipi_core_cfg/ipg_clk_root */
+					 <&clks IMX6QDL_CLK_HSI_TX>,	/* mipi_core_cfg/video_27m_clk_root */
+					 <&clks IMX6QDL_CLK_HSI_TX>,	/* mipi_core_cfg/video_27m_clk_root */
+					 <&clks IMX6QDL_CLK_EIM_PODF>;	/* shoid be ipu1_ipu_hsp_clk_root on S/DL, axi_clk_root on D/Q */
+				clock-names = "pclk", "cfg", "ref", "pixel";
 				#address-cells = <1>;
 				#size-cells = <0>;
+				status = "disabled";
 			};
 
 			mipi_dsi: mipi@021e0000 {
-- 
2.9.3

[PATCH 16/22] ARM: dts: nitrogen6x: Add dtsi for BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board

[Philipp Zabel]

Add device tree nodes for the BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver
board with a TC358743 connected to the Nitrogen6X MIPI CSI-2 input
connector.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 .../boot/dts/imx6qdl-nitrogen6x-bd-hdmi-mipi.dtsi  | 73 ++++++++++++++++++++++
 1 file changed, 73 insertions(+)
 create mode 100644 arch/arm/boot/dts/imx6qdl-nitrogen6x-bd-hdmi-mipi.dtsi

diff --git a/arch/arm/boot/dts/imx6qdl-nitrogen6x-bd-hdmi-mipi.dtsi b/arch/arm/boot/dts/imx6qdl-nitrogen6x-bd-hdmi-mipi.dtsi
new file mode 100644
index 0000000..e110874
--- /dev/null
+++ b/arch/arm/boot/dts/imx6qdl-nitrogen6x-bd-hdmi-mipi.dtsi
@@ -0,0 +1,73 @@
+/*
+ * Copyright 2015 Philipp Zabel, Pengutronix
+ *
+ * The code contained herein is licensed under the GNU General Public
+ * License. You may obtain a copy of the GNU General Public License
+ * Version 2 or later at the following locations:
+ *
+ * http://www.opensource.org/licenses/gpl-license.html
+ * http://www.gnu.org/copyleft/gpl.html
+ */
+#include <dt-bindings/gpio/gpio.h>
+#include <dt-bindings/interrupt-controller/irq.h>
+
+/ {
+	hdmi_osc: hdmi-osc {
+		compatible = "fixed-clock";
+		clock-output-names = "hdmi-osc";
+		clock-frequency = <27000000>;
+		#clock-cells = <0>;
+	};
+};
+
+&i2c2 {
+	tc358743: tc358743@0f {
+		compatible = "toshiba,tc358743";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_tc358743>;
+		reg = <0x0f>;
+		clocks = <&hdmi_osc>;
+		clock-names = "refclk";
+		reset-gpios = <&gpio6 9 GPIO_ACTIVE_LOW>;
+		interrupt-parent = <&gpio2>;
+		interrupts = <5 IRQ_TYPE_LEVEL_HIGH>;
+
+		port@0 {
+			tc358743_out: endpoint {
+				remote-endpoint = <&mipi_csi2_in>;
+				data-lanes = <1 2 3 4>;
+				clock-lanes = <0>;
+				clock-noncontinuous;
+				link-frequencies = /bits/ 64 <297000000>;
+			};
+		};
+	};
+};
+
+&iomuxc {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_hog>;
+
+	imx6q-nitrogen6x-tc358743 {
+		pinctrl_tc358743: tc358743grp {
+			fsl,pins = <
+				MX6QDL_PAD_NANDF_WP_B__GPIO6_IO09 0x4000b0b0	/* RESETN */
+				MX6QDL_PAD_NANDF_D5__GPIO2_IO05   0x400130b0	/* INT */
+			>;
+		};
+	};
+};
+
+&mipi_csi {
+	status = "okay";
+
+	port@0 {
+		mipi_csi2_in: endpoint {
+			remote-endpoint = <&tc358743_out>;
+			data-lanes = <1 2 3 4>;
+			clock-lanes = <0>;
+			clock-noncontinuous;
+			link-frequencies = /bits/ 64 <297000000>;
+		};
+	};
+};
-- 
2.9.3

[PATCH 17/22] gpu: ipuv3: add ipu_csi_set_downsize

[Philipp Zabel]

Support downsizing to 1/2 width and/or height in the CSI.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/gpu/ipu-v3/ipu-csi.c | 16 ++++++++++++++++
 include/video/imx-ipu-v3.h   |  1 +
 2 files changed, 17 insertions(+)

diff --git a/drivers/gpu/ipu-v3/ipu-csi.c b/drivers/gpu/ipu-v3/ipu-csi.c
index 06631ac..77f172e 100644
--- a/drivers/gpu/ipu-v3/ipu-csi.c
+++ b/drivers/gpu/ipu-v3/ipu-csi.c
@@ -527,6 +527,22 @@ void ipu_csi_set_window(struct ipu_csi *csi, struct v4l2_rect *w)
 }
 EXPORT_SYMBOL_GPL(ipu_csi_set_window);
 
+void ipu_csi_set_downsize(struct ipu_csi *csi, bool horiz, bool vert)
+{
+	unsigned long flags;
+	u32 reg;
+
+	spin_lock_irqsave(&csi->lock, flags);
+
+	reg = ipu_csi_read(csi, CSI_OUT_FRM_CTRL);
+	reg &= ~(CSI_HORI_DOWNSIZE_EN | CSI_VERT_DOWNSIZE_EN);
+	reg |= (horiz ? CSI_HORI_DOWNSIZE_EN : 0) |
+	       (vert ? CSI_VERT_DOWNSIZE_EN : 0);
+	ipu_csi_write(csi, reg, CSI_OUT_FRM_CTRL);
+
+	spin_unlock_irqrestore(&csi->lock, flags);
+}
+
 void ipu_csi_set_test_generator(struct ipu_csi *csi, bool active,
 				u32 r_value, u32 g_value, u32 b_value,
 				u32 pix_clk)
diff --git a/include/video/imx-ipu-v3.h b/include/video/imx-ipu-v3.h
index 7adeaae0..5f2f26d 100644
--- a/include/video/imx-ipu-v3.h
+++ b/include/video/imx-ipu-v3.h
@@ -271,6 +271,7 @@ int ipu_csi_init_interface(struct ipu_csi *csi,
 bool ipu_csi_is_interlaced(struct ipu_csi *csi);
 void ipu_csi_get_window(struct ipu_csi *csi, struct v4l2_rect *w);
 void ipu_csi_set_window(struct ipu_csi *csi, struct v4l2_rect *w);
+void ipu_csi_set_downsize(struct ipu_csi *csi, bool horiz, bool vert);
 void ipu_csi_set_test_generator(struct ipu_csi *csi, bool active,
 				u32 r_value, u32 g_value, u32 b_value,
 				u32 pix_clk);
-- 
2.9.3

[PATCH 18/22] [media] imx-ipuv3-csi: support downsizing

[Philipp Zabel]

Add support for the CSI internal horizontal and vertical downsizing.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/media/platform/imx/imx-ipuv3-csi.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/drivers/media/platform/imx/imx-ipuv3-csi.c b/drivers/media/platform/imx/imx-ipuv3-csi.c
index 699460e6..e8a6a7b 100644
--- a/drivers/media/platform/imx/imx-ipuv3-csi.c
+++ b/drivers/media/platform/imx/imx-ipuv3-csi.c
@@ -167,8 +167,16 @@ static int ipucsi_subdev_set_format(struct v4l2_subdev *sd,
 		width = clamp_t(unsigned int, sdformat->format.width, 16, 8192);
 		height = clamp_t(unsigned int, sdformat->format.height, 16, 4096);
 	} else {
-		width = ipucsi->format_mbus[0].width;
-		height = ipucsi->format_mbus[0].height;
+		if (sdformat->format.width <
+		    (ipucsi->format_mbus[0].width * 3 / 4))
+			width = ipucsi->format_mbus[0].width / 2;
+		else
+			width = ipucsi->format_mbus[0].width;
+		if (sdformat->format.height <
+		    (ipucsi->format_mbus[0].height * 3 / 4))
+			height = ipucsi->format_mbus[0].height / 2;
+		else
+			height = ipucsi->format_mbus[0].height;
 	}
 
 	mbusformat = __ipucsi_get_pad_format(sd, cfg, sdformat->pad,
@@ -212,14 +220,14 @@ static int ipucsi_subdev_s_stream(struct v4l2_subdev *sd, int enable)
 		window.width = fmt[0].width;
 		window.height = fmt[0].height;
 		ipu_csi_set_window(ipucsi->csi, &window);
+		ipu_csi_set_downsize(ipucsi->csi,
+				     fmt[0].width == 2 * fmt[1].width,
+				     fmt[0].height == 2 * fmt[1].height);
 
 		/* Is CSI data source MCT (MIPI)? */
 		mux_mct = (mbus_config.type == V4L2_MBUS_CSI2);
-
 		ipu_set_csi_src_mux(ipucsi->ipu, ipucsi->id, mux_mct);
-		if (mux_mct)
-			ipu_csi_set_mipi_datatype(ipucsi->csi, /*VC*/ 0,
-						  &fmt[0]);
+		ipu_csi_set_mipi_datatype(ipucsi->csi, /*VC*/ 0, &fmt[0]);
 
 		ret = ipu_csi_init_interface(ipucsi->csi, &mbus_config,
 					     &fmt[0]);
-- 
2.9.3

[PATCH 19/22] [media] add mux and video interface bridge entity functions

[Philipp Zabel]

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++++
 include/uapi/linux/media.h                        |  6 ++++++
 2 files changed, 28 insertions(+)

diff --git a/Documentation/media/uapi/mediactl/media-types.rst b/Documentation/media/uapi/mediactl/media-types.rst
index 3e03dc2..023be29 100644
--- a/Documentation/media/uapi/mediactl/media-types.rst
+++ b/Documentation/media/uapi/mediactl/media-types.rst
@@ -298,6 +298,28 @@ Types and flags used to represent the media graph elements
 	  received on its sink pad and outputs the statistics data on
 	  its source pad.
 
+    -  ..  row 29
+
+       ..  _MEDIA-ENT-F-MUX:
+
+       -  ``MEDIA_ENT_F_MUX``
+
+       - Video multiplexer. An entity capable of multiplexing must have at
+         least two sink pads and one source pad, and must pass the video
+         frame(s) received from the active sink pad to the source pad. Video
+         frame(s) from the inactive sink pads are discarded.
+
+    -  ..  row 30
+
+       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
+
+       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
+
+       - Video interface bridge. A video interface bridge entity must have at
+         least one sink pad and one source pad. It receives video frame(s) on
+         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
+         converts them and outputs them on its source pad in another bus format
+         (eDP, MIPI CSI-2, parallel, ...).
 
 ..  tabularcolumns:: |p{5.5cm}|p{12.0cm}|
 
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index 4890787..08a8bfa 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -105,6 +105,12 @@ struct media_device_info {
 #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
 
 /*
+ * Switch and bridge entitites
+ */
+#define MEDIA_ENT_F_MUX				(MEDIA_ENT_F_BASE + 0x5001)
+#define MEDIA_ENT_F_VID_IF_BRIDGE		(MEDIA_ENT_F_BASE + 0x5002)
+
+/*
  * Connectors
  */
 /* It is a responsibility of the entity drivers to add connectors and links */
-- 
2.9.3

[PATCH 20/22] [media] video-multiplexer: set entity function to mux

[Philipp Zabel]

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/media/platform/video-multiplexer.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/media/platform/video-multiplexer.c b/drivers/media/platform/video-multiplexer.c
index e9137ba..bcd1688 100644
--- a/drivers/media/platform/video-multiplexer.c
+++ b/drivers/media/platform/video-multiplexer.c
@@ -138,6 +138,7 @@ static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
 		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
 	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
 
+	vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
 	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
 				     vidsw->pads);
 	if (ret < 0)
-- 
2.9.3

[PATCH 21/22] [media] imx: Set i.MX MIPI CSI-2 entity function to bridge

[Philipp Zabel]

The i.MX6 MIPI CSI2 bridge converts the external MIPI CSI2 input into
a SoC internal parallel bus connected to the IPU CSIs via the CSI2IPU
gasket.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/media/platform/imx/imx-mipi-csi2.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/media/platform/imx/imx-mipi-csi2.c b/drivers/media/platform/imx/imx-mipi-csi2.c
index 0613187..85d2ef7 100644
--- a/drivers/media/platform/imx/imx-mipi-csi2.c
+++ b/drivers/media/platform/imx/imx-mipi-csi2.c
@@ -585,6 +585,7 @@ static int mipi_csi2_probe(struct platform_device *pdev)
 	csi2->pads[2].flags = MEDIA_PAD_FL_SOURCE;
 	csi2->pads[3].flags = MEDIA_PAD_FL_SOURCE;
 	csi2->pads[4].flags = MEDIA_PAD_FL_SOURCE;
+	csi2->subdev.entity.function = MEDIA_ENT_F_VID_IF_BRIDGE;
 	ret = media_entity_pads_init(&csi2->subdev.entity, MIPI_CSI2_PADS,
 				csi2->pads);
 	if (ret < 0)
-- 
2.9.3

[PATCH 22/22] [media] tc358743: set entity function to video interface bridge

[Philipp Zabel]

The TC358743 is an HDMI to MIPI CSI2-2 bridge.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
---
 drivers/media/i2c/tc358743.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/media/i2c/tc358743.c b/drivers/media/i2c/tc358743.c
index dfa45d2..c7a8f55 100644
--- a/drivers/media/i2c/tc358743.c
+++ b/drivers/media/i2c/tc358743.c
@@ -1891,6 +1891,7 @@ static int tc358743_probe(struct i2c_client *client,
 	}
 
 	state->pad.flags = MEDIA_PAD_FL_SOURCE;
+	sd->entity.function = MEDIA_ENT_F_VID_IF_BRIDGE;
 	err = media_entity_pads_init(&sd->entity, 1, &state->pad);
 	if (err < 0)
 		goto err_hdl;
-- 
2.9.3

Re: [PATCH 01/22] [media] v4l2-async: move code out of v4l2_async_notifier_register into v4l2_async_test_nofity_all

[Marek Vasut]

On 10/07/2016 06:00 PM, Philipp Zabel wrote:
> This will be reused in the following patch to catch already registered,
> newly added asynchronous subdevices from v4l2_async_register_subdev.
> 
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> ---
>  drivers/media/v4l2-core/v4l2-async.c | 38 +++++++++++++++++++++---------------
>  1 file changed, 22 insertions(+), 16 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-async.c b/drivers/media/v4l2-core/v4l2-async.c
> index 5bada20..c4f1930 100644
> --- a/drivers/media/v4l2-core/v4l2-async.c
> +++ b/drivers/media/v4l2-core/v4l2-async.c
> @@ -134,11 +134,31 @@ static void v4l2_async_cleanup(struct v4l2_subdev *sd)
>  	sd->dev = NULL;
>  }
>  
> +static int v4l2_async_test_notify_all(struct v4l2_async_notifier *notifier)
> +{
> +	struct v4l2_subdev *sd, *tmp;
> +
> +	list_for_each_entry_safe(sd, tmp, &subdev_list, async_list) {
> +		struct v4l2_async_subdev *asd;
> +		int ret;
> +
> +		asd = v4l2_async_belongs(notifier, sd);
> +		if (!asd)
> +			continue;
> +
> +		ret = v4l2_async_test_notify(notifier, sd, asd);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	return 0;
> +}
> +
>  int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
>  				 struct v4l2_async_notifier *notifier)
>  {
> -	struct v4l2_subdev *sd, *tmp;
>  	struct v4l2_async_subdev *asd;
> +	int ret;
>  	int i;
>  
>  	if (!notifier->num_subdevs || notifier->num_subdevs > V4L2_MAX_SUBDEVS)
> @@ -171,23 +191,9 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
>  	/* Keep also completed notifiers on the list */
>  	list_add(&notifier->list, &notifier_list);
>  
> -	list_for_each_entry_safe(sd, tmp, &subdev_list, async_list) {
> -		int ret;
> -
> -		asd = v4l2_async_belongs(notifier, sd);
> -		if (!asd)
> -			continue;
> -
> -		ret = v4l2_async_test_notify(notifier, sd, asd);
> -		if (ret < 0) {
> -			mutex_unlock(&list_lock);
> -			return ret;
> -		}
> -	}

Shouldn't you call ret = v4l2_async_test_notify_all() here now instead ?

>  	mutex_unlock(&list_lock);
>  
> -	return 0;
> +	return ret;
>  }
>  EXPORT_SYMBOL(v4l2_async_notifier_register);
>  
> 


-- 
Best regards,
Marek Vasut

Re: [PATCH 01/22] [media] v4l2-async: move code out of v4l2_async_notifier_register into v4l2_async_test_nofity_all

[Philipp Zabel]

Am Freitag, den 07.10.2016, 18:30 +0200 schrieb Marek Vasut:
> On 10/07/2016 06:00 PM, Philipp Zabel wrote:
> > This will be reused in the following patch to catch already registered,
> > newly added asynchronous subdevices from v4l2_async_register_subdev.
> > 
> > Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> > ---
> >  drivers/media/v4l2-core/v4l2-async.c | 38 +++++++++++++++++++++---------------
> >  1 file changed, 22 insertions(+), 16 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/v4l2-async.c b/drivers/media/v4l2-core/v4l2-async.c
> > index 5bada20..c4f1930 100644
> > --- a/drivers/media/v4l2-core/v4l2-async.c
> > +++ b/drivers/media/v4l2-core/v4l2-async.c
> > @@ -134,11 +134,31 @@ static void v4l2_async_cleanup(struct v4l2_subdev *sd)
> >  	sd->dev = NULL;
> >  }
> >  
> > +static int v4l2_async_test_notify_all(struct v4l2_async_notifier *notifier)
> > +{
> > +	struct v4l2_subdev *sd, *tmp;
> > +
> > +	list_for_each_entry_safe(sd, tmp, &subdev_list, async_list) {
> > +		struct v4l2_async_subdev *asd;
> > +		int ret;
> > +
> > +		asd = v4l2_async_belongs(notifier, sd);
> > +		if (!asd)
> > +			continue;
> > +
> > +		ret = v4l2_async_test_notify(notifier, sd, asd);
> > +		if (ret < 0)
> > +			return ret;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> >  int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
> >  				 struct v4l2_async_notifier *notifier)
> >  {
> > -	struct v4l2_subdev *sd, *tmp;
> >  	struct v4l2_async_subdev *asd;
> > +	int ret;
> >  	int i;
> >  
> >  	if (!notifier->num_subdevs || notifier->num_subdevs > V4L2_MAX_SUBDEVS)
> > @@ -171,23 +191,9 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
> >  	/* Keep also completed notifiers on the list */
> >  	list_add(&notifier->list, &notifier_list);
> >  
> > -	list_for_each_entry_safe(sd, tmp, &subdev_list, async_list) {
> > -		int ret;
> > -
> > -		asd = v4l2_async_belongs(notifier, sd);
> > -		if (!asd)
> > -			continue;
> > -
> > -		ret = v4l2_async_test_notify(notifier, sd, asd);
> > -		if (ret < 0) {
> > -			mutex_unlock(&list_lock);
> > -			return ret;
> > -		}
> > -	}
> 
> Shouldn't you call ret = v4l2_async_test_notify_all() here now instead ?

Absolutely, thanks. I've lost this in a rebase and the problem got
hidden by patch 2 replacing it again.

regards
Philipp

Re: [PATCH 03/22] [media] v4l: of: add v4l2_of_subdev_registered

[Marek Vasut]

On 10/07/2016 06:00 PM, Philipp Zabel wrote:
> Provide a default registered callback for device tree probed subdevices
> that use OF graph bindings to add still missing source subdevices to
> the async notifier waiting list.
> This is only necessary for subdevices that have input ports to which
> other subdevices are connected that are not initially known to the
> master/bridge device when it sets up the notifier.
> 
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>

[...]

> +int v4l2_of_subdev_registered(struct v4l2_subdev *sd)
> +{
> +	struct device_node *ep;
> +
> +	for_each_endpoint_of_node(sd->of_node, ep) {
> +		struct v4l2_of_link link;
> +		struct media_entity *entity;
> +		unsigned int pad;
> +		int ret;
> +
> +		ret = v4l2_of_parse_link(ep, &link);
> +		if (ret)
> +			continue;
> +
> +		/*
> +		 * Assume 1:1 correspondence between OF node and entity,
> +		 * and between OF port numbers and pad indices.
> +		 */
> +		entity = &sd->entity;

This here will not compile if CONFIG_MEDIA_CONTROLLER is not set,
because ->entity will be missing from struct v4l2_subdev {} .

> +		pad = link.local_port;
> +
> +		if (pad >= entity->num_pads)
> +			return -EINVAL;
> +
> +		/* Add source subdevs to async notifier */
> +		if (entity->pads[pad].flags & MEDIA_PAD_FL_SINK) {
> +			struct v4l2_async_subdev *asd;
> +
> +			asd = devm_kzalloc(sd->dev, sizeof(*asd), GFP_KERNEL);
> +			if (!asd) {
> +				v4l2_of_put_link(&link);
> +				return -ENOMEM;
> +			}
> +
> +			asd->match_type = V4L2_ASYNC_MATCH_OF;
> +			asd->match.of.node = link.remote_node;
> +
> +			__v4l2_async_notifier_add_subdev(sd->notifier, asd);
> +		}
> +
> +		v4l2_of_put_link(&link);
> +	}
> +
> +	return 0;
> +}
> diff --git a/include/media/v4l2-of.h b/include/media/v4l2-of.h
> index 4dc34b2..67d4f8b 100644
> --- a/include/media/v4l2-of.h
> +++ b/include/media/v4l2-of.h
> @@ -22,6 +22,8 @@
>  #include <media/v4l2-mediabus.h>
>  
>  struct device_node;
> +struct v4l2_device;
> +struct v4l2_subdev;
>  
>  /**
>   * struct v4l2_of_bus_mipi_csi2 - MIPI CSI-2 bus data structure
> @@ -95,6 +97,9 @@ void v4l2_of_free_endpoint(struct v4l2_of_endpoint *endpoint);
>  int v4l2_of_parse_link(const struct device_node *node,
>  		       struct v4l2_of_link *link);
>  void v4l2_of_put_link(struct v4l2_of_link *link);
> +int v4l2_of_subdev_registered(struct v4l2_subdev *sd);
> +struct v4l2_subdev *v4l2_find_subdev_by_node(struct v4l2_device *v4l2_dev,
> +					     struct device_node *node);
>  #else /* CONFIG_OF */
>  
>  static inline int v4l2_of_parse_endpoint(const struct device_node *node,
> @@ -123,6 +128,13 @@ static inline void v4l2_of_put_link(struct v4l2_of_link *link)
>  {
>  }
>  
> +#define v4l2_of_subdev_registered NULL
> +
> +struct v4l2_subdev *v4l2_find_subdev_by_node(struct v4l2_device *v4l2_dev,
> +					     struct device_node *node)
> +{
> +	return NULL;
> +}
>  #endif /* CONFIG_OF */
>  
>  #endif /* _V4L2_OF_H */
> 


-- 
Best regards,
Marek Vasut

Re: [PATCH 18/22] [media] imx-ipuv3-csi: support downsizing

[Marek Vasut]

On 10/07/2016 06:01 PM, Philipp Zabel wrote:
> Add support for the CSI internal horizontal and vertical downsizing.
> 
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> ---
>  drivers/media/platform/imx/imx-ipuv3-csi.c | 20 ++++++++++++++------
>  1 file changed, 14 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/media/platform/imx/imx-ipuv3-csi.c b/drivers/media/platform/imx/imx-ipuv3-csi.c
> index 699460e6..e8a6a7b 100644
> --- a/drivers/media/platform/imx/imx-ipuv3-csi.c
> +++ b/drivers/media/platform/imx/imx-ipuv3-csi.c
> @@ -167,8 +167,16 @@ static int ipucsi_subdev_set_format(struct v4l2_subdev *sd,
>  		width = clamp_t(unsigned int, sdformat->format.width, 16, 8192);
>  		height = clamp_t(unsigned int, sdformat->format.height, 16, 4096);
>  	} else {
> -		width = ipucsi->format_mbus[0].width;
> -		height = ipucsi->format_mbus[0].height;
> +		if (sdformat->format.width <
> +		    (ipucsi->format_mbus[0].width * 3 / 4))
> +			width = ipucsi->format_mbus[0].width / 2;
> +		else
> +			width = ipucsi->format_mbus[0].width;
> +		if (sdformat->format.height <
> +		    (ipucsi->format_mbus[0].height * 3 / 4))
> +			height = ipucsi->format_mbus[0].height / 2;
> +		else
> +			height = ipucsi->format_mbus[0].height;
>  	}
>  
>  	mbusformat = __ipucsi_get_pad_format(sd, cfg, sdformat->pad,
> @@ -212,14 +220,14 @@ static int ipucsi_subdev_s_stream(struct v4l2_subdev *sd, int enable)
>  		window.width = fmt[0].width;
>  		window.height = fmt[0].height;
>  		ipu_csi_set_window(ipucsi->csi, &window);
> +		ipu_csi_set_downsize(ipucsi->csi,
> +				     fmt[0].width == 2 * fmt[1].width,
> +				     fmt[0].height == 2 * fmt[1].height);
>  
>  		/* Is CSI data source MCT (MIPI)? */
>  		mux_mct = (mbus_config.type == V4L2_MBUS_CSI2);
> -
>  		ipu_set_csi_src_mux(ipucsi->ipu, ipucsi->id, mux_mct);
> -		if (mux_mct)
> -			ipu_csi_set_mipi_datatype(ipucsi->csi, /*VC*/ 0,
> -						  &fmt[0]);
> +		ipu_csi_set_mipi_datatype(ipucsi->csi, /*VC*/ 0, &fmt[0]);

This probably needs fixing , so that the correct VC is passed in ?

>  		ret = ipu_csi_init_interface(ipucsi->csi, &mbus_config,
>  					     &fmt[0]);
> 


-- 
Best regards,
Marek Vasut

Re: [PATCH 12/22] [media] tc358743: put lanes in STOP state before starting streaming

[Marek Vasut]

On 10/07/2016 06:00 PM, Philipp Zabel wrote:
> Without calling tc358743_set_csi from the new prepare_stream callback
> (or calling tc358743_s_dv_timings or tc358743_set_fmt from userspace
> after stopping the stream), the i.MX6 MIPI CSI2 input fails waiting
> for lanes to enter STOP state when streaming is started again.

What is the impact of that failure ? How does it manifest itself ?

> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> ---
>  drivers/media/i2c/tc358743.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/drivers/media/i2c/tc358743.c b/drivers/media/i2c/tc358743.c
> index 1e3a0dd2..dfa45d2 100644
> --- a/drivers/media/i2c/tc358743.c
> +++ b/drivers/media/i2c/tc358743.c
> @@ -1463,6 +1463,14 @@ static int tc358743_g_mbus_config(struct v4l2_subdev *sd,
>  	return 0;
>  }
>  
> +static int tc358743_prepare_stream(struct v4l2_subdev *sd)
> +{
> +	/* Put all lanes in PL-11 state (STOPSTATE) */
> +	tc358743_set_csi(sd);
> +
> +	return 0;
> +}
> +
>  static int tc358743_s_stream(struct v4l2_subdev *sd, int enable)
>  {
>  	enable_stream(sd, enable);
> @@ -1637,6 +1645,7 @@ static const struct v4l2_subdev_video_ops tc358743_video_ops = {
>  	.g_dv_timings = tc358743_g_dv_timings,
>  	.query_dv_timings = tc358743_query_dv_timings,
>  	.g_mbus_config = tc358743_g_mbus_config,
> +	.prepare_stream = tc358743_prepare_stream,
>  	.s_stream = tc358743_s_stream,
>  };
>  
> 


-- 
Best regards,
Marek Vasut

Re: [PATCH 02/22] [media] v4l2-async: allow subdevices to add further subdevices to the notifier waiting list

[Sakari Ailus]

Hi Philipp,

On Fri, Oct 07, 2016 at 06:00:47PM +0200, Philipp Zabel wrote:
> Currently the v4l2_async_notifier needs to be given a list of matches
> for all expected subdevices on creation. When chaining subdevices that
> are asynchronously probed via device tree, the bridge device that sets
> up the notifier does not know the complete list of subdevices, as it
> can only parse its own device tree node to obtain information about
> the nearest neighbor subdevices.
> To support indirectly connected subdevices, we need to support amending
> the existing notifier waiting list with newly found neighbor subdevices
> with each registered subdevice.

Could you elaborate a little what's the exact use case for this? What kind
of a device?

Thanks.

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: [PATCH 02/22] [media] v4l2-async: allow subdevices to add further subdevices to the notifier waiting list

[Sakari Ailus]

Hi Philipp,

On Fri, Oct 07, 2016 at 06:00:47PM +0200, Philipp Zabel wrote:
> Currently the v4l2_async_notifier needs to be given a list of matches
> for all expected subdevices on creation. When chaining subdevices that
> are asynchronously probed via device tree, the bridge device that sets
> up the notifier does not know the complete list of subdevices, as it
> can only parse its own device tree node to obtain information about
> the nearest neighbor subdevices.
> To support indirectly connected subdevices, we need to support amending
> the existing notifier waiting list with newly found neighbor subdevices
> with each registered subdevice.
> 
> This can be achieved by adding new v42l_async_subdev matches to the
> notifier waiting list during the v4l2_subdev registered callback, which
> is called under the list lock from either v4l2_async_register_subdev or
> v4l2_async_notifier_register. For this purpose a new exported function
> __v4l2_async_notifier_add_subdev is added.
> 
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> ---
>  drivers/media/v4l2-core/v4l2-async.c | 78 ++++++++++++++++++++++++++++++++++--
>  include/media/v4l2-async.h           | 12 ++++++
>  2 files changed, 86 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-async.c b/drivers/media/v4l2-core/v4l2-async.c
> index c4f1930..404eeea 100644
> --- a/drivers/media/v4l2-core/v4l2-async.c
> +++ b/drivers/media/v4l2-core/v4l2-async.c
> @@ -109,6 +109,7 @@ static int v4l2_async_test_notify(struct v4l2_async_notifier *notifier,
>  		if (ret < 0)
>  			return ret;
>  	}
> +
>  	/* Move from the global subdevice list to notifier's done */
>  	list_move(&sd->async_list, &notifier->done);
>  
> @@ -158,7 +159,7 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
>  				 struct v4l2_async_notifier *notifier)
>  {
>  	struct v4l2_async_subdev *asd;
> -	int ret;
> +	struct list_head *tail;
>  	int i;
>  
>  	if (!notifier->num_subdevs || notifier->num_subdevs > V4L2_MAX_SUBDEVS)
> @@ -191,17 +192,71 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
>  	/* Keep also completed notifiers on the list */
>  	list_add(&notifier->list, &notifier_list);
>  
> +	do {
> +		int ret;
> +
> +		tail = notifier->waiting.prev;
> +
> +		ret = v4l2_async_test_notify_all(notifier);
> +		if (ret < 0) {
> +			mutex_unlock(&list_lock);
> +			return ret;
> +		}
> +
> +		/*
> +		 * If entries were added to the notifier waiting list, check
> +		 * again if the corresponding subdevices are already available.
> +		 */
> +	} while (tail != notifier->waiting.prev);
> +
>  	mutex_unlock(&list_lock);
>  
> -	return ret;
> +	return 0;
>  }
>  EXPORT_SYMBOL(v4l2_async_notifier_register);
>  
> +int __v4l2_async_notifier_add_subdev(struct v4l2_async_notifier *notifier,
> +				     struct v4l2_async_subdev *asd)
> +{
> +	struct v4l2_async_subdev *tmp_asd;
> +
> +	lockdep_assert_held(&list_lock);
> +
> +	if (asd->match_type != V4L2_ASYNC_MATCH_OF)
> +		return -EINVAL;
> +
> +	/*
> +	 * First check if the same notifier is already on the waiting or done
> +	 * lists. This can happen if a subdevice with multiple outputs is added
> +	 * by all its downstream subdevices.
> +	 */
> +	list_for_each_entry(tmp_asd, &notifier->waiting, list)
> +		if (tmp_asd->match.of.node == asd->match.of.node)
> +			return 0;
> +	list_for_each_entry(tmp_asd, &notifier->done, list)
> +		if (tmp_asd->match.of.node == asd->match.of.node)
> +			return 0;
> +
> +	/*
> +	 * Add the new async subdev to the notifier waiting list, so
> +	 * v4l2_async_belongs may use it to compare against entries in
> +	 * subdev_list.
> +	 * In case the subdev matching asd has already been passed in the
> +	 * subdev_list walk in v4l2_async_notifier_register, or if
> +	 * we are called from v4l2_async_register_subdev, the subdev_list
> +	 * will have to be walked again.
> +	 */
> +	list_add_tail(&asd->list, &notifier->waiting);
> +
> +	return 0;
> +}
> +
>  void v4l2_async_notifier_unregister(struct v4l2_async_notifier *notifier)
>  {
>  	struct v4l2_subdev *sd, *tmp;
> -	unsigned int notif_n_subdev = notifier->num_subdevs;
> -	unsigned int n_subdev = min(notif_n_subdev, V4L2_MAX_SUBDEVS);
> +	unsigned int notif_n_subdev = 0;
> +	unsigned int n_subdev;
> +	struct list_head *list;
>  	struct device **dev;
>  	int i = 0;
>  
> @@ -218,6 +273,10 @@ void v4l2_async_notifier_unregister(struct v4l2_async_notifier *notifier)
>  
>  	list_del(&notifier->list);
>  
> +	list_for_each(list, &notifier->done)
> +		++notif_n_subdev;
> +	n_subdev = min(notif_n_subdev, V4L2_MAX_SUBDEVS);
> +

Shouldn't this change go to a separate patch? It seems unrelated.

>  	list_for_each_entry_safe(sd, tmp, &notifier->done, async_list) {
>  		struct device *d;
>  
> @@ -294,8 +353,19 @@ int v4l2_async_register_subdev(struct v4l2_subdev *sd)
>  	list_for_each_entry(notifier, &notifier_list, list) {
>  		struct v4l2_async_subdev *asd = v4l2_async_belongs(notifier, sd);
>  		if (asd) {
> +			struct list_head *tail = notifier->waiting.prev;
>  			int ret = v4l2_async_test_notify(notifier, sd, asd);
> +
> +			/*
> +			 * If entries were added to the notifier waiting list,
> +			 * check if the corresponding subdevices are already
> +			 * available.
> +			 */
> +			if (tail != notifier->waiting.prev)
> +				ret = v4l2_async_test_notify_all(notifier);
> +
>  			mutex_unlock(&list_lock);
> +
>  			return ret;
>  		}
>  	}
> diff --git a/include/media/v4l2-async.h b/include/media/v4l2-async.h
> index 8e2a236..e4e4b11 100644
> --- a/include/media/v4l2-async.h
> +++ b/include/media/v4l2-async.h
> @@ -114,6 +114,18 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
>  				 struct v4l2_async_notifier *notifier);
>  
>  /**
> + * __v4l2_async_notifier_add_subdev - adds a subdevice to the notifier waitlist
> + *
> + * @v4l2_notifier: notifier the calling subdev is bound to

s/v4l2_//

> + * @asd: asynchronous subdev match
> + *
> + * To be called from inside a subdevices' registered_async callback to add
> + * additional subdevices to the notifier waiting list.
> + */
> +int __v4l2_async_notifier_add_subdev(struct v4l2_async_notifier *notifier,
> +				     struct v4l2_async_subdev *asd);
> +
> +/**
>   * v4l2_async_notifier_unregister - unregisters a subdevice asynchronous notifier
>   *
>   * @notifier: pointer to &struct v4l2_async_notifier

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: [PATCH 03/22] [media] v4l: of: add v4l2_of_subdev_registered

[Sakari Ailus]

Hi Philipp,

On Fri, Oct 07, 2016 at 06:00:48PM +0200, Philipp Zabel wrote:
> Provide a default registered callback for device tree probed subdevices
> that use OF graph bindings to add still missing source subdevices to
> the async notifier waiting list.
> This is only necessary for subdevices that have input ports to which
> other subdevices are connected that are not initially known to the
> master/bridge device when it sets up the notifier.
> 
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> ---
>  drivers/media/v4l2-core/v4l2-of.c | 68 +++++++++++++++++++++++++++++++++++++++
>  include/media/v4l2-of.h           | 12 +++++++
>  2 files changed, 80 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-of.c b/drivers/media/v4l2-core/v4l2-of.c
> index 93b3368..fbdd6b4 100644
> --- a/drivers/media/v4l2-core/v4l2-of.c
> +++ b/drivers/media/v4l2-core/v4l2-of.c
> @@ -19,6 +19,7 @@
>  #include <linux/types.h>
>  
>  #include <media/v4l2-of.h>
> +#include <media/v4l2-device.h>

Alphabetical order, please.

>  
>  static int v4l2_of_parse_csi_bus(const struct device_node *node,
>  				 struct v4l2_of_endpoint *endpoint)
> @@ -314,3 +315,70 @@ void v4l2_of_put_link(struct v4l2_of_link *link)
>  	of_node_put(link->remote_node);
>  }
>  EXPORT_SYMBOL(v4l2_of_put_link);
> +
> +struct v4l2_subdev *v4l2_find_subdev_by_node(struct v4l2_device *v4l2_dev,
> +					     struct device_node *node)
> +{
> +	struct v4l2_subdev *sd;
> +
> +	list_for_each_entry(sd, &v4l2_dev->subdevs, list) {
> +		if (sd->of_node == node)
> +			return sd;
> +	}

The braces aren't really needed. Up to you.

> +
> +	return NULL;
> +}

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: [PATCH 04/22] [media] v4l2-subdev.h: add prepare_stream op

[Sakari Ailus]

Hi Philipp,

On Fri, Oct 07, 2016 at 06:00:49PM +0200, Philipp Zabel wrote:
> In some cases, for example MIPI CSI-2 input on i.MX6, the sending and
> receiving subdevice need to be prepared in lock-step before the actual
> streaming can start. In the i.MX6 MIPI CSI-2 case, the sender needs to
> put its MIPI CSI-2 transmitter lanes into stop state, and the receiver
> needs to configure its D-PHY and detect the stop state on all active
> lanes. Only then the sender can be enabled to stream data and the
> receiver can lock its PLL to the clock lane.

Is there a need to explicitly control this? Shouldn't this already be the
case when the transmitting device is powered on and is not streaming?

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: [16/22] ARM: dts: nitrogen6x: Add dtsi for BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board

[Gary Bisson]

Hi Philipp, All,

On Fri, Oct 07, 2016 at 06:01:01PM +0200, Philipp Zabel wrote:
> Add device tree nodes for the BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver
> board with a TC358743 connected to the Nitrogen6X MIPI CSI-2 input
> connector.

I've tested this series on my Nitrogen6x + BD_HDMI_MIPI daughter board
and have a few questions.

First, why is the tc358743 node in a separate dtsi file? Is this in
order to avoid a failed probe during bootup if the daughter board is
not present? Is this what should be done for every capture device that
targets this platform (like the OV5640 or OV5642)?

Can you provide some details on your testing procedure? In my case I've
reached a point where I get the same 'media-ctl --print-dot' output as
the one from your cover letter but I can't seem to set the EDID nor to
have a gstreamer pipeline (to fakesink). All the EDID v4l2-ctl commands
return "Inappropriate ioctl for device".

Do not hesitate to CC me to Boundary Devices related patches so I can
test them and give some feedback.

Regards,
Gary

Re: [PATCH 02/22] [media] v4l2-async: allow subdevices to add further subdevices to the notifier waiting list

[Philipp Zabel]

On Fri, Oct 7, 2016 at 11:52 PM, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> Hi Philipp,
>
> On Fri, Oct 07, 2016 at 06:00:47PM +0200, Philipp Zabel wrote:
>> Currently the v4l2_async_notifier needs to be given a list of matches
>> for all expected subdevices on creation. When chaining subdevices that
>> are asynchronously probed via device tree, the bridge device that sets
>> up the notifier does not know the complete list of subdevices, as it
>> can only parse its own device tree node to obtain information about
>> the nearest neighbor subdevices.
>> To support indirectly connected subdevices, we need to support amending
>> the existing notifier waiting list with newly found neighbor subdevices
>> with each registered subdevice.
>
> Could you elaborate a little what's the exact use case for this? What kind
> of a device?

On i.MX6 there's a

DW MIPI CSI2 host -> Mux -> IPU/CSI

path and all three are asynchronous subdevices in my patchset and only
the last one is directly known to the media device from the device
tree, since each driver should only parse its own device tree node an
can not follow the of_graph over multiple steps.
Another use case I have seen in the wild are external GPIO controlled
multiplexers or LVDS serializer/deserializer pairs between a parallel
camera and parallel capture interface. In each case the bridge node
can only determine its closest neighbor from the device tree (the mux,
the LVDS deserializer) but does not know about the indirectly
connected device nodes further upstream.

regards
Philipp

Re: [PATCH 02/22] [media] v4l2-async: allow subdevices to add further subdevices to the notifier waiting list

[Sakari Ailus]

Hi Philipp,

On Wed, Oct 12, 2016 at 03:26:48PM +0200, Philipp Zabel wrote:
> On Fri, Oct 7, 2016 at 11:52 PM, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > Hi Philipp,
> >
> > On Fri, Oct 07, 2016 at 06:00:47PM +0200, Philipp Zabel wrote:
> >> Currently the v4l2_async_notifier needs to be given a list of matches
> >> for all expected subdevices on creation. When chaining subdevices that
> >> are asynchronously probed via device tree, the bridge device that sets
> >> up the notifier does not know the complete list of subdevices, as it
> >> can only parse its own device tree node to obtain information about
> >> the nearest neighbor subdevices.
> >> To support indirectly connected subdevices, we need to support amending
> >> the existing notifier waiting list with newly found neighbor subdevices
> >> with each registered subdevice.
> >
> > Could you elaborate a little what's the exact use case for this? What kind
> > of a device?
> 
> On i.MX6 there's a
> 
> DW MIPI CSI2 host -> Mux -> IPU/CSI
> 
> path and all three are asynchronous subdevices in my patchset and only
> the last one is directly known to the media device from the device
> tree, since each driver should only parse its own device tree node an
> can not follow the of_graph over multiple steps.

Ok. Are all these devices part of the SoC? Is the mux doing something else
than just sitting in between the two? :-)

> Another use case I have seen in the wild are external GPIO controlled
> multiplexers or LVDS serializer/deserializer pairs between a parallel
> camera and parallel capture interface. In each case the bridge node
> can only determine its closest neighbor from the device tree (the mux,
> the LVDS deserializer) but does not know about the indirectly
> connected device nodes further upstream.

Yeah, true.

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: [PATCH 02/22] [media] v4l2-async: allow subdevices to add further subdevices to the notifier waiting list

[Philipp Zabel]

Am Donnerstag, den 13.10.2016, 14:56 +0300 schrieb Sakari Ailus:
> Hi Philipp,
> 
> On Wed, Oct 12, 2016 at 03:26:48PM +0200, Philipp Zabel wrote:
> > On Fri, Oct 7, 2016 at 11:52 PM, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > > Hi Philipp,
> > >
> > > On Fri, Oct 07, 2016 at 06:00:47PM +0200, Philipp Zabel wrote:
> > >> Currently the v4l2_async_notifier needs to be given a list of matches
> > >> for all expected subdevices on creation. When chaining subdevices that
> > >> are asynchronously probed via device tree, the bridge device that sets
> > >> up the notifier does not know the complete list of subdevices, as it
> > >> can only parse its own device tree node to obtain information about
> > >> the nearest neighbor subdevices.
> > >> To support indirectly connected subdevices, we need to support amending
> > >> the existing notifier waiting list with newly found neighbor subdevices
> > >> with each registered subdevice.
> > >
> > > Could you elaborate a little what's the exact use case for this? What kind
> > > of a device?
> > 
> > On i.MX6 there's a
> > 
> > DW MIPI CSI2 host -> Mux -> IPU/CSI
> > 
> > path and all three are asynchronous subdevices in my patchset and only
> > the last one is directly known to the media device from the device
> > tree, since each driver should only parse its own device tree node an
> > can not follow the of_graph over multiple steps.
> 
> Ok. Are all these devices part of the SoC? Is the mux doing something else
> than just sitting in between the two? :-)

Yes, in this case the muxes are part of the SoC. Depending on the SoC
variant they have a different number of inputs, and they are controlled
via the IOMUXC block that also controls the pin configuration. They
don't have any additional functionality beyond selecting one active
input.

regards
Philipp

Re: [PATCH 02/22] [media] v4l2-async: allow subdevices to add further subdevices to the notifier waiting list

[Philipp Zabel]

Am Samstag, den 08.10.2016, 01:43 +0300 schrieb Sakari Ailus:
[...]
> >  void v4l2_async_notifier_unregister(struct v4l2_async_notifier *notifier)
> >  {
> >  	struct v4l2_subdev *sd, *tmp;
> > -	unsigned int notif_n_subdev = notifier->num_subdevs;
> > -	unsigned int n_subdev = min(notif_n_subdev, V4L2_MAX_SUBDEVS);
> > +	unsigned int notif_n_subdev = 0;
> > +	unsigned int n_subdev;
> > +	struct list_head *list;
> >  	struct device **dev;
> >  	int i = 0;
> >  
> > @@ -218,6 +273,10 @@ void v4l2_async_notifier_unregister(struct v4l2_async_notifier *notifier)
> >  
> >  	list_del(&notifier->list);
> >  
> > +	list_for_each(list, &notifier->done)
> > +		++notif_n_subdev;
> > +	n_subdev = min(notif_n_subdev, V4L2_MAX_SUBDEVS);
> > +
> 
> Shouldn't this change go to a separate patch? It seems unrelated.

Thanks, this was intended to count the notifier done list instead of
relying on notifier->num_subdevs because of the additional asynchronous
subdevs added to the notifier that are not part of the original array.
Unfortunately this change is a few lines too late, it belongs before the
device cache is allocated. I'll fix this and add a comment.

I don't want to increment notifier->num_subdevs in
__v4l2_async_notifier_add_subdev because the caller of
v4l2_async_notifier_register might still use it to measure the original
array.

> >  	list_for_each_entry_safe(sd, tmp, &notifier->done, async_list) {
> >  		struct device *d;
> >  
> > @@ -294,8 +353,19 @@ int v4l2_async_register_subdev(struct v4l2_subdev *sd)
> >  	list_for_each_entry(notifier, &notifier_list, list) {
> >  		struct v4l2_async_subdev *asd = v4l2_async_belongs(notifier, sd);
> >  		if (asd) {
> > +			struct list_head *tail = notifier->waiting.prev;
> >  			int ret = v4l2_async_test_notify(notifier, sd, asd);
> > +
> > +			/*
> > +			 * If entries were added to the notifier waiting list,
> > +			 * check if the corresponding subdevices are already
> > +			 * available.
> > +			 */
> > +			if (tail != notifier->waiting.prev)
> > +				ret = v4l2_async_test_notify_all(notifier);
> > +
> >  			mutex_unlock(&list_lock);
> > +
> >  			return ret;
> >  		}
> >  	}
> > diff --git a/include/media/v4l2-async.h b/include/media/v4l2-async.h
> > index 8e2a236..e4e4b11 100644
> > --- a/include/media/v4l2-async.h
> > +++ b/include/media/v4l2-async.h
> > @@ -114,6 +114,18 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
> >  				 struct v4l2_async_notifier *notifier);
> >  
> >  /**
> > + * __v4l2_async_notifier_add_subdev - adds a subdevice to the notifier waitlist
> > + *
> > + * @v4l2_notifier: notifier the calling subdev is bound to
> 
> s/v4l2_//

I'd be happy to, but why should the v4l2 prefix be removed?

regards
Philipp

Re: [PATCH 04/22] [media] v4l2-subdev.h: add prepare_stream op

[Philipp Zabel]

Am Samstag, den 08.10.2016, 02:16 +0300 schrieb Sakari Ailus:
> Hi Philipp,
> 
> On Fri, Oct 07, 2016 at 06:00:49PM +0200, Philipp Zabel wrote:
> > In some cases, for example MIPI CSI-2 input on i.MX6, the sending and
> > receiving subdevice need to be prepared in lock-step before the actual
> > streaming can start. In the i.MX6 MIPI CSI-2 case, the sender needs to
> > put its MIPI CSI-2 transmitter lanes into stop state, and the receiver
> > needs to configure its D-PHY and detect the stop state on all active
> > lanes. Only then the sender can be enabled to stream data and the
> > receiver can lock its PLL to the clock lane.
> 
> Is there a need to explicitly control this? Shouldn't this already be the
> case when the transmitting device is powered on and is not streaming?

Even if the transmitter is expected to keep the lanes in this stop state
all the time while the subdevice is powered but not streaming, I still
have to wait for stop state detection before enabling the transmitter,
and only then enable the reciever.
I'll remove the prepare_streaming callback in the next version and
instead let the subdevices propagate s_stream upstream instead in the
next version.

regards
Philipp

Re: [PATCH 18/22] [media] imx-ipuv3-csi: support downsizing

[Philipp Zabel]

Am Freitag, den 07.10.2016, 21:01 +0200 schrieb Marek Vasut:
> On 10/07/2016 06:01 PM, Philipp Zabel wrote:
> > Add support for the CSI internal horizontal and vertical downsizing.
> > 
> > Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> > ---
> >  drivers/media/platform/imx/imx-ipuv3-csi.c | 20 ++++++++++++++------
> >  1 file changed, 14 insertions(+), 6 deletions(-)
> > 
> > diff --git a/drivers/media/platform/imx/imx-ipuv3-csi.c b/drivers/media/platform/imx/imx-ipuv3-csi.c
> > index 699460e6..e8a6a7b 100644
> > --- a/drivers/media/platform/imx/imx-ipuv3-csi.c
> > +++ b/drivers/media/platform/imx/imx-ipuv3-csi.c
> > @@ -167,8 +167,16 @@ static int ipucsi_subdev_set_format(struct v4l2_subdev *sd,
> >  		width = clamp_t(unsigned int, sdformat->format.width, 16, 8192);
> >  		height = clamp_t(unsigned int, sdformat->format.height, 16, 4096);
> >  	} else {
> > -		width = ipucsi->format_mbus[0].width;
> > -		height = ipucsi->format_mbus[0].height;
> > +		if (sdformat->format.width <
> > +		    (ipucsi->format_mbus[0].width * 3 / 4))
> > +			width = ipucsi->format_mbus[0].width / 2;
> > +		else
> > +			width = ipucsi->format_mbus[0].width;
> > +		if (sdformat->format.height <
> > +		    (ipucsi->format_mbus[0].height * 3 / 4))
> > +			height = ipucsi->format_mbus[0].height / 2;
> > +		else
> > +			height = ipucsi->format_mbus[0].height;
> >  	}
> >  
> >  	mbusformat = __ipucsi_get_pad_format(sd, cfg, sdformat->pad,
> > @@ -212,14 +220,14 @@ static int ipucsi_subdev_s_stream(struct v4l2_subdev *sd, int enable)
> >  		window.width = fmt[0].width;
> >  		window.height = fmt[0].height;
> >  		ipu_csi_set_window(ipucsi->csi, &window);
> > +		ipu_csi_set_downsize(ipucsi->csi,
> > +				     fmt[0].width == 2 * fmt[1].width,
> > +				     fmt[0].height == 2 * fmt[1].height);
> >  
> >  		/* Is CSI data source MCT (MIPI)? */
> >  		mux_mct = (mbus_config.type == V4L2_MBUS_CSI2);
> > -
> >  		ipu_set_csi_src_mux(ipucsi->ipu, ipucsi->id, mux_mct);
> > -		if (mux_mct)
> > -			ipu_csi_set_mipi_datatype(ipucsi->csi, /*VC*/ 0,
> > -						  &fmt[0]);
> > +		ipu_csi_set_mipi_datatype(ipucsi->csi, /*VC*/ 0, &fmt[0]);
> 
> This probably needs fixing , so that the correct VC is passed in ?

Absolutely, right now I don't know how though.
We are still missing API to set the MIPI CSI-2 virtual channel.

regards
Philipp

Re: [PATCH 03/22] [media] v4l: of: add v4l2_of_subdev_registered

[Philipp Zabel]

Am Freitag, den 07.10.2016, 20:50 +0200 schrieb Marek Vasut:
> On 10/07/2016 06:00 PM, Philipp Zabel wrote:
> > Provide a default registered callback for device tree probed subdevices
> > that use OF graph bindings to add still missing source subdevices to
> > the async notifier waiting list.
> > This is only necessary for subdevices that have input ports to which
> > other subdevices are connected that are not initially known to the
> > master/bridge device when it sets up the notifier.
> > 
> > Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> 
> [...]
> 
> > +int v4l2_of_subdev_registered(struct v4l2_subdev *sd)
> > +{
> > +	struct device_node *ep;
> > +
> > +	for_each_endpoint_of_node(sd->of_node, ep) {
> > +		struct v4l2_of_link link;
> > +		struct media_entity *entity;
> > +		unsigned int pad;
> > +		int ret;
> > +
> > +		ret = v4l2_of_parse_link(ep, &link);
> > +		if (ret)
> > +			continue;
> > +
> > +		/*
> > +		 * Assume 1:1 correspondence between OF node and entity,
> > +		 * and between OF port numbers and pad indices.
> > +		 */
> > +		entity = &sd->entity;
> 
> This here will not compile if CONFIG_MEDIA_CONTROLLER is not set,
> because ->entity will be missing from struct v4l2_subdev {} .

Thanks, I'll fix this.

regards
Philipp

Re: [PATCH 03/22] [media] v4l: of: add v4l2_of_subdev_registered

[Philipp Zabel]

Am Samstag, den 08.10.2016, 01:50 +0300 schrieb Sakari Ailus:
> Hi Philipp,
> 
> On Fri, Oct 07, 2016 at 06:00:48PM +0200, Philipp Zabel wrote:
> > Provide a default registered callback for device tree probed subdevices
> > that use OF graph bindings to add still missing source subdevices to
> > the async notifier waiting list.
> > This is only necessary for subdevices that have input ports to which
> > other subdevices are connected that are not initially known to the
> > master/bridge device when it sets up the notifier.
> > 
> > Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> > ---
> >  drivers/media/v4l2-core/v4l2-of.c | 68 +++++++++++++++++++++++++++++++++++++++
> >  include/media/v4l2-of.h           | 12 +++++++
> >  2 files changed, 80 insertions(+)
> > 
> > diff --git a/drivers/media/v4l2-core/v4l2-of.c b/drivers/media/v4l2-core/v4l2-of.c
> > index 93b3368..fbdd6b4 100644
> > --- a/drivers/media/v4l2-core/v4l2-of.c
> > +++ b/drivers/media/v4l2-core/v4l2-of.c
> > @@ -19,6 +19,7 @@
> >  #include <linux/types.h>
> >  
> >  #include <media/v4l2-of.h>
> > +#include <media/v4l2-device.h>
> 
> Alphabetical order, please.

Will fix, thanks.

> >  static int v4l2_of_parse_csi_bus(const struct device_node *node,
> >  				 struct v4l2_of_endpoint *endpoint)
> > @@ -314,3 +315,70 @@ void v4l2_of_put_link(struct v4l2_of_link *link)
> >  	of_node_put(link->remote_node);
> >  }
> >  EXPORT_SYMBOL(v4l2_of_put_link);
> > +
> > +struct v4l2_subdev *v4l2_find_subdev_by_node(struct v4l2_device *v4l2_dev,
> > +					     struct device_node *node)
> > +{
> > +	struct v4l2_subdev *sd;
> > +
> > +	list_for_each_entry(sd, &v4l2_dev->subdevs, list) {
> > +		if (sd->of_node == node)
> > +			return sd;
> > +	}
> 
> The braces aren't really needed. Up to you.

I'll remove them in the next version.

regards
Philipp

Re: [PATCH 12/22] [media] tc358743: put lanes in STOP state before starting streaming

[Philipp Zabel]

Am Freitag, den 07.10.2016, 21:02 +0200 schrieb Marek Vasut:
> On 10/07/2016 06:00 PM, Philipp Zabel wrote:
> > Without calling tc358743_set_csi from the new prepare_stream callback
> > (or calling tc358743_s_dv_timings or tc358743_set_fmt from userspace
> > after stopping the stream), the i.MX6 MIPI CSI2 input fails waiting
> > for lanes to enter STOP state when streaming is started again.
> 
> What is the impact of that failure ? How does it manifest itself ?

The i.MX MIPI CSI-2 driver fails waiting for the lanes to enter stop
state. If the error is ignored, stream startup is not reliable.

I'll handle this a bit differently in the next version.

regards
Philipp

Re: [16/22] ARM: dts: nitrogen6x: Add dtsi for BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver board

[Philipp Zabel]

Hi Gary,

Am Montag, den 10.10.2016, 17:48 +0200 schrieb Gary Bisson:
> Hi Philipp, All,
> 
> On Fri, Oct 07, 2016 at 06:01:01PM +0200, Philipp Zabel wrote:
> > Add device tree nodes for the BD_HDMI_MIPI HDMI to MIPI CSI-2 receiver
> > board with a TC358743 connected to the Nitrogen6X MIPI CSI-2 input
> > connector.
> 
> I've tested this series on my Nitrogen6x + BD_HDMI_MIPI daughter board
> and have a few questions.
> 
> First, why is the tc358743 node in a separate dtsi file? Is this in
> order to avoid a failed probe during bootup if the daughter board is
> not present? Is this what should be done for every capture device that
> targets this platform (like the OV5640 or OV5642)?
> 
> Can you provide some details on your testing procedure? In my case I've
> reached a point where I get the same 'media-ctl --print-dot' output as
> the one from your cover letter but I can't seem to set the EDID nor to
> have a gstreamer pipeline (to fakesink).

If I force EDID on the sender, it also works without setting EDID on the
tc358743:

# Make the detected dv timings current on TC358743
media-ctl --set-dv '"tc358743 1-000f":0'

# Enable link between TC358743 and MIPI CSI-2 receiver, set format
media-ctl --links '"tc358743 1-000f":0->"mipi-csi2":0[1]'
media-ctl --set-v4l2 '"tc358743 1-000f":0[fmt:UYVY/1920x1080]'

# Enable link between MIPI CSI-2 receiver and mux, set format,
# this switches the mux input to MIPI CSI-2
media-ctl --links '"mipi-csi2":1->"mipi_ipu1_mux":0[1]'
media-ctl --set-v4l2 '"mipi-csi2":1[fmt:UYVY/1920x1080]'

# Enable link between mux and CSI, set format
media-ctl --links '"mipi_ipu1_mux":2->"IPU0 CSI0":0[1]'
media-ctl --set-v4l2 '"mipi_ipu1_mux":2[fmt:UYVY/1920x1080]'
# this disables frame skipping and sets the nominal frame interval
# that will be returned by v4l2-ctl --get-parm
media-ctl --set-v4l2 '"IPU0 CSI0":0[fmt:UYVY/1920x1080@1/60]'

# Set capture format at CSI output
media-ctl --set-v4l2 '"IPU0 CSI0":1[fmt:UYVY2X8/960x540]'

I'd like to handle this format propagation in libv4l2.

# Capture
gst-launch-1.0 v4l2src device=/dev/video0 io-mode=dmabuf ! video/x-raw,format=YUY2,framerate=30/1 ! kmssink

>  All the EDID v4l2-ctl commands
> return "Inappropriate ioctl for device".

The capture driver currently does not collect its subdevices' controls
to present them on the main video device. Also this would only work for
controls that appear only once in the configured pipeline.

I once hacked v4l2-ctl to allow it to directly set subdev controls on
the /dev/v4l2-subdev nodes: https://patchwork.kernel.org/patch/6097201/
but the correct solution seems to be to have a subdevice specific
QUERYCAP alternative.

> Do not hesitate to CC me to Boundary Devices related patches so I can
> test them and give some feedback.

I'll do that, thanks in advance for testing.

regards
Philipp

Re: [PATCH 02/22] [media] v4l2-async: allow subdevices to add further subdevices to the notifier waiting list

[Ian Arkver]

On 14/10/16 16:48, Philipp Zabel wrote:
> Am Samstag, den 08.10.2016, 01:43 +0300 schrieb Sakari Ailus:
> [...]
> [...]
>>> diff --git a/include/media/v4l2-async.h b/include/media/v4l2-async.h
>>> index 8e2a236..e4e4b11 100644
>>> --- a/include/media/v4l2-async.h
>>> +++ b/include/media/v4l2-async.h
>>> @@ -114,6 +114,18 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
>>>   				 struct v4l2_async_notifier *notifier);
>>>   
>>>   /**
>>> + * __v4l2_async_notifier_add_subdev - adds a subdevice to the notifier waitlist
>>> + *
>>> + * @v4l2_notifier: notifier the calling subdev is bound to
>> s/v4l2_//
> I'd be happy to, but why should the v4l2 prefix be removed?
>
> regards
> Philipp
I think Sakari is just pointing out that the comment doesn't match the 
function argument name.

Regards,
Ian

Re: [PATCH 02/22] [media] v4l2-async: allow subdevices to add further subdevices to the notifier waiting list

[Philipp Zabel]

Am Freitag, den 14.10.2016, 18:06 +0100 schrieb Ian Arkver:
> On 14/10/16 16:48, Philipp Zabel wrote:
> > Am Samstag, den 08.10.2016, 01:43 +0300 schrieb Sakari Ailus:
> > [...]
> > [...]
> >>> diff --git a/include/media/v4l2-async.h b/include/media/v4l2-async.h
> >>> index 8e2a236..e4e4b11 100644
> >>> --- a/include/media/v4l2-async.h
> >>> +++ b/include/media/v4l2-async.h
> >>> @@ -114,6 +114,18 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev,
> >>>   				 struct v4l2_async_notifier *notifier);
> >>>   
> >>>   /**
> >>> + * __v4l2_async_notifier_add_subdev - adds a subdevice to the notifier waitlist
> >>> + *
> >>> + * @v4l2_notifier: notifier the calling subdev is bound to
> >> s/v4l2_//
> > I'd be happy to, but why should the v4l2 prefix be removed?
> >
> > regards
> > Philipp
> I think Sakari is just pointing out that the comment doesn't match the 
> function argument name.

Ouch, that's very obvious now that I understand :)
Thank you for pointing this out.

regards
Philipp

Re: [PATCH 18/22] [media] imx-ipuv3-csi: support downsizing

[Marek Vasut]

On 10/14/2016 05:48 PM, Philipp Zabel wrote:
> Am Freitag, den 07.10.2016, 21:01 +0200 schrieb Marek Vasut:
>> On 10/07/2016 06:01 PM, Philipp Zabel wrote:
>>> Add support for the CSI internal horizontal and vertical downsizing.
>>>
>>> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
>>> ---
>>>  drivers/media/platform/imx/imx-ipuv3-csi.c | 20 ++++++++++++++------
>>>  1 file changed, 14 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/drivers/media/platform/imx/imx-ipuv3-csi.c b/drivers/media/platform/imx/imx-ipuv3-csi.c
>>> index 699460e6..e8a6a7b 100644
>>> --- a/drivers/media/platform/imx/imx-ipuv3-csi.c
>>> +++ b/drivers/media/platform/imx/imx-ipuv3-csi.c
>>> @@ -167,8 +167,16 @@ static int ipucsi_subdev_set_format(struct v4l2_subdev *sd,
>>>  		width = clamp_t(unsigned int, sdformat->format.width, 16, 8192);
>>>  		height = clamp_t(unsigned int, sdformat->format.height, 16, 4096);
>>>  	} else {
>>> -		width = ipucsi->format_mbus[0].width;
>>> -		height = ipucsi->format_mbus[0].height;
>>> +		if (sdformat->format.width <
>>> +		    (ipucsi->format_mbus[0].width * 3 / 4))
>>> +			width = ipucsi->format_mbus[0].width / 2;
>>> +		else
>>> +			width = ipucsi->format_mbus[0].width;
>>> +		if (sdformat->format.height <
>>> +		    (ipucsi->format_mbus[0].height * 3 / 4))
>>> +			height = ipucsi->format_mbus[0].height / 2;
>>> +		else
>>> +			height = ipucsi->format_mbus[0].height;
>>>  	}
>>>  
>>>  	mbusformat = __ipucsi_get_pad_format(sd, cfg, sdformat->pad,
>>> @@ -212,14 +220,14 @@ static int ipucsi_subdev_s_stream(struct v4l2_subdev *sd, int enable)
>>>  		window.width = fmt[0].width;
>>>  		window.height = fmt[0].height;
>>>  		ipu_csi_set_window(ipucsi->csi, &window);
>>> +		ipu_csi_set_downsize(ipucsi->csi,
>>> +				     fmt[0].width == 2 * fmt[1].width,
>>> +				     fmt[0].height == 2 * fmt[1].height);
>>>  
>>>  		/* Is CSI data source MCT (MIPI)? */
>>>  		mux_mct = (mbus_config.type == V4L2_MBUS_CSI2);
>>> -
>>>  		ipu_set_csi_src_mux(ipucsi->ipu, ipucsi->id, mux_mct);
>>> -		if (mux_mct)
>>> -			ipu_csi_set_mipi_datatype(ipucsi->csi, /*VC*/ 0,
>>> -						  &fmt[0]);
>>> +		ipu_csi_set_mipi_datatype(ipucsi->csi, /*VC*/ 0, &fmt[0]);
>>
>> This probably needs fixing , so that the correct VC is passed in ?
> 
> Absolutely, right now I don't know how though.
> We are still missing API to set the MIPI CSI-2 virtual channel.

Right. And since most cameras use VC0 anyway, it's unlikely anyone will
be severely affected by this, so this shouldn't be considered a blocker
for this patchset. Maybe add a comment, something along the lines of
"FIXME: We are still missing an API for setting VC != 0" .


-- 
Best regards,
Marek Vasut

Re: [PATCH 04/22] [media] v4l2-subdev.h: add prepare_stream op

[Sakari Ailus]

On Fri, Oct 14, 2016 at 05:48:43PM +0200, Philipp Zabel wrote:
> Am Samstag, den 08.10.2016, 02:16 +0300 schrieb Sakari Ailus:
> > Hi Philipp,
> > 
> > On Fri, Oct 07, 2016 at 06:00:49PM +0200, Philipp Zabel wrote:
> > > In some cases, for example MIPI CSI-2 input on i.MX6, the sending and
> > > receiving subdevice need to be prepared in lock-step before the actual
> > > streaming can start. In the i.MX6 MIPI CSI-2 case, the sender needs to
> > > put its MIPI CSI-2 transmitter lanes into stop state, and the receiver
> > > needs to configure its D-PHY and detect the stop state on all active
> > > lanes. Only then the sender can be enabled to stream data and the
> > > receiver can lock its PLL to the clock lane.
> > 
> > Is there a need to explicitly control this? Shouldn't this already be the
> > case when the transmitting device is powered on and is not streaming?
> 
> Even if the transmitter is expected to keep the lanes in this stop state
> all the time while the subdevice is powered but not streaming, I still
> have to wait for stop state detection before enabling the transmitter,
> and only then enable the reciever.
> I'll remove the prepare_streaming callback in the next version and
> instead let the subdevices propagate s_stream upstream instead in the
> next version.

Ack.

As discussed, I'll provide a patch to document this behaviour on CSI-2. I
believe the current drivers implicitly implement it but you're the first one
to ask the question. :-)

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: [PATCH 07/22] [media] imx: Add IPUv3 media common code

[Ian Arkver]

On 07/10/16 17:00, Philipp Zabel wrote:
> From: Sascha Hauer <s.hauer@pengutronix.de>
>
> Add video4linux API routines common to drivers for units that
> accept or provide video data via the i.MX IPU IDMAC channels,
> such as capture, mem2mem scaler or deinterlacer drivers.
>
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> ---
>   drivers/media/platform/imx/Kconfig   |   3 +
>   drivers/media/platform/imx/Makefile  |   1 +
>   drivers/media/platform/imx/imx-ipu.c | 321 +++++++++++++++++++++++++++++++++++
>   drivers/media/platform/imx/imx-ipu.h |  34 ++++
>   4 files changed, 359 insertions(+)
>   create mode 100644 drivers/media/platform/imx/imx-ipu.c
>   create mode 100644 drivers/media/platform/imx/imx-ipu.h
>
> diff --git a/drivers/media/platform/imx/Kconfig b/drivers/media/platform/imx/Kconfig
> index 3bd699c..1662bb0b 100644
> --- a/drivers/media/platform/imx/Kconfig
> +++ b/drivers/media/platform/imx/Kconfig
> @@ -5,3 +5,6 @@ config MEDIA_IMX
>   	---help---
>   	  This driver provides a SoC wide media controller device that all
>   	  multimedia components in i.MX5 and i.MX6 SoCs can register with.
> +
> +config VIDEO_IMX_IPU_COMMON
> +	tristate
> diff --git a/drivers/media/platform/imx/Makefile b/drivers/media/platform/imx/Makefile
> index 74bed76..0ba601a 100644
> --- a/drivers/media/platform/imx/Makefile
> +++ b/drivers/media/platform/imx/Makefile
> @@ -1 +1,2 @@
>   obj-$(CONFIG_MEDIA_IMX)			+= imx-media.o
> +obj-$(CONFIG_VIDEO_IMX_IPU_COMMON)	+= imx-ipu.o
> diff --git a/drivers/media/platform/imx/imx-ipu.c b/drivers/media/platform/imx/imx-ipu.c
> new file mode 100644
> index 0000000..da1deb0
> --- /dev/null
> +++ b/drivers/media/platform/imx/imx-ipu.c
> @@ -0,0 +1,321 @@
> +/*
> + * i.MX IPUv3 common v4l2 support
> + *
> + * Copyright (C) 2011 Pengutronix, Sascha Hauer <kernel@pengutronix.de>
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version 2
> + * of the License, or (at your option) any later version.
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +#include <linux/module.h>
> +#include <media/v4l2-common.h>
> +#include <media/v4l2-dev.h>
> +#include <media/v4l2-ioctl.h>
> +
> +#include "imx-ipu.h"
> +
> +/*
> + * These formats are in order of preference: interleaved YUV first,
> + * because those are the most bandwidth efficient, followed by
> + * chroma-interleaved formats, and planar formats last.
> + * In each category, YUV 4:2:0 may be preferrable to 4:2:2 for bandwidth
> + * reasons, if the IDMAC channel supports double read/write reduction
> + * (all write channels, VDIC read channels).
> + */
> +static struct ipu_fmt ipu_fmt_yuv[] = {
> +	{
> +		.fourcc = V4L2_PIX_FMT_YUYV,
> +		.bytes_per_pixel = 2,
> +	}, {
> +		.fourcc = V4L2_PIX_FMT_UYVY,
> +		.bytes_per_pixel = 2,
> +	}, {
> +		.fourcc = V4L2_PIX_FMT_NV12,
> +		.bytes_per_pixel = 1,
> +	}, {
> +		.fourcc = V4L2_PIX_FMT_NV16,
> +		.bytes_per_pixel = 1,
> +	}, {
> +		.fourcc = V4L2_PIX_FMT_YUV420,
> +		.bytes_per_pixel = 1,
> +	}, {
> +		.fourcc = V4L2_PIX_FMT_YVU420,
> +		.bytes_per_pixel = 1,
> +	}, {
> +		.fourcc = V4L2_PIX_FMT_YUV422P,
> +		.bytes_per_pixel = 1,
> +	},
> +};
> +
> +static struct ipu_fmt ipu_fmt_rgb[] = {
> +	{
> +		.fourcc = V4L2_PIX_FMT_RGB32,
> +		.bytes_per_pixel = 4,
> +	}, {
> +		.fourcc = V4L2_PIX_FMT_RGB24,
> +		.bytes_per_pixel = 3,
> +	}, {
> +		.fourcc = V4L2_PIX_FMT_BGR24,
> +		.bytes_per_pixel = 3,
> +	}, {
> +		.fourcc = V4L2_PIX_FMT_RGB565,
> +		.bytes_per_pixel = 2,
> +	},
> +	{
> +		.fourcc = V4L2_PIX_FMT_BGR32,
> +		.bytes_per_pixel = 4,
> +	},
> +};
>
Maybe a trivial comment, but is it worthwhile to constify these two?

Regards,
Ian