[PATCH v3 0/19] s5p-fimc driver conversion to media controller and control framework

[PATCH v3 0/19] s5p-fimc driver conversion to media controller and control framework

[Sylwester Nawrocki]

Hello,

following is a third version of a patchset converting s5p-fimc driver 
to the media controller API and control framework.

Except fixing minor bugs in the sensor registration code, the first patch
adding power management support has been significantly reworked. 
There is no yet support for a per frame power gating, I'd like to add
it in 3.2 as it seem to require modfications of v4l-mem2mem framework.

I'd like to get this patchset in v3.1, unless there are any issues pointed out. 

The changeset can be pulled from (within a few hours from now, after the
repository synchronisation cron job runs):

http://git.infradead.org/users/kmpark/linux-2.6-samsung
branch: s5p-fimc-next

Changes since v2:
- reworked (runtime) power management;
- added pm_runtime_get_sync/pm_runtime_put around sensor registration
  code so the clock for sensors is enabled during host driver's probe();
- reworked try_crop operation handler to support multiple of the prescaler
  ratio relationship constraint for format at the sink pad;
- corrected fimc_md_unregister_entities() function


Below is an original cover letter of v2.

-----------------------------------------------------------------------------
In this version the ioctl handlers for format setting at the video node and
the FIMC subdev has been reworked so they use common functions for capture
data format adjustment to the hardware capabilities. This prevent trouble
from any differences in handling ioctls at the subdev and the video node
when those are used simultaneously.

Except that I have killed a few bugs that jumped out when the driver was tried
to be used as a kernel module. 

Additionally to allow the driver to be used in V4L2 video node compatibility
mode (when sensor subdev is configured by the host driver rather that directly
by the applications through /dev/v4l-subdev*) a sysfs entry is added.
All that needs to be done to disable sensors configuration from /dev/video*
node level, in order to use sub-device user space API, is to write correct
string to s5p-fimc-md platform device "subdev_conf_mode" sysfs entry, e.g.

echo sub-dev > /sys/devices/platform/s5p-fimc-md/subdev_conf_mode
or to revert:
echo vid-dev > /sys/devices/platform/s5p-fimc-md/subdev_conf_mode

The following procedure is adopted for format and crop/composition
configuration at the FIMC.{n} (capture) subdevs:

1) set format at sink pad (this will reset sink crop rectangle)
2) set crop rectangle at sink pad (optional)
3) set rotate control (optional)
4) set crop (composition) at source pad (optional). Here scaling constraints
   are checked according to whether sink pad crop has been set or not and
   whether 90/270 deg rotation is set.
5) set format at source pad
6) set format at corresponding video node

Rotation can also be changed while streaming, in this case when 90/270 deg
rotation is attempted to be set and scaling bounds are exceeded 
(max. 64x downscaling) s_ctrl returns EINVAL.

I have tried this driver with v4l-compliance and it returned only 1 fail
on S/G_PRIORITY.
								
This patch set depends on my previous s5p-fimc bugfixes, available at: 
http://git.infradead.org/users/kmpark/linux-2.6-samsung/shortlog/refs/heads/s5p-fimc

as well as the control framework updates from Hans Verkuil:
http://www.spinics.net/lists/linux-media/msg33552.html
(patch 1...6, 8, 11).

Full source tree can be found at:
 http://git.infradead.org/users/kmpark/linux-2.6-samsung
 branch: s5p-fimc-next

I have skipped patch one patch in this series comparing to first version:
s5p-fimc: Add support for runtime PM in the mem-to-mem driver
This patch is available in the above git repository. 
I need to work a bit more on that one.

Comments and suggestions are welcome!
------------------------------------------------------------------------------


Thanks,
--
Sylwester Nawrocki
Samsung Poland R&D Center

[PATCH v3 01/19] s5p-fimc: Add support for runtime PM in the mem-to-mem driver

[Sylwester Nawrocki]

Only the FIMC entity memory-to-memory operation is fully covered,
for the camera capture suspend/resume only stubs are provided.
It's all what is needed to enable the driver on EXYNOS4 series with
power domain driver enabled. Camera capture pipeline suspend/resume
is covered by a separate patch.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |   18 ++
 drivers/media/video/s5p-fimc/fimc-core.c    |  273 ++++++++++++++++++++-------
 drivers/media/video/s5p-fimc/fimc-core.h    |   16 +-
 drivers/media/video/s5p-fimc/fimc-reg.c     |    2 +-
 4 files changed, 231 insertions(+), 78 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index 0d730e5..ea74e4b 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -17,6 +17,7 @@
 #include <linux/interrupt.h>
 #include <linux/device.h>
 #include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
 #include <linux/list.h>
 #include <linux/slab.h>
 #include <linux/clk.h>
@@ -196,6 +197,16 @@ static int fimc_stop_capture(struct fimc_dev *fimc)
 	return 0;
 }
 
+int fimc_capture_suspend(struct fimc_dev *fimc)
+{
+	return -EBUSY;
+}
+
+int fimc_capture_resume(struct fimc_dev *fimc)
+{
+	return 0;
+}
+
 static int start_streaming(struct vb2_queue *q)
 {
 	struct fimc_ctx *ctx = q->drv_priv;
@@ -381,9 +392,14 @@ static int fimc_capture_open(struct file *file)
 	if (fimc_m2m_active(fimc))
 		return -EBUSY;
 
+	ret = pm_runtime_get_sync(&fimc->pdev->dev);
+	if (ret)
+		return ret;
+
 	if (++fimc->vid_cap.refcnt == 1) {
 		ret = fimc_isp_subdev_init(fimc, 0);
 		if (ret) {
+			pm_runtime_put_sync(&fimc->pdev->dev);
 			fimc->vid_cap.refcnt--;
 			return -EIO;
 		}
@@ -411,6 +427,8 @@ static int fimc_capture_close(struct file *file)
 		fimc_subdev_unregister(fimc);
 	}
 
+	pm_runtime_put_sync(&fimc->pdev->dev);
+
 	return 0;
 }
 
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index aa55066..12e2d19 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -18,6 +18,7 @@
 #include <linux/interrupt.h>
 #include <linux/device.h>
 #include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
 #include <linux/list.h>
 #include <linux/io.h>
 #include <linux/slab.h>
@@ -301,7 +302,6 @@ int fimc_set_scaler_info(struct fimc_ctx *ctx)
 static void fimc_m2m_job_finish(struct fimc_ctx *ctx, int vb_state)
 {
 	struct vb2_buffer *src_vb, *dst_vb;
-	struct fimc_dev *fimc = ctx->fimc_dev;
 
 	if (!ctx || !ctx->m2m_ctx)
 		return;
@@ -312,43 +312,52 @@ static void fimc_m2m_job_finish(struct fimc_ctx *ctx, int vb_state)
 	if (src_vb && dst_vb) {
 		v4l2_m2m_buf_done(src_vb, vb_state);
 		v4l2_m2m_buf_done(dst_vb, vb_state);
-		v4l2_m2m_job_finish(fimc->m2m.m2m_dev, ctx->m2m_ctx);
+		v4l2_m2m_job_finish(ctx->fimc_dev->m2m.m2m_dev,
+				    ctx->m2m_ctx);
 	}
 }
 
 /* Complete the transaction which has been scheduled for execution. */
-static void fimc_m2m_shutdown(struct fimc_ctx *ctx)
+static int fimc_m2m_shutdown(struct fimc_ctx *ctx)
 {
 	struct fimc_dev *fimc = ctx->fimc_dev;
 	int ret;
 
 	if (!fimc_m2m_pending(fimc))
-		return;
+		return 0;
 
 	fimc_ctx_state_lock_set(FIMC_CTX_SHUT, ctx);
 
 	ret = wait_event_timeout(fimc->irq_queue,
 			   !fimc_ctx_state_is_set(FIMC_CTX_SHUT, ctx),
 			   FIMC_SHUTDOWN_TIMEOUT);
-	/*
-	 * In case of a timeout the buffers are not released in the interrupt
-	 * handler so return them here with the error flag set, if there are
-	 * any on the queue.
-	 */
-	if (ret == 0)
-		fimc_m2m_job_finish(ctx, VB2_BUF_STATE_ERROR);
+
+	return ret == 0 ? -ETIMEDOUT : ret;
+}
+
+static int start_streaming(struct vb2_queue *q)
+{
+	struct fimc_ctx *ctx = q->drv_priv;
+	int ret;
+
+	ret = pm_runtime_get_sync(&ctx->fimc_dev->pdev->dev);
+	return ret > 0 ? 0 : ret;
 }
 
 static int stop_streaming(struct vb2_queue *q)
 {
 	struct fimc_ctx *ctx = q->drv_priv;
+	int ret;
 
-	fimc_m2m_shutdown(ctx);
+	ret = fimc_m2m_shutdown(ctx);
+	if (ret == -ETIMEDOUT)
+		fimc_m2m_job_finish(ctx, VB2_BUF_STATE_ERROR);
 
+	pm_runtime_put(&ctx->fimc_dev->pdev->dev);
 	return 0;
 }
 
-static void fimc_capture_irq_handler(struct fimc_dev *fimc)
+void fimc_capture_irq_handler(struct fimc_dev *fimc)
 {
 	struct fimc_vid_cap *cap = &fimc->vid_cap;
 	struct fimc_vid_buffer *v_buf;
@@ -403,7 +412,7 @@ static void fimc_capture_irq_handler(struct fimc_dev *fimc)
 	    fimc_hw_get_frame_index(fimc), cap->active_buf_cnt);
 }
 
-static irqreturn_t fimc_isr(int irq, void *priv)
+static irqreturn_t fimc_irq_handler(int irq, void *priv)
 {
 	struct fimc_dev *fimc = priv;
 	struct fimc_vid_cap *cap = &fimc->vid_cap;
@@ -411,9 +420,17 @@ static irqreturn_t fimc_isr(int irq, void *priv)
 
 	fimc_hw_clear_irq(fimc);
 
+	spin_lock(&fimc->slock);
+
 	if (test_and_clear_bit(ST_M2M_PEND, &fimc->state)) {
+		if (test_and_clear_bit(ST_M2M_SUSPENDING, &fimc->state)) {
+			set_bit(ST_M2M_SUSPENDED, &fimc->state);
+			wake_up(&fimc->irq_queue);
+			goto out;
+		}
 		ctx = v4l2_m2m_get_curr_priv(fimc->m2m.m2m_dev);
 		if (ctx != NULL) {
+			spin_unlock(&fimc->slock);
 			fimc_m2m_job_finish(ctx, VB2_BUF_STATE_DONE);
 
 			spin_lock(&ctx->slock);
@@ -423,21 +440,18 @@ static irqreturn_t fimc_isr(int irq, void *priv)
 			}
 			spin_unlock(&ctx->slock);
 		}
-
 		return IRQ_HANDLED;
-	}
-
-	spin_lock(&fimc->slock);
-
-	if (test_bit(ST_CAPT_PEND, &fimc->state)) {
-		fimc_capture_irq_handler(fimc);
+	} else {
+		if (test_bit(ST_CAPT_PEND, &fimc->state)) {
+			fimc_capture_irq_handler(fimc);
 
-		if (cap->active_buf_cnt == 1) {
-			fimc_deactivate_capture(fimc);
-			clear_bit(ST_CAPT_STREAM, &fimc->state);
+			if (cap->active_buf_cnt == 1) {
+				fimc_deactivate_capture(fimc);
+				clear_bit(ST_CAPT_STREAM, &fimc->state);
+			}
 		}
 	}
-
+out:
 	spin_unlock(&fimc->slock);
 	return IRQ_HANDLED;
 }
@@ -635,10 +649,10 @@ static void fimc_dma_run(void *priv)
 		return;
 
 	fimc = ctx->fimc_dev;
-
-	spin_lock_irqsave(&ctx->slock, flags);
+	spin_lock_irqsave(&fimc->slock, flags);
 	set_bit(ST_M2M_PEND, &fimc->state);
 
+	spin_lock(&ctx->slock);
 	ctx->state |= (FIMC_SRC_ADDR | FIMC_DST_ADDR);
 	ret = fimc_prepare_config(ctx, ctx->state);
 	if (ret)
@@ -649,8 +663,6 @@ static void fimc_dma_run(void *priv)
 		ctx->state |= FIMC_PARAMS;
 		fimc->m2m.ctx = ctx;
 	}
-
-	spin_lock(&fimc->slock);
 	fimc_hw_set_input_addr(fimc, &ctx->s_frame.paddr);
 
 	if (ctx->state & FIMC_PARAMS) {
@@ -680,10 +692,9 @@ static void fimc_dma_run(void *priv)
 	ctx->state &= (FIMC_CTX_M2M | FIMC_CTX_CAP |
 		       FIMC_SRC_FMT | FIMC_DST_FMT);
 	fimc_hw_activate_input_dma(fimc, true);
-	spin_unlock(&fimc->slock);
-
 dma_unlock:
-	spin_unlock_irqrestore(&ctx->slock, flags);
+	spin_unlock(&ctx->slock);
+	spin_unlock_irqrestore(&fimc->slock, flags);
 }
 
 static void fimc_job_abort(void *priv)
@@ -762,6 +773,7 @@ static struct vb2_ops fimc_qops = {
 	.wait_prepare	 = fimc_unlock,
 	.wait_finish	 = fimc_lock,
 	.stop_streaming	 = stop_streaming,
+	.start_streaming = start_streaming,
 };
 
 static int fimc_m2m_querycap(struct file *file, void *priv,
@@ -873,7 +885,6 @@ int fimc_vidioc_try_fmt_mplane(struct file *file, void *priv,
 	u32 max_width, mod_x, mod_y, mask;
 	int i, is_output = 0;
 
-
 	if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
 		if (fimc_ctx_state_is_set(FIMC_CTX_CAP, ctx))
 			return -EINVAL;
@@ -1408,9 +1419,6 @@ static int fimc_m2m_open(struct file *file)
 	if (fimc->vid_cap.refcnt > 0)
 		return -EBUSY;
 
-	fimc->m2m.refcnt++;
-	set_bit(ST_OUTDMA_RUN, &fimc->state);
-
 	ctx = kzalloc(sizeof *ctx, GFP_KERNEL);
 	if (!ctx)
 		return -ENOMEM;
@@ -1434,6 +1442,9 @@ static int fimc_m2m_open(struct file *file)
 		return err;
 	}
 
+	if (fimc->m2m.refcnt++ == 0)
+		set_bit(ST_M2M_RUN, &fimc->state);
+
 	return 0;
 }
 
@@ -1446,10 +1457,10 @@ static int fimc_m2m_release(struct file *file)
 		task_pid_nr(current), fimc->state, fimc->m2m.refcnt);
 
 	v4l2_m2m_ctx_release(ctx->m2m_ctx);
-	kfree(ctx);
-	if (--fimc->m2m.refcnt <= 0)
-		clear_bit(ST_OUTDMA_RUN, &fimc->state);
 
+	if (--fimc->m2m.refcnt <= 0)
+		clear_bit(ST_M2M_RUN, &fimc->state);
+	kfree(ctx);
 	return 0;
 }
 
@@ -1551,7 +1562,7 @@ err_m2m_r1:
 	return ret;
 }
 
-static void fimc_unregister_m2m_device(struct fimc_dev *fimc)
+void fimc_unregister_m2m_device(struct fimc_dev *fimc)
 {
 	if (fimc) {
 		v4l2_m2m_release(fimc->m2m.m2m_dev);
@@ -1561,14 +1572,12 @@ static void fimc_unregister_m2m_device(struct fimc_dev *fimc)
 	}
 }
 
-static void fimc_clk_release(struct fimc_dev *fimc)
+static void fimc_clk_put(struct fimc_dev *fimc)
 {
 	int i;
 	for (i = 0; i < fimc->num_clocks; i++) {
-		if (fimc->clock[i]) {
-			clk_disable(fimc->clock[i]);
+		if (fimc->clock[i])
 			clk_put(fimc->clock[i]);
-		}
 	}
 }
 
@@ -1577,15 +1586,50 @@ static int fimc_clk_get(struct fimc_dev *fimc)
 	int i;
 	for (i = 0; i < fimc->num_clocks; i++) {
 		fimc->clock[i] = clk_get(&fimc->pdev->dev, fimc_clocks[i]);
-
-		if (!IS_ERR_OR_NULL(fimc->clock[i])) {
-			clk_enable(fimc->clock[i]);
+		if (!IS_ERR_OR_NULL(fimc->clock[i]))
 			continue;
-		}
 		dev_err(&fimc->pdev->dev, "failed to get fimc clock: %s\n",
 			fimc_clocks[i]);
 		return -ENXIO;
 	}
+
+	return 0;
+}
+
+static int fimc_m2m_suspend(struct fimc_dev *fimc)
+{
+	unsigned long flags;
+	int timeout;
+
+	spin_lock_irqsave(&fimc->slock, flags);
+	if (!fimc_m2m_pending(fimc)) {
+		spin_unlock_irqrestore(&fimc->slock, flags);
+		return 0;
+	}
+	clear_bit(ST_M2M_SUSPENDED, &fimc->state);
+	set_bit(ST_M2M_SUSPENDING, &fimc->state);
+	spin_unlock_irqrestore(&fimc->slock, flags);
+
+	timeout = wait_event_timeout(fimc->irq_queue,
+			     test_bit(ST_M2M_SUSPENDED, &fimc->state),
+			     FIMC_SHUTDOWN_TIMEOUT);
+
+	clear_bit(ST_M2M_SUSPENDING, &fimc->state);
+	return timeout == 0 ? -EAGAIN: 0;
+}
+
+static int fimc_m2m_resume(struct fimc_dev *fimc)
+{
+	unsigned long flags;
+
+	/* Need to fully setup hardware at first run. */
+	spin_lock_irqsave(&fimc->slock, flags);
+	fimc->m2m.ctx = NULL;
+	spin_unlock_irqrestore(&fimc->slock, flags);
+
+	if (test_and_clear_bit(ST_M2M_SUSPENDED, &fimc->state))
+		fimc_m2m_job_finish(fimc->m2m.ctx,
+				    VB2_BUF_STATE_ERROR);
 	return 0;
 }
 
@@ -1614,11 +1658,13 @@ static int fimc_probe(struct platform_device *pdev)
 		return -ENOMEM;
 
 	fimc->id = pdev->id;
+
 	fimc->variant = drv_data->variant[fimc->id];
 	fimc->pdev = pdev;
 	pdata = pdev->dev.platform_data;
 	fimc->pdata = pdata;
-	fimc->state = ST_IDLE;
+
+	set_bit(ST_LPM, &fimc->state);
 
 	init_waitqueue_head(&fimc->irq_queue);
 	spin_lock_init(&fimc->slock);
@@ -1655,27 +1701,32 @@ static int fimc_probe(struct platform_device *pdev)
 		fimc->num_clocks++;
 	}
 
-	ret = fimc_clk_get(fimc);
-	if (ret)
-		goto err_regs_unmap;
-	clk_set_rate(fimc->clock[CLK_BUS], drv_data->lclk_frequency);
-
 	res = platform_get_resource(pdev, IORESOURCE_IRQ, 0);
 	if (!res) {
 		dev_err(&pdev->dev, "failed to get IRQ resource\n");
 		ret = -ENXIO;
-		goto err_clk;
+		goto err_regs_unmap;
 	}
 	fimc->irq = res->start;
 
-	fimc_hw_reset(fimc);
+	ret = fimc_clk_get(fimc);
+	if (ret)
+		goto err_regs_unmap;
+	clk_set_rate(fimc->clock[CLK_BUS], drv_data->lclk_frequency);
+	clk_enable(fimc->clock[CLK_BUS]);
+
+	platform_set_drvdata(pdev, fimc);
 
-	ret = request_irq(fimc->irq, fimc_isr, 0, pdev->name, fimc);
+	ret = request_irq(fimc->irq, fimc_irq_handler, 0, pdev->name, fimc);
 	if (ret) {
 		dev_err(&pdev->dev, "failed to install irq (%d)\n", ret);
 		goto err_clk;
 	}
 
+	pm_runtime_enable(&pdev->dev);
+	ret = pm_runtime_get_sync(&fimc->pdev->dev);
+	if (ret < 0)
+		goto err_clk;
 	/* Initialize contiguous memory allocator */
 	fimc->alloc_ctx = vb2_dma_contig_init_ctx(&fimc->pdev->dev);
 	if (IS_ERR(fimc->alloc_ctx)) {
@@ -1692,18 +1743,12 @@ static int fimc_probe(struct platform_device *pdev)
 		ret = fimc_register_capture_device(fimc);
 		if (ret)
 			goto err_m2m;
-		clk_disable(fimc->clock[CLK_CAM]);
 	}
-	/*
-	 * Exclude the additional output DMA address registers by masking
-	 * them out on HW revisions that provide extended capabilites.
-	 */
-	if (fimc->variant->out_buf_count > 4)
-		fimc_hw_set_dma_seq(fimc, 0xF);
 
 	dev_dbg(&pdev->dev, "%s(): fimc-%d registered successfully\n",
 		__func__, fimc->id);
 
+	pm_runtime_put(&fimc->pdev->dev);
 	return 0;
 
 err_m2m:
@@ -1711,7 +1756,7 @@ err_m2m:
 err_irq:
 	free_irq(fimc->irq, fimc);
 err_clk:
-	fimc_clk_release(fimc);
+	fimc_clk_put(fimc);
 err_regs_unmap:
 	iounmap(fimc->regs);
 err_req_region:
@@ -1723,27 +1768,105 @@ err_info:
 	return ret;
 }
 
-static int __devexit fimc_remove(struct platform_device *pdev)
+static int fimc_runtime_resume(struct device *dev)
 {
-	struct fimc_dev *fimc =
-		(struct fimc_dev *)platform_get_drvdata(pdev);
+	struct fimc_dev *fimc =	dev_get_drvdata(dev);
 
-	free_irq(fimc->irq, fimc);
+	dbg("fimc%d: state: 0x%lx", fimc->id, fimc->state);
+
+	/* Enable clocks and perform basic initalization */
+	clk_enable(fimc->clock[CLK_GATE]);
+	fimc_hw_reset(fimc);
+	if (fimc->variant->out_buf_count > 4)
+		fimc_hw_set_dma_seq(fimc, 0xF);
+
+	/* Resume the capture or mem-to-mem device */
+	if (fimc_capture_in_use(fimc))
+		return fimc_capture_resume(fimc);
+	else if (fimc_m2m_pending(fimc))
+		return fimc_m2m_resume(fimc);
+	return 0;
+}
+
+static int fimc_runtime_suspend(struct device *dev)
+{
+	struct fimc_dev *fimc =	dev_get_drvdata(dev);
+	int ret = 0;
+
+	if (fimc_capture_in_use(fimc))
+		ret = fimc_capture_suspend(fimc);
+	else
+		ret = fimc_m2m_suspend(fimc);
+	if (!ret)
+		clk_disable(fimc->clock[CLK_GATE]);
+
+	dbg("fimc%d: state: 0x%lx", fimc->id, fimc->state);
+	return ret;
+}
+
+#ifdef CONFIG_PM_SLEEP
+static int fimc_resume(struct device *dev)
+{
+	struct fimc_dev *fimc =	dev_get_drvdata(dev);
+	unsigned long flags;
+
+	dbg("fimc%d: state: 0x%lx", fimc->id, fimc->state);
+
+	/* Do not resume if the device was idle before system suspend */
+	spin_lock_irqsave(&fimc->slock, flags);
+	if (!test_and_clear_bit(ST_LPM, &fimc->state) ||
+	    (!fimc_m2m_active(fimc) && !fimc_capture_in_use(fimc))) {
+		spin_unlock_irqrestore(&fimc->slock, flags);
+		return 0;
+	}
 	fimc_hw_reset(fimc);
+	if (fimc->variant->out_buf_count > 4)
+		fimc_hw_set_dma_seq(fimc, 0xF);
+	spin_unlock_irqrestore(&fimc->slock, flags);
+
+	if (fimc_capture_in_use(fimc))
+		return fimc_capture_resume(fimc);
+
+	return fimc_m2m_resume(fimc);
+}
+
+static int fimc_suspend(struct device *dev)
+{
+	struct fimc_dev *fimc =	dev_get_drvdata(dev);
+
+	dbg("fimc%d: state: 0x%lx", fimc->id, fimc->state);
+
+	if (test_and_set_bit(ST_LPM, &fimc->state))
+		return 0;
+	if (fimc_capture_in_use(fimc))
+		return fimc_capture_suspend(fimc);
+
+	return fimc_m2m_suspend(fimc);
+}
+#endif /* CONFIG_PM_SLEEP */
+
+static int __devexit fimc_remove(struct platform_device *pdev)
+{
+	struct fimc_dev *fimc = platform_get_drvdata(pdev);
+
+	pm_runtime_disable(&pdev->dev);
+	fimc_runtime_suspend(&pdev->dev);
+	pm_runtime_set_suspended(&pdev->dev);
 
 	fimc_unregister_m2m_device(fimc);
 	fimc_unregister_capture_device(fimc);
 
-	fimc_clk_release(fimc);
-
 	vb2_dma_contig_cleanup_ctx(fimc->alloc_ctx);
 
+	clk_disable(fimc->clock[CLK_BUS]);
+	fimc_clk_put(fimc);
+	free_irq(fimc->irq, fimc);
 	iounmap(fimc->regs);
 	release_resource(fimc->regs_res);
 	kfree(fimc->regs_res);
 	kfree(fimc);
 
-	dev_info(&pdev->dev, "%s driver unloaded\n", pdev->name);
+	dev_info(&pdev->dev, "driver unloaded\n");
 	return 0;
 }
 
@@ -1906,6 +2029,11 @@ static struct platform_device_id fimc_driver_ids[] = {
 };
 MODULE_DEVICE_TABLE(platform, fimc_driver_ids);
 
+static const struct dev_pm_ops fimc_pm_ops = {
+	SET_SYSTEM_SLEEP_PM_OPS(fimc_suspend, fimc_resume)
+	SET_RUNTIME_PM_OPS(fimc_runtime_suspend, fimc_runtime_resume, NULL)
+};
+
 static struct platform_driver fimc_driver = {
 	.probe		= fimc_probe,
 	.remove	= __devexit_p(fimc_remove),
@@ -1913,6 +2041,7 @@ static struct platform_driver fimc_driver = {
 	.driver = {
 		.name	= MODULE_NAME,
 		.owner	= THIS_MODULE,
+		.pm     = &fimc_pm_ops,
 	}
 };
 
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 1f70772..21dfcac 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -48,22 +48,26 @@ enum {
 };
 
 enum fimc_dev_flags {
-	/* for m2m node */
-	ST_IDLE,
-	ST_OUTDMA_RUN,
+	ST_LPM,
+	/* m2m node */
+	ST_M2M_RUN,
 	ST_M2M_PEND,
-	/* for capture node */
+	ST_M2M_SUSPENDING,
+	ST_M2M_SUSPENDED,
+	/* capture node */
 	ST_CAPT_PEND,
 	ST_CAPT_RUN,
 	ST_CAPT_STREAM,
 	ST_CAPT_SHUT,
+	ST_CAPT_INUSE,
 };
 
-#define fimc_m2m_active(dev) test_bit(ST_OUTDMA_RUN, &(dev)->state)
+#define fimc_m2m_active(dev) test_bit(ST_M2M_RUN, &(dev)->state)
 #define fimc_m2m_pending(dev) test_bit(ST_M2M_PEND, &(dev)->state)
 
 #define fimc_capture_running(dev) test_bit(ST_CAPT_RUN, &(dev)->state)
 #define fimc_capture_pending(dev) test_bit(ST_CAPT_PEND, &(dev)->state)
+#define fimc_capture_in_use(dev) test_bit(ST_CAPT_INUSE, &(dev)->state)
 
 enum fimc_datapath {
 	FIMC_CAMERA,
@@ -644,6 +648,8 @@ void fimc_unregister_capture_device(struct fimc_dev *fimc);
 int fimc_sensor_sd_init(struct fimc_dev *fimc, int index);
 int fimc_vid_cap_buf_queue(struct fimc_dev *fimc,
 			     struct fimc_vid_buffer *fimc_vb);
+int fimc_capture_suspend(struct fimc_dev *fimc);
+int fimc_capture_resume(struct fimc_dev *fimc);
 
 /* Locking: the caller holds fimc->slock */
 static inline void fimc_activate_capture(struct fimc_ctx *ctx)
diff --git a/drivers/media/video/s5p-fimc/fimc-reg.c b/drivers/media/video/s5p-fimc/fimc-reg.c
index 4893b2d..938dadf 100644
--- a/drivers/media/video/s5p-fimc/fimc-reg.c
+++ b/drivers/media/video/s5p-fimc/fimc-reg.c
@@ -30,7 +30,7 @@ void fimc_hw_reset(struct fimc_dev *dev)
 	cfg = readl(dev->regs + S5P_CIGCTRL);
 	cfg |= (S5P_CIGCTRL_SWRST | S5P_CIGCTRL_IRQ_LEVEL);
 	writel(cfg, dev->regs + S5P_CIGCTRL);
-	udelay(1000);
+	udelay(10);
 
 	cfg = readl(dev->regs + S5P_CIGCTRL);
 	cfg &= ~S5P_CIGCTRL_SWRST;
-- 
1.7.5.4

[PATCH v3 02/19] s5p-fimc: Add media entity initialization

[Sylwester Nawrocki]

Add intialization of the media entities for video capture
and mem-to-mem video nodes.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |   28 ++++++++++++++++----------
 drivers/media/video/s5p-fimc/fimc-core.c    |   27 +++++++++++++++----------
 drivers/media/video/s5p-fimc/fimc-core.h    |    4 +++
 3 files changed, 37 insertions(+), 22 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index ea74e4b..c9be15c 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -838,9 +838,8 @@ int fimc_register_capture_device(struct fimc_dev *fimc)
 	fr->width = fr->f_width = fr->o_width = 640;
 	fr->height = fr->f_height = fr->o_height = 480;
 
-	if (!v4l2_dev->name[0])
-		snprintf(v4l2_dev->name, sizeof(v4l2_dev->name),
-			 "%s.capture", dev_name(&fimc->pdev->dev));
+	snprintf(v4l2_dev->name, sizeof(v4l2_dev->name),
+		 "%s.capture", dev_name(&fimc->pdev->dev));
 
 	ret = v4l2_device_register(NULL, v4l2_dev);
 	if (ret)
@@ -852,11 +851,11 @@ int fimc_register_capture_device(struct fimc_dev *fimc)
 		goto err_v4l2_reg;
 	}
 
-	snprintf(vfd->name, sizeof(vfd->name), "%s:cap",
-		 dev_name(&fimc->pdev->dev));
+	strlcpy(vfd->name, v4l2_dev->name, sizeof(vfd->name));
 
 	vfd->fops	= &fimc_capture_fops;
 	vfd->ioctl_ops	= &fimc_capture_ioctl_ops;
+	vfd->v4l2_dev	= v4l2_dev;
 	vfd->minor	= -1;
 	vfd->release	= video_device_release;
 	vfd->lock	= &fimc->lock;
@@ -886,6 +885,11 @@ int fimc_register_capture_device(struct fimc_dev *fimc)
 
 	vb2_queue_init(q);
 
+	fimc->vid_cap.vd_pad.flags = MEDIA_PAD_FL_SINK;
+	ret = media_entity_init(&vfd->entity, 1, &fimc->vid_cap.vd_pad, 0);
+	if (ret)
+		goto err_ent;
+
 	ret = video_register_device(vfd, VFL_TYPE_GRABBER, -1);
 	if (ret) {
 		v4l2_err(v4l2_dev, "Failed to register video device\n");
@@ -895,10 +899,11 @@ int fimc_register_capture_device(struct fimc_dev *fimc)
 	v4l2_info(v4l2_dev,
 		  "FIMC capture driver registered as /dev/video%d\n",
 		  vfd->num);
-
 	return 0;
 
 err_vd_reg:
+	media_entity_cleanup(&vfd->entity);
+err_ent:
 	video_device_release(vfd);
 err_v4l2_reg:
 	v4l2_device_unregister(v4l2_dev);
@@ -910,10 +915,11 @@ err_info:
 
 void fimc_unregister_capture_device(struct fimc_dev *fimc)
 {
-	struct fimc_vid_cap *capture = &fimc->vid_cap;
+	struct video_device *vfd = fimc->vid_cap.vfd;
 
-	if (capture->vfd)
-		video_unregister_device(capture->vfd);
-
-	kfree(capture->ctx);
+	if (vfd) {
+		media_entity_cleanup(&vfd->entity);
+		video_unregister_device(vfd);
+	}
+	kfree(fimc->vid_cap.ctx);
 }
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index 12e2d19..c97494d 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -1507,10 +1507,8 @@ static int fimc_register_m2m_device(struct fimc_dev *fimc)
 	pdev = fimc->pdev;
 	v4l2_dev = &fimc->m2m.v4l2_dev;
 
-	/* set name if it is empty */
-	if (!v4l2_dev->name[0])
-		snprintf(v4l2_dev->name, sizeof(v4l2_dev->name),
-			 "%s.m2m", dev_name(&pdev->dev));
+	snprintf(v4l2_dev->name, sizeof(v4l2_dev->name),
+		 "%s.m2m", dev_name(&pdev->dev));
 
 	ret = v4l2_device_register(&pdev->dev, v4l2_dev);
 	if (ret)
@@ -1524,6 +1522,7 @@ static int fimc_register_m2m_device(struct fimc_dev *fimc)
 
 	vfd->fops	= &fimc_m2m_fops;
 	vfd->ioctl_ops	= &fimc_m2m_ioctl_ops;
+	vfd->v4l2_dev	= v4l2_dev;
 	vfd->minor	= -1;
 	vfd->release	= video_device_release;
 	vfd->lock	= &fimc->lock;
@@ -1541,17 +1540,22 @@ static int fimc_register_m2m_device(struct fimc_dev *fimc)
 		goto err_m2m_r2;
 	}
 
+	ret = media_entity_init(&vfd->entity, 0, NULL, 0);
+	if (ret)
+		goto err_m2m_r3;
+
 	ret = video_register_device(vfd, VFL_TYPE_GRABBER, -1);
 	if (ret) {
 		v4l2_err(v4l2_dev,
 			 "%s(): failed to register video device\n", __func__);
-		goto err_m2m_r3;
+		goto err_m2m_r4;
 	}
 	v4l2_info(v4l2_dev,
 		  "FIMC m2m driver registered as /dev/video%d\n", vfd->num);
 
 	return 0;
-
+err_m2m_r4:
+	media_entity_cleanup(&vfd->entity);
 err_m2m_r3:
 	v4l2_m2m_release(fimc->m2m.m2m_dev);
 err_m2m_r2:
@@ -1564,12 +1568,13 @@ err_m2m_r1:
 
 void fimc_unregister_m2m_device(struct fimc_dev *fimc)
 {
-	if (fimc) {
-		v4l2_m2m_release(fimc->m2m.m2m_dev);
-		video_unregister_device(fimc->m2m.vfd);
+	if (fimc == NULL)
+		return;
 
-		v4l2_device_unregister(&fimc->m2m.v4l2_dev);
-	}
+	v4l2_m2m_release(fimc->m2m.m2m_dev);
+	v4l2_device_unregister(&fimc->m2m.v4l2_dev);
+	media_entity_cleanup(&fimc->m2m.vfd->entity);
+	video_unregister_device(fimc->m2m.vfd);
 }
 
 static void fimc_clk_put(struct fimc_dev *fimc)
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 21dfcac..55c1410 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -16,6 +16,8 @@
 #include <linux/types.h>
 #include <linux/videodev2.h>
 #include <linux/io.h>
+
+#include <media/media-entity.h>
 #include <media/videobuf2-core.h>
 #include <media/v4l2-device.h>
 #include <media/v4l2-mem2mem.h>
@@ -298,6 +300,7 @@ struct fimc_m2m_device {
  * @vfd: video device node for camera capture mode
  * @v4l2_dev: v4l2_device struct to manage subdevs
  * @sd: pointer to camera sensor subdevice currently in use
+ * @vd_pad: fimc video capture node pad
  * @fmt: Media Bus format configured at selected image sensor
  * @pending_buf_q: the pending buffer queue head
  * @active_buf_q: the queue head of buffers scheduled in hardware
@@ -315,6 +318,7 @@ struct fimc_vid_cap {
 	struct video_device		*vfd;
 	struct v4l2_device		v4l2_dev;
 	struct v4l2_subdev		*sd;;
+	struct media_pad		vd_pad;
 	struct v4l2_mbus_framefmt	fmt;
 	struct list_head		pending_buf_q;
 	struct list_head		active_buf_q;
-- 
1.7.5.4

[PATCH v3 03/19] s5p-fimc: Remove registration of video nodes from probe()

[Sylwester Nawrocki]

Do not register video nodes during FIMC device probe. Also make
fimc_register_m2m_device() public for use by the media device driver.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |    1 +
 drivers/media/video/s5p-fimc/fimc-core.c    |   30 +++++++-------------------
 drivers/media/video/s5p-fimc/fimc-core.h    |    1 +
 3 files changed, 10 insertions(+), 22 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index c9be15c..a806e48 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -922,4 +922,5 @@ void fimc_unregister_capture_device(struct fimc_dev *fimc)
 		video_unregister_device(vfd);
 	}
 	kfree(fimc->vid_cap.ctx);
+	fimc->vid_cap.ctx = NULL;
 }
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index c97494d..36a8d63 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -1494,7 +1494,7 @@ static struct v4l2_m2m_ops m2m_ops = {
 	.job_abort	= fimc_job_abort,
 };
 
-static int fimc_register_m2m_device(struct fimc_dev *fimc)
+int fimc_register_m2m_device(struct fimc_dev *fimc)
 {
 	struct video_device *vfd;
 	struct platform_device *pdev;
@@ -1568,13 +1568,16 @@ err_m2m_r1:
 
 void fimc_unregister_m2m_device(struct fimc_dev *fimc)
 {
-	if (fimc == NULL)
+	if (!fimc)
 		return;
 
-	v4l2_m2m_release(fimc->m2m.m2m_dev);
+	if (fimc->m2m.m2m_dev)
+		v4l2_m2m_release(fimc->m2m.m2m_dev);
 	v4l2_device_unregister(&fimc->m2m.v4l2_dev);
-	media_entity_cleanup(&fimc->m2m.vfd->entity);
-	video_unregister_device(fimc->m2m.vfd);
+	if (fimc->m2m.vfd) {
+		media_entity_cleanup(&fimc->m2m.vfd->entity);
+		video_unregister_device(fimc->m2m.vfd);
+	}
 }
 
 static void fimc_clk_put(struct fimc_dev *fimc)
@@ -1739,25 +1742,12 @@ static int fimc_probe(struct platform_device *pdev)
 		goto err_irq;
 	}
 
-	ret = fimc_register_m2m_device(fimc);
-	if (ret)
-		goto err_irq;
-
-	/* At least one camera sensor is required to register capture node */
-	if (cap_input_index >= 0) {
-		ret = fimc_register_capture_device(fimc);
-		if (ret)
-			goto err_m2m;
-	}
-
 	dev_dbg(&pdev->dev, "%s(): fimc-%d registered successfully\n",
 		__func__, fimc->id);
 
 	pm_runtime_put(&fimc->pdev->dev);
 	return 0;
 
-err_m2m:
-	fimc_unregister_m2m_device(fimc);
 err_irq:
 	free_irq(fimc->irq, fimc);
 err_clk:
@@ -1769,7 +1759,6 @@ err_req_region:
 	kfree(fimc->regs_res);
 err_info:
 	kfree(fimc);
-
 	return ret;
 }
 
@@ -1858,9 +1847,6 @@ static int __devexit fimc_remove(struct platform_device *pdev)
 	fimc_runtime_suspend(&pdev->dev);
 	pm_runtime_set_suspended(&pdev->dev);
 
-	fimc_unregister_m2m_device(fimc);
-	fimc_unregister_capture_device(fimc);
-
 	vb2_dma_contig_cleanup_ctx(fimc->alloc_ctx);
 
 	clk_disable(fimc->clock[CLK_BUS]);
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 55c1410..c088dac 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -644,6 +644,7 @@ int fimc_set_scaler_info(struct fimc_ctx *ctx);
 int fimc_prepare_config(struct fimc_ctx *ctx, u32 flags);
 int fimc_prepare_addr(struct fimc_ctx *ctx, struct vb2_buffer *vb,
 		      struct fimc_frame *frame, struct fimc_addr *paddr);
+int fimc_register_m2m_device(struct fimc_dev *fimc);
 
 /* -----------------------------------------------------*/
 /* fimc-capture.c					*/
-- 
1.7.5.4

[PATCH v3 04/19] s5p-fimc: Remove sclk_cam clock handling

[Sylwester Nawrocki]

The external sclk_cam clocks will be handled at the media device
driver level.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-core.c |   12 ++----------
 drivers/media/video/s5p-fimc/fimc-core.h |    3 +--
 2 files changed, 3 insertions(+), 12 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index 36a8d63..fd6a308 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -30,7 +30,7 @@
 #include "fimc-core.h"
 
 static char *fimc_clocks[MAX_FIMC_CLOCKS] = {
-	"sclk_fimc", "fimc", "sclk_cam"
+	"sclk_fimc", "fimc"
 };
 
 static struct fimc_fmt fimc_formats[] = {
@@ -1648,7 +1648,6 @@ static int fimc_probe(struct platform_device *pdev)
 	struct samsung_fimc_driverdata *drv_data;
 	struct s5p_platform_fimc *pdata;
 	int ret = 0;
-	int cap_input_index = -1;
 
 	dev_dbg(&pdev->dev, "%s():\n", __func__);
 
@@ -1701,14 +1700,6 @@ static int fimc_probe(struct platform_device *pdev)
 		goto err_req_region;
 	}
 
-	fimc->num_clocks = MAX_FIMC_CLOCKS - 1;
-
-	/* Check if a video capture node needs to be registered. */
-	if (pdata && pdata->num_clients > 0) {
-		cap_input_index = 0;
-		fimc->num_clocks++;
-	}
-
 	res = platform_get_resource(pdev, IORESOURCE_IRQ, 0);
 	if (!res) {
 		dev_err(&pdev->dev, "failed to get IRQ resource\n");
@@ -1717,6 +1708,7 @@ static int fimc_probe(struct platform_device *pdev)
 	}
 	fimc->irq = res->start;
 
+	fimc->num_clocks = MAX_FIMC_CLOCKS;
 	ret = fimc_clk_get(fimc);
 	if (ret)
 		goto err_regs_unmap;
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index c088dac..f059216 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -34,7 +34,7 @@
 
 /* Time to wait for next frame VSYNC interrupt while stopping operation. */
 #define FIMC_SHUTDOWN_TIMEOUT	((100*HZ)/1000)
-#define MAX_FIMC_CLOCKS		3
+#define MAX_FIMC_CLOCKS		2
 #define MODULE_NAME		"s5p-fimc"
 #define FIMC_MAX_DEVS		4
 #define FIMC_MAX_OUT_BUFS	4
@@ -46,7 +46,6 @@
 enum {
 	CLK_BUS,
 	CLK_GATE,
-	CLK_CAM,
 };
 
 enum fimc_dev_flags {
-- 
1.7.5.4

[PATCH v3 05/19] s5p-fimc: Limit number of available inputs to one

[Sylwester Nawrocki]

Camera sensors at FIMC input are no longer selected with S_INPUT ioctl.
They will be attached to required FIMC entity through pipeline
re-configuration at the media device level.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |   43 ++++-----------------------
 1 files changed, 6 insertions(+), 37 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index a806e48..5b08b69 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -562,57 +562,26 @@ static int fimc_cap_s_fmt_mplane(struct file *file, void *priv,
 }
 
 static int fimc_cap_enum_input(struct file *file, void *priv,
-				     struct v4l2_input *i)
+			       struct v4l2_input *i)
 {
 	struct fimc_ctx *ctx = priv;
-	struct s5p_platform_fimc *pldata = ctx->fimc_dev->pdata;
-	struct s5p_fimc_isp_info *isp_info;
 
-	if (i->index >= pldata->num_clients)
+	if (i->index != 0)
 		return -EINVAL;
 
-	isp_info = &pldata->isp_info[i->index];
 
 	i->type = V4L2_INPUT_TYPE_CAMERA;
-	strncpy(i->name, isp_info->board_info->type, 32);
 	return 0;
 }
 
-static int fimc_cap_s_input(struct file *file, void *priv,
-				  unsigned int i)
+static int fimc_cap_s_input(struct file *file, void *priv, unsigned int i)
 {
-	struct fimc_ctx *ctx = priv;
-	struct fimc_dev *fimc = ctx->fimc_dev;
-	struct s5p_platform_fimc *pdata = fimc->pdata;
-
-	if (fimc_capture_active(ctx->fimc_dev))
-		return -EBUSY;
-
-	if (i >= pdata->num_clients)
-		return -EINVAL;
-
-
-	if (fimc->vid_cap.sd) {
-		int ret = v4l2_subdev_call(fimc->vid_cap.sd, core, s_power, 0);
-		if (ret)
-			err("s_power failed: %d", ret);
-
-		clk_disable(fimc->clock[CLK_CAM]);
-	}
-
-	/* Release the attached sensor subdevice. */
-	fimc_subdev_unregister(fimc);
-
-	return fimc_isp_subdev_init(fimc, i);
+	return i == 0 ? i : -EINVAL;
 }
 
-static int fimc_cap_g_input(struct file *file, void *priv,
-				       unsigned int *i)
+static int fimc_cap_g_input(struct file *file, void *priv, unsigned int *i)
 {
-	struct fimc_ctx *ctx = priv;
-	struct fimc_vid_cap *cap = &ctx->fimc_dev->vid_cap;
-
-	*i = cap->input_index;
+	*i = 0;
 	return 0;
 }
 
-- 
1.7.5.4

[PATCH v3 06/19] s5p-fimc: Remove sensor management code from FIMC capture driver

[Sylwester Nawrocki]

The sensor subdevs need to be shared between all available FIMC instances.
Remove their registration from FIMC capture driver so they can then be
registered to the media device driver.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |  139 +--------------------------
 drivers/media/video/s5p-fimc/fimc-core.h    |    2 +-
 2 files changed, 2 insertions(+), 139 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index 5b08b69..e7aa61e 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -16,12 +16,9 @@
 #include <linux/bug.h>
 #include <linux/interrupt.h>
 #include <linux/device.h>
-#include <linux/platform_device.h>
 #include <linux/pm_runtime.h>
 #include <linux/list.h>
 #include <linux/slab.h>
-#include <linux/clk.h>
-#include <linux/i2c.h>
 
 #include <linux/videodev2.h>
 #include <media/v4l2-device.h>
@@ -32,126 +29,6 @@
 
 #include "fimc-core.h"
 
-static struct v4l2_subdev *fimc_subdev_register(struct fimc_dev *fimc,
-					    struct s5p_fimc_isp_info *isp_info)
-{
-	struct i2c_adapter *i2c_adap;
-	struct fimc_vid_cap *vid_cap = &fimc->vid_cap;
-	struct v4l2_subdev *sd = NULL;
-
-	i2c_adap = i2c_get_adapter(isp_info->i2c_bus_num);
-	if (!i2c_adap)
-		return ERR_PTR(-ENOMEM);
-
-	sd = v4l2_i2c_new_subdev_board(&vid_cap->v4l2_dev, i2c_adap,
-				       isp_info->board_info, NULL);
-	if (!sd) {
-		v4l2_err(&vid_cap->v4l2_dev, "failed to acquire subdev\n");
-		return NULL;
-	}
-
-	v4l2_info(&vid_cap->v4l2_dev, "subdevice %s registered successfuly\n",
-		isp_info->board_info->type);
-
-	return sd;
-}
-
-static void fimc_subdev_unregister(struct fimc_dev *fimc)
-{
-	struct fimc_vid_cap *vid_cap = &fimc->vid_cap;
-	struct i2c_client *client;
-
-	if (vid_cap->input_index < 0)
-		return;	/* Subdevice already released or not registered. */
-
-	if (vid_cap->sd) {
-		v4l2_device_unregister_subdev(vid_cap->sd);
-		client = v4l2_get_subdevdata(vid_cap->sd);
-		i2c_unregister_device(client);
-		i2c_put_adapter(client->adapter);
-		vid_cap->sd = NULL;
-	}
-
-	vid_cap->input_index = -1;
-}
-
-/**
- * fimc_subdev_attach - attach v4l2_subdev to camera host interface
- *
- * @fimc: FIMC device information
- * @index: index to the array of available subdevices,
- *	   -1 for full array search or non negative value
- *	   to select specific subdevice
- */
-static int fimc_subdev_attach(struct fimc_dev *fimc, int index)
-{
-	struct fimc_vid_cap *vid_cap = &fimc->vid_cap;
-	struct s5p_platform_fimc *pdata = fimc->pdata;
-	struct s5p_fimc_isp_info *isp_info;
-	struct v4l2_subdev *sd;
-	int i;
-
-	for (i = 0; i < pdata->num_clients; ++i) {
-		isp_info = &pdata->isp_info[i];
-
-		if (index >= 0 && i != index)
-			continue;
-
-		sd = fimc_subdev_register(fimc, isp_info);
-		if (!IS_ERR_OR_NULL(sd)) {
-			vid_cap->sd = sd;
-			vid_cap->input_index = i;
-
-			return 0;
-		}
-	}
-
-	vid_cap->input_index = -1;
-	vid_cap->sd = NULL;
-	v4l2_err(&vid_cap->v4l2_dev, "fimc%d: sensor attach failed\n",
-		 fimc->id);
-	return -ENODEV;
-}
-
-static int fimc_isp_subdev_init(struct fimc_dev *fimc, unsigned int index)
-{
-	struct s5p_fimc_isp_info *isp_info;
-	struct s5p_platform_fimc *pdata = fimc->pdata;
-	int ret;
-
-	if (index >= pdata->num_clients)
-		return -EINVAL;
-
-	isp_info = &pdata->isp_info[index];
-
-	if (isp_info->clk_frequency)
-		clk_set_rate(fimc->clock[CLK_CAM], isp_info->clk_frequency);
-
-	ret = clk_enable(fimc->clock[CLK_CAM]);
-	if (ret)
-		return ret;
-
-	ret = fimc_subdev_attach(fimc, index);
-	if (ret)
-		return ret;
-
-	ret = fimc_hw_set_camera_polarity(fimc, isp_info);
-	if (ret)
-		return ret;
-
-	ret = v4l2_subdev_call(fimc->vid_cap.sd, core, s_power, 1);
-	if (!ret)
-		return ret;
-
-	/* enabling power failed so unregister subdev */
-	fimc_subdev_unregister(fimc);
-
-	v4l2_err(&fimc->vid_cap.v4l2_dev, "ISP initialization failed: %d\n",
-		 ret);
-
-	return ret;
-}
-
 static int fimc_stop_capture(struct fimc_dev *fimc)
 {
 	unsigned long flags;
@@ -396,15 +273,7 @@ static int fimc_capture_open(struct file *file)
 	if (ret)
 		return ret;
 
-	if (++fimc->vid_cap.refcnt == 1) {
-		ret = fimc_isp_subdev_init(fimc, 0);
-		if (ret) {
-			pm_runtime_put_sync(&fimc->pdev->dev);
-			fimc->vid_cap.refcnt--;
-			return -EIO;
-		}
-	}
-
+	++fimc->vid_cap.refcnt;
 	file->private_data = fimc->vid_cap.ctx;
 
 	return 0;
@@ -419,12 +288,6 @@ static int fimc_capture_close(struct file *file)
 	if (--fimc->vid_cap.refcnt == 0) {
 		fimc_stop_capture(fimc);
 		vb2_queue_release(&fimc->vid_cap.vbq);
-
-		v4l2_err(&fimc->vid_cap.v4l2_dev, "releasing ISP\n");
-
-		v4l2_subdev_call(fimc->vid_cap.sd, core, s_power, 0);
-		clk_disable(fimc->clock[CLK_CAM]);
-		fimc_subdev_unregister(fimc);
 	}
 
 	pm_runtime_put_sync(&fimc->pdev->dev);
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index f059216..210301e 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -11,6 +11,7 @@
 
 /*#define DEBUG*/
 
+#include <linux/platform_device.h>
 #include <linux/sched.h>
 #include <linux/spinlock.h>
 #include <linux/types.h>
@@ -649,7 +650,6 @@ int fimc_register_m2m_device(struct fimc_dev *fimc);
 /* fimc-capture.c					*/
 int fimc_register_capture_device(struct fimc_dev *fimc);
 void fimc_unregister_capture_device(struct fimc_dev *fimc);
-int fimc_sensor_sd_init(struct fimc_dev *fimc, int index);
 int fimc_vid_cap_buf_queue(struct fimc_dev *fimc,
 			     struct fimc_vid_buffer *fimc_vb);
 int fimc_capture_suspend(struct fimc_dev *fimc);
-- 
1.7.5.4

[PATCH v3 07/19] s5p-fimc: Remove v4l2_device from video capture and m2m driver

[Sylwester Nawrocki]

Currently there is a v4l2_device instance being registered per each
(capture and memory-to-memory) video node created per FIMC H/W instance.
This patch is a prerequisite for using the top level v4l2_device
instantiated by the media device driver.
To retain current debug trace semantic (so it's possible to distinguish
between the capture and m2m FIMC) the video_device is used in place
of v4l2_device where appropriate.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |   43 ++++++----------
 drivers/media/video/s5p-fimc/fimc-core.c    |   70 ++++++++++----------------
 drivers/media/video/s5p-fimc/fimc-core.h    |   15 +++---
 drivers/media/video/s5p-fimc/fimc-reg.c     |    7 ++-
 4 files changed, 55 insertions(+), 80 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index e7aa61e..70892e4 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -130,7 +130,6 @@ static int start_streaming(struct vb2_queue *q)
 	fimc->vid_cap.buf_index = 0;
 
 	set_bit(ST_CAPT_PEND, &fimc->state);
-
 	return 0;
 }
 
@@ -177,7 +176,6 @@ static int buffer_prepare(struct vb2_buffer *vb)
 {
 	struct vb2_queue *vq = vb->vb2_queue;
 	struct fimc_ctx *ctx = vq->drv_priv;
-	struct v4l2_device *v4l2_dev = &ctx->fimc_dev->m2m.v4l2_dev;
 	int i;
 
 	if (!ctx->d_frame.fmt || vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
@@ -187,7 +185,8 @@ static int buffer_prepare(struct vb2_buffer *vb)
 		unsigned long size = get_plane_size(&ctx->d_frame, i);
 
 		if (vb2_plane_size(vb, i) < size) {
-			v4l2_err(v4l2_dev, "User buffer too small (%ld < %ld)\n",
+			v4l2_err(ctx->fimc_dev->vid_cap.vfd,
+				 "User buffer too small (%ld < %ld)\n",
 				 vb2_plane_size(vb, i), size);
 			return -EINVAL;
 		}
@@ -398,7 +397,8 @@ static int fimc_cap_s_fmt_mplane(struct file *file, void *priv,
 	pix = &f->fmt.pix_mp;
 	frame->fmt = find_format(f, FMT_FLAGS_M2M | FMT_FLAGS_CAM);
 	if (!frame->fmt) {
-		err("fimc target format not found\n");
+		v4l2_err(fimc->vid_cap.vfd,
+			 "Not supported capture (FIMC target) color format\n");
 		return -EINVAL;
 	}
 
@@ -458,7 +458,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 		return -EBUSY;
 
 	if (!(ctx->state & FIMC_DST_FMT)) {
-		v4l2_err(&fimc->vid_cap.v4l2_dev, "Format is not set\n");
+		v4l2_err(fimc->vid_cap.vfd, "Format is not set\n");
 		return -EINVAL;
 	}
 
@@ -588,9 +588,8 @@ static int fimc_cap_s_crop(struct file *file, void *fh,
 		return ret;
 
 	if (!(ctx->state & FIMC_DST_FMT)) {
-		v4l2_err(&fimc->vid_cap.v4l2_dev,
-			 "Capture color format not set\n");
-		return -EINVAL; /* TODO: make sure this is the right value */
+		v4l2_err(fimc->vid_cap.vfd, "Capture format is not set\n");
+		return -EINVAL;
 	}
 
 	f = &ctx->s_frame;
@@ -599,7 +598,7 @@ static int fimc_cap_s_crop(struct file *file, void *fh,
 				      ctx->d_frame.width, ctx->d_frame.height,
 				      ctx->rotation);
 	if (ret) {
-		v4l2_err(&fimc->vid_cap.v4l2_dev, "Out of the scaler range\n");
+		v4l2_err(fimc->vid_cap.vfd, "Out of the scaler range\n");
 		return ret;
 	}
 
@@ -643,16 +642,16 @@ static const struct v4l2_ioctl_ops fimc_capture_ioctl_ops = {
 };
 
 /* fimc->lock must be already initialized */
-int fimc_register_capture_device(struct fimc_dev *fimc)
+int fimc_register_capture_device(struct fimc_dev *fimc,
+				 struct v4l2_device *v4l2_dev)
 {
-	struct v4l2_device *v4l2_dev = &fimc->vid_cap.v4l2_dev;
 	struct video_device *vfd;
 	struct fimc_vid_cap *vid_cap;
 	struct fimc_ctx *ctx;
 	struct v4l2_format f;
 	struct fimc_frame *fr;
 	struct vb2_queue *q;
-	int ret;
+	int ret = -ENOMEM;
 
 	ctx = kzalloc(sizeof *ctx, GFP_KERNEL);
 	if (!ctx)
@@ -670,20 +669,14 @@ int fimc_register_capture_device(struct fimc_dev *fimc)
 	fr->width = fr->f_width = fr->o_width = 640;
 	fr->height = fr->f_height = fr->o_height = 480;
 
-	snprintf(v4l2_dev->name, sizeof(v4l2_dev->name),
-		 "%s.capture", dev_name(&fimc->pdev->dev));
-
-	ret = v4l2_device_register(NULL, v4l2_dev);
-	if (ret)
-		goto err_info;
-
 	vfd = video_device_alloc();
 	if (!vfd) {
 		v4l2_err(v4l2_dev, "Failed to allocate video device\n");
-		goto err_v4l2_reg;
+		goto err_vd_alloc;
 	}
 
-	strlcpy(vfd->name, v4l2_dev->name, sizeof(vfd->name));
+	snprintf(vfd->name, sizeof(vfd->name), "%s.capture",
+		 dev_name(&fimc->pdev->dev));
 
 	vfd->fops	= &fimc_capture_fops;
 	vfd->ioctl_ops	= &fimc_capture_ioctl_ops;
@@ -728,8 +721,7 @@ int fimc_register_capture_device(struct fimc_dev *fimc)
 		goto err_vd_reg;
 	}
 
-	v4l2_info(v4l2_dev,
-		  "FIMC capture driver registered as /dev/video%d\n",
+	v4l2_info(vfd, "FIMC capture driver registered as /dev/video%d\n",
 		  vfd->num);
 	return 0;
 
@@ -737,11 +729,8 @@ err_vd_reg:
 	media_entity_cleanup(&vfd->entity);
 err_ent:
 	video_device_release(vfd);
-err_v4l2_reg:
-	v4l2_device_unregister(v4l2_dev);
-err_info:
+err_vd_alloc:
 	kfree(ctx);
-	dev_err(&fimc->pdev->dev, "failed to install\n");
 	return ret;
 }
 
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index fd6a308..8e2f2ee 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -236,10 +236,11 @@ static int fimc_get_scaler_factor(u32 src, u32 tar, u32 *ratio, u32 *shift)
 
 int fimc_set_scaler_info(struct fimc_ctx *ctx)
 {
+	struct samsung_fimc_variant *variant = ctx->fimc_dev->variant;
+	struct device *dev = &ctx->fimc_dev->pdev->dev;
 	struct fimc_scaler *sc = &ctx->scaler;
 	struct fimc_frame *s_frame = &ctx->s_frame;
 	struct fimc_frame *d_frame = &ctx->d_frame;
-	struct samsung_fimc_variant *variant = ctx->fimc_dev->variant;
 	int tx, ty, sx, sy;
 	int ret;
 
@@ -251,15 +252,14 @@ int fimc_set_scaler_info(struct fimc_ctx *ctx)
 		ty = d_frame->height;
 	}
 	if (tx <= 0 || ty <= 0) {
-		v4l2_err(&ctx->fimc_dev->m2m.v4l2_dev,
-			"invalid target size: %d x %d", tx, ty);
+		dev_err(dev, "Invalid target size: %dx%d", tx, ty);
 		return -EINVAL;
 	}
 
 	sx = s_frame->width;
 	sy = s_frame->height;
 	if (sx <= 0 || sy <= 0) {
-		err("invalid source size: %d x %d", sx, sy);
+		dev_err(dev, "Invalid source size: %dx%d", sx, sy);
 		return -EINVAL;
 	}
 	sc->real_width = sx;
@@ -898,7 +898,7 @@ int fimc_vidioc_try_fmt_mplane(struct file *file, void *priv,
 	mask = is_output ? FMT_FLAGS_M2M : FMT_FLAGS_M2M | FMT_FLAGS_CAM;
 	fmt = find_format(f, mask);
 	if (!fmt) {
-		v4l2_err(&fimc->m2m.v4l2_dev, "Fourcc format (0x%X) invalid.\n",
+		v4l2_err(fimc->v4l2_dev, "Fourcc format (0x%X) invalid.\n",
 			 pix->pixelformat);
 		return -EINVAL;
 	}
@@ -973,7 +973,7 @@ static int fimc_m2m_s_fmt_mplane(struct file *file, void *priv,
 	vq = v4l2_m2m_get_vq(ctx->m2m_ctx, f->type);
 
 	if (vb2_is_busy(vq)) {
-		v4l2_err(&fimc->m2m.v4l2_dev, "queue (%d) busy\n", f->type);
+		v4l2_err(fimc->m2m.vfd, "queue (%d) busy\n", f->type);
 		return -EBUSY;
 	}
 
@@ -982,7 +982,7 @@ static int fimc_m2m_s_fmt_mplane(struct file *file, void *priv,
 	} else if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
 		frame = &ctx->d_frame;
 	} else {
-		v4l2_err(&fimc->m2m.v4l2_dev,
+		v4l2_err(fimc->m2m.vfd,
 			 "Wrong buffer/video queue type (%d)\n", f->type);
 		return -EINVAL;
 	}
@@ -1110,7 +1110,7 @@ int fimc_vidioc_g_ctrl(struct file *file, void *priv,
 			return v4l2_subdev_call(fimc->vid_cap.sd, core,
 						g_ctrl, ctrl);
 		} else {
-			v4l2_err(&fimc->m2m.v4l2_dev, "Invalid control\n");
+			v4l2_err(fimc->m2m.vfd, "Invalid control\n");
 			return -EINVAL;
 		}
 	}
@@ -1128,8 +1128,7 @@ int check_ctrl_val(struct fimc_ctx *ctx,  struct v4l2_control *ctrl)
 
 	if (ctrl->value < c->minimum || ctrl->value > c->maximum
 		|| (c->step != 0 && ctrl->value % c->step != 0)) {
-		v4l2_err(&ctx->fimc_dev->m2m.v4l2_dev,
-		"Invalid control value\n");
+		v4l2_err(ctx->fimc_dev->m2m.vfd, "Invalid control value\n");
 		return -ERANGE;
 	}
 
@@ -1165,7 +1164,7 @@ int fimc_s_ctrl(struct fimc_ctx *ctx, struct v4l2_control *ctrl)
 		}
 
 		if (ret) {
-			v4l2_err(&fimc->m2m.v4l2_dev, "Out of scaler range\n");
+			v4l2_err(fimc->m2m.vfd, "Out of scaler range\n");
 			return -EINVAL;
 		}
 
@@ -1177,7 +1176,7 @@ int fimc_s_ctrl(struct fimc_ctx *ctx, struct v4l2_control *ctrl)
 		break;
 
 	default:
-		v4l2_err(&fimc->m2m.v4l2_dev, "Invalid control\n");
+		v4l2_err(fimc->v4l2_dev, "Invalid control\n");
 		return -EINVAL;
 	}
 
@@ -1245,7 +1244,7 @@ int fimc_try_crop(struct fimc_ctx *ctx, struct v4l2_crop *cr)
 	int i;
 
 	if (cr->c.top < 0 || cr->c.left < 0) {
-		v4l2_err(&fimc->m2m.v4l2_dev,
+		v4l2_err(fimc->m2m.vfd,
 			"doesn't support negative values for top & left\n");
 		return -EINVAL;
 	}
@@ -1326,7 +1325,7 @@ static int fimc_m2m_s_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 						      ctx->rotation);
 		}
 		if (ret) {
-			v4l2_err(&fimc->m2m.v4l2_dev, "Out of scaler range\n");
+			v4l2_err(fimc->m2m.vfd, "Out of scaler range\n");
 			return -EINVAL;
 		}
 	}
@@ -1494,30 +1493,23 @@ static struct v4l2_m2m_ops m2m_ops = {
 	.job_abort	= fimc_job_abort,
 };
 
-int fimc_register_m2m_device(struct fimc_dev *fimc)
+int fimc_register_m2m_device(struct fimc_dev *fimc,
+			     struct v4l2_device *v4l2_dev)
 {
 	struct video_device *vfd;
 	struct platform_device *pdev;
-	struct v4l2_device *v4l2_dev;
 	int ret = 0;
 
 	if (!fimc)
 		return -ENODEV;
 
 	pdev = fimc->pdev;
-	v4l2_dev = &fimc->m2m.v4l2_dev;
-
-	snprintf(v4l2_dev->name, sizeof(v4l2_dev->name),
-		 "%s.m2m", dev_name(&pdev->dev));
-
-	ret = v4l2_device_register(&pdev->dev, v4l2_dev);
-	if (ret)
-		goto err_m2m_r1;
+	fimc->v4l2_dev = v4l2_dev;
 
 	vfd = video_device_alloc();
 	if (!vfd) {
 		v4l2_err(v4l2_dev, "Failed to allocate video device\n");
-		goto err_m2m_r1;
+		return -ENOMEM;
 	}
 
 	vfd->fops	= &fimc_m2m_fops;
@@ -1527,42 +1519,35 @@ int fimc_register_m2m_device(struct fimc_dev *fimc)
 	vfd->release	= video_device_release;
 	vfd->lock	= &fimc->lock;
 
-	snprintf(vfd->name, sizeof(vfd->name), "%s:m2m", dev_name(&pdev->dev));
-
+	snprintf(vfd->name, sizeof(vfd->name), "%s.m2m", dev_name(&pdev->dev));
 	video_set_drvdata(vfd, fimc);
-	platform_set_drvdata(pdev, fimc);
 
 	fimc->m2m.vfd = vfd;
 	fimc->m2m.m2m_dev = v4l2_m2m_init(&m2m_ops);
 	if (IS_ERR(fimc->m2m.m2m_dev)) {
 		v4l2_err(v4l2_dev, "failed to initialize v4l2-m2m device\n");
 		ret = PTR_ERR(fimc->m2m.m2m_dev);
-		goto err_m2m_r2;
+		goto err_init;
 	}
 
 	ret = media_entity_init(&vfd->entity, 0, NULL, 0);
 	if (ret)
-		goto err_m2m_r3;
+		goto err_me;
 
 	ret = video_register_device(vfd, VFL_TYPE_GRABBER, -1);
 	if (ret) {
-		v4l2_err(v4l2_dev,
-			 "%s(): failed to register video device\n", __func__);
-		goto err_m2m_r4;
+		v4l2_err(vfd, "Failed to register video device\n");
+		goto err_vd;
 	}
-	v4l2_info(v4l2_dev,
-		  "FIMC m2m driver registered as /dev/video%d\n", vfd->num);
-
+	v4l2_info(vfd, "FIMC m2m driver registered as /dev/video%d\n", vfd->num);
 	return 0;
-err_m2m_r4:
+
+err_vd:
 	media_entity_cleanup(&vfd->entity);
-err_m2m_r3:
+err_me:
 	v4l2_m2m_release(fimc->m2m.m2m_dev);
-err_m2m_r2:
+err_init:
 	video_device_release(fimc->m2m.vfd);
-err_m2m_r1:
-	v4l2_device_unregister(v4l2_dev);
-
 	return ret;
 }
 
@@ -1573,7 +1558,6 @@ void fimc_unregister_m2m_device(struct fimc_dev *fimc)
 
 	if (fimc->m2m.m2m_dev)
 		v4l2_m2m_release(fimc->m2m.m2m_dev);
-	v4l2_device_unregister(&fimc->m2m.v4l2_dev);
 	if (fimc->m2m.vfd) {
 		media_entity_cleanup(&fimc->m2m.vfd->entity);
 		video_unregister_device(fimc->m2m.vfd);
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 210301e..fe82bf7 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -281,14 +281,12 @@ struct fimc_frame {
 /**
  * struct fimc_m2m_device - v4l2 memory-to-memory device data
  * @vfd: the video device node for v4l2 m2m mode
- * @v4l2_dev: v4l2 device for m2m mode
  * @m2m_dev: v4l2 memory-to-memory device data
  * @ctx: hardware context data
  * @refcnt: the reference counter
  */
 struct fimc_m2m_device {
 	struct video_device	*vfd;
-	struct v4l2_device	v4l2_dev;
 	struct v4l2_m2m_dev	*m2m_dev;
 	struct fimc_ctx		*ctx;
 	int			refcnt;
@@ -298,7 +296,6 @@ struct fimc_m2m_device {
  * struct fimc_vid_cap - camera capture device information
  * @ctx: hardware context data
  * @vfd: video device node for camera capture mode
- * @v4l2_dev: v4l2_device struct to manage subdevs
  * @sd: pointer to camera sensor subdevice currently in use
  * @vd_pad: fimc video capture node pad
  * @fmt: Media Bus format configured at selected image sensor
@@ -316,7 +313,6 @@ struct fimc_vid_cap {
 	struct fimc_ctx			*ctx;
 	struct vb2_alloc_ctx		*alloc_ctx;
 	struct video_device		*vfd;
-	struct v4l2_device		v4l2_dev;
 	struct v4l2_subdev		*sd;;
 	struct media_pad		vd_pad;
 	struct v4l2_mbus_framefmt	fmt;
@@ -407,6 +403,7 @@ struct fimc_ctx;
  * @regs_res:	the resource claimed for IO registers
  * @irq:	FIMC interrupt number
  * @irq_queue:	interrupt handler waitqueue
+ * @v4l2_dev:	root v4l2_device
  * @m2m:	memory-to-memory V4L2 device information
  * @vid_cap:	camera capture device information
  * @state:	flags used to synchronize m2m and capture mode operation
@@ -425,6 +422,7 @@ struct fimc_dev {
 	struct resource			*regs_res;
 	int				irq;
 	wait_queue_head_t		irq_queue;
+	struct v4l2_device		*v4l2_dev;
 	struct fimc_m2m_device		m2m;
 	struct fimc_vid_cap		vid_cap;
 	unsigned long			state;
@@ -569,7 +567,7 @@ static inline struct fimc_frame *ctx_get_frame(struct fimc_ctx *ctx,
 	} else if (V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE == type) {
 		frame = &ctx->d_frame;
 	} else {
-		v4l2_err(&ctx->fimc_dev->m2m.v4l2_dev,
+		v4l2_err(ctx->fimc_dev->v4l2_dev,
 			"Wrong buffer/video queue type (%d)\n", type);
 		return ERR_PTR(-EINVAL);
 	}
@@ -644,11 +642,14 @@ int fimc_set_scaler_info(struct fimc_ctx *ctx);
 int fimc_prepare_config(struct fimc_ctx *ctx, u32 flags);
 int fimc_prepare_addr(struct fimc_ctx *ctx, struct vb2_buffer *vb,
 		      struct fimc_frame *frame, struct fimc_addr *paddr);
-int fimc_register_m2m_device(struct fimc_dev *fimc);
+int fimc_register_m2m_device(struct fimc_dev *fimc,
+			     struct v4l2_device *v4l2_dev);
+void fimc_unregister_m2m_device(struct fimc_dev *fimc);
 
 /* -----------------------------------------------------*/
 /* fimc-capture.c					*/
-int fimc_register_capture_device(struct fimc_dev *fimc);
+int fimc_register_capture_device(struct fimc_dev *fimc,
+				 struct v4l2_device *v4l2_dev);
 void fimc_unregister_capture_device(struct fimc_dev *fimc);
 int fimc_vid_cap_buf_queue(struct fimc_dev *fimc,
 			     struct fimc_vid_buffer *fimc_vb);
diff --git a/drivers/media/video/s5p-fimc/fimc-reg.c b/drivers/media/video/s5p-fimc/fimc-reg.c
index 938dadf..c688263 100644
--- a/drivers/media/video/s5p-fimc/fimc-reg.c
+++ b/drivers/media/video/s5p-fimc/fimc-reg.c
@@ -596,7 +596,7 @@ int fimc_hw_set_camera_source(struct fimc_dev *fimc,
 		}
 
 		if (i == ARRAY_SIZE(pix_desc)) {
-			v4l2_err(&fimc->vid_cap.v4l2_dev,
+			v4l2_err(fimc->vid_cap.vfd,
 				 "Camera color format not supported: %d\n",
 				 fimc->vid_cap.fmt.code);
 			return -EINVAL;
@@ -661,8 +661,9 @@ int fimc_hw_set_camera_type(struct fimc_dev *fimc,
 		if (vid_cap->fmt.code == V4L2_MBUS_FMT_VYUY8_2X8) {
 			tmp = S5P_CSIIMGFMT_YCBCR422_8BIT;
 		} else {
-			err("camera image format not supported: %d",
-			    vid_cap->fmt.code);
+			v4l2_err(fimc->vid_cap.vfd,
+				 "Not supported camera pixel format: %d",
+				 vid_cap->fmt.code);
 			return -EINVAL;
 		}
 		tmp |= (cam->csi_data_align == 32) << 8;
-- 
1.7.5.4

[PATCH v3 08/19] s5p-fimc: Add the media device driver

[Sylwester Nawrocki]

The fimc media device driver is hooked onto "s5p-fimc-md" platform device.
Such a platform device need to be added in a board initialization code
and then camera sensors need to be specified as it's platform data.
The "s5p-fimc-md" device is a top level entity for all FIMC, mipi-csis
and sensor devices.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/Kconfig                 |    2 +-
 drivers/media/video/s5p-fimc/Makefile       |    2 +-
 drivers/media/video/s5p-fimc/fimc-core.c    |   33 +-
 drivers/media/video/s5p-fimc/fimc-core.h    |   16 +-
 drivers/media/video/s5p-fimc/fimc-mdevice.c |  804 +++++++++++++++++++++++++++
 drivers/media/video/s5p-fimc/fimc-mdevice.h |  118 ++++
 include/media/s5p_fimc.h                    |    2 +
 7 files changed, 956 insertions(+), 21 deletions(-)
 create mode 100644 drivers/media/video/s5p-fimc/fimc-mdevice.c
 create mode 100644 drivers/media/video/s5p-fimc/fimc-mdevice.h

diff --git a/drivers/media/video/Kconfig b/drivers/media/video/Kconfig
index 2cc9552..d5ad61b 100644
--- a/drivers/media/video/Kconfig
+++ b/drivers/media/video/Kconfig
@@ -935,7 +935,7 @@ config VIDEO_MX2
 
 config  VIDEO_SAMSUNG_S5P_FIMC
 	tristate "Samsung S5P and EXYNOS4 camera host interface driver"
-	depends on VIDEO_DEV && VIDEO_V4L2 && PLAT_S5P
+	depends on VIDEO_DEV && VIDEO_V4L2 && PLAT_S5P && I2C && VIDEO_V4L2_SUBDEV_API
 	select VIDEOBUF2_DMA_CONTIG
 	select V4L2_MEM2MEM_DEV
 	---help---
diff --git a/drivers/media/video/s5p-fimc/Makefile b/drivers/media/video/s5p-fimc/Makefile
index df6954a..33dec7f 100644
--- a/drivers/media/video/s5p-fimc/Makefile
+++ b/drivers/media/video/s5p-fimc/Makefile
@@ -1,4 +1,4 @@
-s5p-fimc-objs := fimc-core.o fimc-reg.o fimc-capture.o
+s5p-fimc-objs := fimc-core.o fimc-reg.o fimc-capture.o fimc-mdevice.o
 s5p-csis-objs := mipi-csis.o
 
 obj-$(CONFIG_VIDEO_S5P_MIPI_CSIS)	+= s5p-csis.o
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index 8e2f2ee..59190aa 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -28,6 +28,7 @@
 #include <media/videobuf2-dma-contig.h>
 
 #include "fimc-core.h"
+#include "fimc-mdevice.h"
 
 static char *fimc_clocks[MAX_FIMC_CLOCKS] = {
 	"sclk_fimc", "fimc"
@@ -1876,6 +1877,7 @@ static struct fimc_pix_limit s5p_pix_limit[4] = {
 static struct samsung_fimc_variant fimc0_variant_s5p = {
 	.has_inp_rot	 = 1,
 	.has_out_rot	 = 1,
+	.has_cam_if	 = 1,
 	.min_inp_pixsize = 16,
 	.min_out_pixsize = 16,
 	.hor_offs_align	 = 8,
@@ -1884,6 +1886,7 @@ static struct samsung_fimc_variant fimc0_variant_s5p = {
 };
 
 static struct samsung_fimc_variant fimc2_variant_s5p = {
+	.has_cam_if	 = 1,
 	.min_inp_pixsize = 16,
 	.min_out_pixsize = 16,
 	.hor_offs_align	 = 8,
@@ -1895,6 +1898,7 @@ static struct samsung_fimc_variant fimc0_variant_s5pv210 = {
 	.pix_hoff	 = 1,
 	.has_inp_rot	 = 1,
 	.has_out_rot	 = 1,
+	.has_cam_if	 = 1,
 	.min_inp_pixsize = 16,
 	.min_out_pixsize = 16,
 	.hor_offs_align	 = 8,
@@ -1906,6 +1910,7 @@ static struct samsung_fimc_variant fimc1_variant_s5pv210 = {
 	.pix_hoff	 = 1,
 	.has_inp_rot	 = 1,
 	.has_out_rot	 = 1,
+	.has_cam_if	 = 1,
 	.has_mainscaler_ext = 1,
 	.min_inp_pixsize = 16,
 	.min_out_pixsize = 16,
@@ -1915,6 +1920,7 @@ static struct samsung_fimc_variant fimc1_variant_s5pv210 = {
 };
 
 static struct samsung_fimc_variant fimc2_variant_s5pv210 = {
+	.has_cam_if	 = 1,
 	.pix_hoff	 = 1,
 	.min_inp_pixsize = 16,
 	.min_out_pixsize = 16,
@@ -1927,6 +1933,7 @@ static struct samsung_fimc_variant fimc0_variant_exynos4 = {
 	.pix_hoff	 = 1,
 	.has_inp_rot	 = 1,
 	.has_out_rot	 = 1,
+	.has_cam_if	 = 1,
 	.has_cistatus2	 = 1,
 	.has_mainscaler_ext = 1,
 	.min_inp_pixsize = 16,
@@ -1936,8 +1943,9 @@ static struct samsung_fimc_variant fimc0_variant_exynos4 = {
 	.pix_limit	 = &s5p_pix_limit[1],
 };
 
-static struct samsung_fimc_variant fimc2_variant_exynos4 = {
+static struct samsung_fimc_variant fimc3_variant_exynos4 = {
 	.pix_hoff	 = 1,
+	.has_cam_if	 = 1,
 	.has_cistatus2	 = 1,
 	.has_mainscaler_ext = 1,
 	.min_inp_pixsize = 16,
@@ -1975,7 +1983,7 @@ static struct samsung_fimc_driverdata fimc_drvdata_exynos4 = {
 		[0] = &fimc0_variant_exynos4,
 		[1] = &fimc0_variant_exynos4,
 		[2] = &fimc0_variant_exynos4,
-		[3] = &fimc2_variant_exynos4,
+		[3] = &fimc3_variant_exynos4,
 	},
 	.num_entities = 4,
 	.lclk_frequency = 166000000UL,
@@ -2003,32 +2011,21 @@ static const struct dev_pm_ops fimc_pm_ops = {
 
 static struct platform_driver fimc_driver = {
 	.probe		= fimc_probe,
-	.remove	= __devexit_p(fimc_remove),
+	.remove		= __devexit_p(fimc_remove),
 	.id_table	= fimc_driver_ids,
 	.driver = {
-		.name	= MODULE_NAME,
+		.name	= FIMC_MODULE_NAME,
 		.owner	= THIS_MODULE,
 		.pm     = &fimc_pm_ops,
 	}
 };
 
-static int __init fimc_init(void)
+int __init fimc_register_driver(void)
 {
-	int ret = platform_driver_register(&fimc_driver);
-	if (ret)
-		err("platform_driver_register failed: %d\n", ret);
-	return ret;
+	return platform_driver_probe(&fimc_driver, fimc_probe);
 }
 
-static void __exit fimc_exit(void)
+void __exit fimc_unregister_driver(void)
 {
 	platform_driver_unregister(&fimc_driver);
 }
-
-module_init(fimc_init);
-module_exit(fimc_exit);
-
-MODULE_AUTHOR("Sylwester Nawrocki <s.nawrocki@samsung.com>");
-MODULE_DESCRIPTION("S5P FIMC camera host interface/video postprocessor driver");
-MODULE_LICENSE("GPL");
-MODULE_VERSION("1.0.1");
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index fe82bf7..25a8005 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -36,7 +36,7 @@
 /* Time to wait for next frame VSYNC interrupt while stopping operation. */
 #define FIMC_SHUTDOWN_TIMEOUT	((100*HZ)/1000)
 #define MAX_FIMC_CLOCKS		2
-#define MODULE_NAME		"s5p-fimc"
+#define FIMC_MODULE_NAME	"s5p-fimc"
 #define FIMC_MAX_DEVS		4
 #define FIMC_MAX_OUT_BUFS	4
 #define SCALER_MAX_HRATIO	64
@@ -308,6 +308,7 @@ struct fimc_m2m_device {
  * @reqbufs_count: the number of buffers requested in REQBUFS ioctl
  * @input_index: input (camera sensor) index
  * @refcnt: driver's private reference counter
+ * @vid_dev_compat: true to enable full pipeline setup in the driver
  */
 struct fimc_vid_cap {
 	struct fimc_ctx			*ctx;
@@ -325,6 +326,7 @@ struct fimc_vid_cap {
 	unsigned int			reqbufs_count;
 	int				input_index;
 	int				refcnt;
+	bool				vid_dev_compat;
 };
 
 /**
@@ -355,6 +357,7 @@ struct fimc_pix_limit {
  * @has_cistatus2: 1 if CISTATUS2 register is present in this IP revision
  * @has_mainscaler_ext: 1 if extended mainscaler ratios in CIEXTEN register
  *			 are present in this IP revision
+ * @has_cam_if: set if this instance has a camera input interface
  * @pix_limit: pixel size constraints for the scaler
  * @min_inp_pixsize: minimum input pixel size
  * @min_out_pixsize: minimum output pixel size
@@ -367,6 +370,7 @@ struct samsung_fimc_variant {
 	unsigned int	has_out_rot:1;
 	unsigned int	has_cistatus2:1;
 	unsigned int	has_mainscaler_ext:1;
+	unsigned int	has_cam_if:1;
 	struct fimc_pix_limit *pix_limit;
 	u16		min_inp_pixsize;
 	u16		min_out_pixsize;
@@ -387,6 +391,12 @@ struct samsung_fimc_driverdata {
 	int		num_entities;
 };
 
+struct fimc_pipeline {
+	struct media_pipeline *pipe;
+	struct v4l2_subdev *sensor;
+	struct v4l2_subdev *csis;
+};
+
 struct fimc_ctx;
 
 /**
@@ -408,6 +418,7 @@ struct fimc_ctx;
  * @vid_cap:	camera capture device information
  * @state:	flags used to synchronize m2m and capture mode operation
  * @alloc_ctx:	videobuf2 memory allocator context
+ * @pipeline:	fimc video capture pipeline data structure
  */
 struct fimc_dev {
 	spinlock_t			slock;
@@ -427,6 +438,7 @@ struct fimc_dev {
 	struct fimc_vid_cap		vid_cap;
 	unsigned long			state;
 	struct vb2_alloc_ctx		*alloc_ctx;
+	struct fimc_pipeline		pipeline;
 };
 
 /**
@@ -645,6 +657,8 @@ int fimc_prepare_addr(struct fimc_ctx *ctx, struct vb2_buffer *vb,
 int fimc_register_m2m_device(struct fimc_dev *fimc,
 			     struct v4l2_device *v4l2_dev);
 void fimc_unregister_m2m_device(struct fimc_dev *fimc);
+int fimc_register_driver(void);
+void fimc_unregister_driver(void);
 
 /* -----------------------------------------------------*/
 /* fimc-capture.c					*/
diff --git a/drivers/media/video/s5p-fimc/fimc-mdevice.c b/drivers/media/video/s5p-fimc/fimc-mdevice.c
new file mode 100644
index 0000000..10c8d5d
--- /dev/null
+++ b/drivers/media/video/s5p-fimc/fimc-mdevice.c
@@ -0,0 +1,804 @@
+/*
+ * S5P/EXYNOS4 SoC series camera host interface media device driver
+ *
+ * Copyright (C) 2011 Samsung Electronics Co., Ltd.
+ * Contact: Sylwester Nawrocki, <s.nawrocki@samsung.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published
+ * by the Free Software Foundation, either version 2 of the License,
+ * or (at your option) any later version.
+ */
+
+#include <linux/bug.h>
+#include <linux/device.h>
+#include <linux/errno.h>
+#include <linux/i2c.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/types.h>
+#include <linux/slab.h>
+#include <linux/version.h>
+#include <media/media-device.h>
+
+#include "fimc-core.h"
+#include "fimc-mdevice.h"
+#include "mipi-csis.h"
+
+static int __fimc_md_set_camclk(struct fimc_md *fmd,
+				struct fimc_sensor_info *s_info,
+				bool on);
+/**
+ * fimc_pipeline_prepare - update pipeline information with subdevice pointers
+ * @fimc: fimc device terminating the pipeline
+ *
+ * Caller holds the graph mutex.
+ */
+void fimc_pipeline_prepare(struct fimc_dev *fimc, struct media_entity *me)
+{
+	struct media_entity_graph graph;
+	struct v4l2_subdev *sd;
+
+	media_entity_graph_walk_start(&graph, me);
+
+	while ((me = media_entity_graph_walk_next(&graph))) {
+		if (media_entity_type(me) != MEDIA_ENT_T_V4L2_SUBDEV)
+			continue;
+		sd = media_entity_to_v4l2_subdev(me);
+
+		if (sd->grp_id == SENSOR_GROUP_ID)
+			fimc->pipeline.sensor = sd;
+		else if (sd->grp_id == CSIS_GROUP_ID)
+			fimc->pipeline.csis = sd;
+	}
+}
+
+/**
+ * __subdev_set_power - change power state of a single subdev
+ * @sd: subdevice to change power state for
+ * @on: 1 to enable power or 0 to disable
+ *
+ * Return result of s_power subdev operation or -ENXIO if sd argument
+ * is NULL. Return 0 if the subdevice does not implement s_power.
+ */
+static int __subdev_set_power(struct v4l2_subdev *sd, int on)
+{
+	int *use_count;
+	int ret;
+
+	if (sd == NULL)
+		return -ENXIO;
+
+	use_count = &sd->entity.use_count;
+	if (on && (*use_count)++ > 0)
+		return 0;
+	else if (!on && (*use_count == 0 || --(*use_count) > 0))
+		return 0;
+	ret = v4l2_subdev_call(sd, core, s_power, on);
+
+	return ret != -ENOIOCTLCMD ? ret : 0;
+}
+
+/**
+ * fimc_pipeline_s_power - change power state of all pipeline subdevs
+ * @fimc: fimc device terminating the pipeline
+ * @state: 1 to enable power or 0 for power down
+ *
+ * Need to be called with the graph mutex held.
+ */
+int fimc_pipeline_s_power(struct fimc_dev *fimc, int state)
+{
+	int ret = 0;
+
+	if (fimc->pipeline.sensor == NULL)
+		return -ENXIO;
+
+	if (state) {
+		ret = __subdev_set_power(fimc->pipeline.csis, 1);
+		if (ret && ret != -ENXIO)
+			return ret;
+		return __subdev_set_power(fimc->pipeline.sensor, 1);
+	}
+
+	ret = __subdev_set_power(fimc->pipeline.sensor, 0);
+	if (ret)
+		return ret;
+	ret = __subdev_set_power(fimc->pipeline.csis, 0);
+
+	return ret == -ENXIO ? 0 : ret;
+}
+
+/**
+ * __fimc_pipeline_initialize - update the pipeline information, enable power
+ *                              of all pipeline subdevs and the sensor clock
+ * @me: media entity to start graph walk with
+ * @prep: true to acquire sensor (and csis) subdevs
+ *
+ * This function must be called with the graph mutex held.
+ */
+static int __fimc_pipeline_initialize(struct fimc_dev *fimc,
+				      struct media_entity *me, bool prep)
+{
+	int ret;
+
+	if (prep)
+		fimc_pipeline_prepare(fimc, me);
+	if (fimc->pipeline.sensor == NULL)
+		return -EINVAL;
+	ret = fimc_md_set_camclk(fimc->pipeline.sensor, true);
+	if (ret)
+		return ret;
+	return fimc_pipeline_s_power(fimc, 1);
+}
+
+int fimc_pipeline_initialize(struct fimc_dev *fimc, struct media_entity *me,
+			     bool prep)
+{
+	int ret;
+
+	mutex_lock(&me->parent->graph_mutex);
+	ret =  __fimc_pipeline_initialize(fimc, me, prep);
+	mutex_unlock(&me->parent->graph_mutex);
+
+	return ret;
+}
+
+/**
+ * __fimc_pipeline_shutdown - disable the sensor clock and pipeline power
+ * @fimc: fimc device terminating the pipeline
+ *
+ * Disable power of all subdevs in the pipeline and turn off the external
+ * sensor clock.
+ * Called with the graph mutex held.
+ */
+int __fimc_pipeline_shutdown(struct fimc_dev *fimc)
+{
+	int ret = 0;
+
+	if (fimc->pipeline.sensor) {
+		ret = fimc_pipeline_s_power(fimc, 0);
+		fimc_md_set_camclk(fimc->pipeline.sensor, false);
+	}
+	return ret == -ENXIO ? 0 : ret;
+}
+
+int fimc_pipeline_shutdown(struct fimc_dev *fimc)
+{
+	struct media_entity *me = &fimc->vid_cap.vfd->entity;
+	int ret;
+
+	mutex_lock(&me->parent->graph_mutex);
+	ret = __fimc_pipeline_shutdown(fimc);
+	mutex_unlock(&me->parent->graph_mutex);
+
+	return ret;
+}
+
+/**
+ * fimc_pipeline_s_stream - invoke s_stream on pipeline subdevs
+ * @fimc: fimc device terminating the pipeline
+ * @on: passed as the s_stream call argument
+ */
+int fimc_pipeline_s_stream(struct fimc_dev *fimc, int on)
+{
+	struct fimc_pipeline *p = &fimc->pipeline;
+	int ret = 0;
+
+	if (p->sensor == NULL)
+		return -ENODEV;
+
+	if ((on && p->csis) || !on)
+		ret = v4l2_subdev_call(on ? p->csis : p->sensor,
+				       video, s_stream, on);
+	if (ret && ret != -ENOIOCTLCMD)
+		return ret;
+	if ((!on && p->csis) || on)
+		ret = v4l2_subdev_call(on ? p->sensor : p->csis,
+				       video, s_stream, on);
+	return ret == -ENOIOCTLCMD ? 0 : ret;
+}
+
+/*
+ * Sensor subdevice helper functions
+ */
+static struct v4l2_subdev *fimc_md_register_sensor(struct fimc_md *fmd,
+				   struct fimc_sensor_info *s_info)
+{
+	struct i2c_adapter *adapter;
+	struct v4l2_subdev *sd = NULL;
+
+	if (!s_info || !fmd)
+		return NULL;
+
+	adapter = i2c_get_adapter(s_info->pdata->i2c_bus_num);
+	if (!adapter)
+		return NULL;
+	sd = v4l2_i2c_new_subdev_board(&fmd->v4l2_dev, adapter,
+				       s_info->pdata->board_info, NULL);
+	if (IS_ERR_OR_NULL(sd)) {
+		v4l2_err(&fmd->v4l2_dev, "Failed to acquire subdev\n");
+		return NULL;
+	}
+	v4l2_set_subdev_hostdata(sd, s_info);
+	sd->grp_id = SENSOR_GROUP_ID;
+
+	v4l2_info(&fmd->v4l2_dev, "Registered sensor subdevice %s\n",
+		  s_info->pdata->board_info->type);
+	return sd;
+}
+
+static void fimc_md_unregister_sensor(struct v4l2_subdev *sd)
+{
+	struct i2c_client *client = v4l2_get_subdevdata(sd);
+
+	if (!client)
+		return;
+	v4l2_device_unregister_subdev(sd);
+	i2c_unregister_device(client);
+	i2c_put_adapter(client->adapter);
+}
+
+static int fimc_md_register_sensor_entities(struct fimc_md *fmd)
+{
+	struct s5p_platform_fimc *pdata = fmd->pdev->dev.platform_data;
+	struct fimc_dev *fd = NULL;
+	int num_clients, ret, i;
+
+	/*
+	 * Runtime resume one of the FIMC entities to make sure
+	 * the sclk_cam clocks are not globally disabled.
+	 */
+	for (i = 0; !fd && i < ARRAY_SIZE(fmd->fimc); i++)
+		if (fmd->fimc[i])
+			fd = fmd->fimc[i];
+	if (!fd)
+		return -ENXIO;
+	ret = pm_runtime_get_sync(&fd->pdev->dev);
+	if (ret < 0)
+		return ret;
+
+	WARN_ON(pdata->num_clients > ARRAY_SIZE(fmd->sensor));
+	num_clients = min_t(u32, pdata->num_clients, ARRAY_SIZE(fmd->sensor));
+
+	fmd->num_sensors = num_clients;
+	for (i = 0; i < num_clients; i++) {
+		fmd->sensor[i].pdata = &pdata->isp_info[i];
+		ret = __fimc_md_set_camclk(fmd, &fmd->sensor[i], true);
+		if (ret)
+			break;
+		fmd->sensor[i].subdev =
+			fimc_md_register_sensor(fmd, &fmd->sensor[i]);
+		ret = __fimc_md_set_camclk(fmd, &fmd->sensor[i], false);
+		if (ret)
+			break;
+	}
+	pm_runtime_put(&fd->pdev->dev);
+	return ret;
+}
+
+/*
+ * MIPI CSIS and FIMC platform devices registration.
+ */
+static int fimc_register_callback(struct device *dev, void *p)
+{
+	struct fimc_dev *fimc = dev_get_drvdata(dev);
+	struct fimc_md *fmd = p;
+	int ret;
+
+	if (!fimc || !fimc->pdev)
+		return 0;
+	if (fimc->pdev->id < 0 || fimc->pdev->id >= FIMC_MAX_DEVS)
+		return 0;
+
+	fmd->fimc[fimc->pdev->id] = fimc;
+	ret = fimc_register_m2m_device(fimc, &fmd->v4l2_dev);
+	if (ret || !fimc->variant->has_cam_if)
+		return ret;
+	ret = fimc_register_capture_device(fimc, &fmd->v4l2_dev);
+	if (!ret)
+		fimc->vid_cap.vid_dev_compat = fmd->vid_dev_compat;
+	return ret;
+}
+
+static int csis_register_callback(struct device *dev, void *p)
+{
+	struct v4l2_subdev *sd = dev_get_drvdata(dev);
+	struct fimc_md *fmd = p;
+	struct platform_device *pdev;
+	int ret;
+
+	if (!sd)
+		return 0;
+	pdev = v4l2_get_subdevdata(sd);
+	if (!pdev || pdev->id < 0 || pdev->id >= CSIS_MAX_ENTITIES)
+		return 0;
+	v4l2_info(sd, "csis%d sd: %s\n", pdev->id, sd->name);
+
+	fmd->csis[pdev->id].sd = sd;
+	sd->grp_id = CSIS_GROUP_ID;
+	ret = v4l2_device_register_subdev(&fmd->v4l2_dev, sd);
+	if (ret)
+		v4l2_err(&fmd->v4l2_dev,
+			 "Failed to register CSIS subdevice: %d\n", ret);
+	return ret;
+}
+
+/**
+ * fimc_md_register_platform_entities - register FIMC and CSIS media entities
+ */
+static int fimc_md_register_platform_entities(struct fimc_md *fmd)
+{
+	struct device_driver *driver;
+	int ret;
+
+	driver = driver_find(FIMC_MODULE_NAME, &platform_bus_type);
+	if (!driver)
+		return -ENODEV;
+	ret = driver_for_each_device(driver, NULL, fmd,
+				     fimc_register_callback);
+	put_driver(driver);
+	if (ret)
+		return ret;
+
+	driver = driver_find(CSIS_DRIVER_NAME, &platform_bus_type);
+	if (driver) {
+		ret = driver_for_each_device(driver, NULL, fmd,
+					     csis_register_callback);
+		put_driver(driver);
+	}
+	return ret;
+}
+
+static void fimc_md_unregister_entities(struct fimc_md *fmd)
+{
+	int i;
+
+	for (i = 0; i < FIMC_MAX_DEVS; i++) {
+		if (fmd->fimc[i] == NULL)
+			continue;
+		fimc_unregister_m2m_device(fmd->fimc[i]);
+		fimc_unregister_capture_device(fmd->fimc[i]);
+		fmd->fimc[i] = NULL;
+	}
+	for (i = 0; i < CSIS_MAX_ENTITIES; i++) {
+		if (fmd->csis[i].sd == NULL)
+			continue;
+		v4l2_device_unregister_subdev(fmd->csis[i].sd);
+		fmd->csis[i].sd = NULL;
+	}
+	for (i = 0; i < fmd->num_sensors; i++) {
+		if (fmd->sensor[i].subdev == NULL)
+			continue;
+		fimc_md_unregister_sensor(fmd->sensor[i].subdev);
+		fmd->sensor[i].subdev = NULL;
+	}
+}
+
+/**
+ * __fimc_md_create_fimc_links - create links to all FIMC entities
+ * @fmd: fimc media device
+ * @source: the source entity to create links to all fimc entities from
+ * @sensor: sensor subdev linked to fimc (fimc_id) entity
+ * @pad: the source entity pad
+ * @fimc_id: index of the fimc device for which link should be enabled
+ */
+static int __fimc_md_create_fimc_links(struct fimc_md *fmd,
+				       struct media_entity *source,
+				       struct v4l2_subdev *sensor,
+				       int pad, int fimc_id)
+{
+	struct fimc_sensor_info *s_info;
+	struct media_entity *sink;
+	unsigned int flags;
+	int ret, i;
+
+	for (i = 0; i < FIMC_MAX_DEVS; i++) {
+		if (!fmd->fimc[i])
+			break;
+		/*
+		 * Some FIMC variants are not fitted with camera capture
+		 * interface. Skip creating a link from sensor for those.
+		 */
+		if (sensor->grp_id == SENSOR_GROUP_ID &&
+		    !fmd->fimc[i]->variant->has_cam_if)
+			continue;
+
+		flags = (i == fimc_id) ? MEDIA_LNK_FL_ENABLED : 0;
+		sink = &fmd->fimc[i]->vid_cap.vfd->entity;
+		ret = media_entity_create_link(source, 0, sink, 0, flags);
+		if (ret)
+			return ret;
+
+		v4l2_info(&fmd->v4l2_dev, "created link [%s] %c> [%s]",
+			  source->name, flags ? '=' : '-', sink->name);
+
+		if (flags == 0)
+			continue;
+		s_info = v4l2_get_subdev_hostdata(sensor);
+		if (!WARN_ON(s_info == NULL)) {
+			unsigned long irq_flags;
+			spin_lock_irqsave(&fmd->slock, irq_flags);
+			s_info->host = fmd->fimc[i];
+			spin_unlock_irqrestore(&fmd->slock, irq_flags);
+		}
+	}
+	return 0;
+}
+
+/**
+ * fimc_md_create_links - create default links between registered entities
+ *
+ * Parallel interface sensor entities are connected directly to FIMC capture
+ * entities. The sensors using MIPI CSIS bus are connected through immutable
+ * link with CSI receiver entity specified by mux_id. Any registered CSIS
+ * entity has a link to each registered FIMC capture entity. Enabled links
+ * are created by default between each subsequent registered sensor and
+ * subsequent FIMC capture entity. The number of default active links is
+ * determined by the number of available sensors or FIMC entities,
+ * whichever is less.
+ */
+static int fimc_md_create_links(struct fimc_md *fmd)
+{
+	struct v4l2_subdev *sensor, *csis;
+	struct s5p_fimc_isp_info *pdata;
+	struct fimc_sensor_info *s_info;
+	struct media_entity *source;
+	int fimc_id = 0;
+	int i, pad;
+	int ret = 0;
+
+	for (i = 0; i < fmd->num_sensors; i++) {
+		if (fmd->sensor[i].subdev == NULL)
+			continue;
+
+		sensor = fmd->sensor[i].subdev;
+		s_info = v4l2_get_subdev_hostdata(sensor);
+		if (!s_info || !s_info->pdata)
+			continue;
+
+		source = NULL;
+		pdata = s_info->pdata;
+
+		switch (pdata->bus_type) {
+		case FIMC_MIPI_CSI2:
+			if (WARN(pdata->mux_id >= CSIS_MAX_ENTITIES,
+				"Wrong CSI channel id: %d\n", pdata->mux_id))
+				return -EINVAL;
+
+			csis = fmd->csis[pdata->mux_id].sd;
+			if (WARN(csis == NULL,
+				 "MIPI-CSI interface specified "
+				 "but s5p-csis module is not loaded!\n"))
+				continue;
+
+			ret = media_entity_create_link(&sensor->entity, 0,
+					      &csis->entity, CSIS_PAD_SINK,
+					      MEDIA_LNK_FL_IMMUTABLE |
+					      MEDIA_LNK_FL_ENABLED);
+			if (ret)
+				return ret;
+
+			v4l2_info(&fmd->v4l2_dev, "created link [%s] => [%s]",
+				  sensor->entity.name, csis->entity.name);
+
+			source = &csis->entity;
+			pad = CSIS_PAD_SOURCE;
+			break;
+
+		case FIMC_ITU_601...FIMC_ITU_656:
+			source = &sensor->entity;
+			pad = 0;
+			break;
+
+		default:
+			v4l2_err(&fmd->v4l2_dev, "Wrong bus_type: %x\n",
+				 pdata->bus_type);
+			return -EINVAL;
+		}
+		if (source == NULL)
+			continue;
+
+		ret = __fimc_md_create_fimc_links(fmd, source, sensor, pad,
+						 fimc_id++);
+	}
+	return ret;
+}
+
+/*
+ * The peripheral sensor clock management.
+ */
+static int fimc_md_get_clocks(struct fimc_md *fmd)
+{
+	char clk_name[32];
+	struct clk *clock;
+	int i;
+
+	for (i = 0; i < FIMC_MAX_CAMCLKS; i++) {
+		snprintf(clk_name, sizeof(clk_name), "sclk_cam%u", i);
+		clock = clk_get(NULL, clk_name);
+		if (IS_ERR_OR_NULL(clock)) {
+			v4l2_err(&fmd->v4l2_dev, "Failed to get clock: %s",
+				  clk_name);
+			return -ENXIO;
+		}
+		fmd->camclk[i].clock = clock;
+	}
+	return 0;
+}
+
+static void fimc_md_put_clocks(struct fimc_md *fmd)
+{
+	int i = FIMC_MAX_CAMCLKS;
+
+	while (--i >= 0) {
+		if (IS_ERR_OR_NULL(fmd->camclk[i].clock))
+			continue;
+		clk_put(fmd->camclk[i].clock);
+		fmd->camclk[i].clock = NULL;
+	}
+}
+
+static int __fimc_md_set_camclk(struct fimc_md *fmd,
+					 struct fimc_sensor_info *s_info,
+					 bool on)
+{
+	struct s5p_fimc_isp_info *pdata = s_info->pdata;
+	struct fimc_camclk_info *camclk;
+	int ret = 0;
+
+	if (WARN_ON(pdata->clk_id >= FIMC_MAX_CAMCLKS) || fmd == NULL)
+		return -EINVAL;
+
+	if (s_info->clk_on == on)
+		return 0;
+	camclk = &fmd->camclk[pdata->clk_id];
+
+	dbg("camclk %d, f: %lu, clk: %p, on: %d",
+	    pdata->clk_id, pdata->clk_frequency, camclk, on);
+
+	if (on) {
+		if (camclk->use_count > 0 &&
+		    camclk->frequency != pdata->clk_frequency)
+			return -EINVAL;
+
+		if (camclk->use_count++ == 0) {
+			clk_set_rate(camclk->clock, pdata->clk_frequency);
+			camclk->frequency = pdata->clk_frequency;
+			ret = clk_enable(camclk->clock);
+		}
+		s_info->clk_on = 1;
+		dbg("Enabled camclk %d: f: %lu", pdata->clk_id,
+		    clk_get_rate(camclk->clock));
+
+		return ret;
+	}
+
+	if (WARN_ON(camclk->use_count == 0))
+		return 0;
+
+	if (--camclk->use_count == 0) {
+		clk_disable(camclk->clock);
+		s_info->clk_on = 0;
+		dbg("Disabled camclk %d", pdata->clk_id);
+	}
+	return ret;
+}
+
+/**
+ * fimc_md_set_camclk - peripheral sensor clock setup
+ * @sd: sensor subdev to configure sclk_cam clock for
+ * @on: 1 to enable or 0 to disable the clock
+ *
+ * There are 2 separate clock outputs available in the SoC for external
+ * image processors. These clocks are shared between all registered FIMC
+ * devices to which sensors can be attached, either directly or through
+ * the MIPI CSI receiver. The clock is allowed here to be used by
+ * multiple sensors concurrently if they use same frequency.
+ * The per sensor subdev clk_on attribute helps to synchronize accesses
+ * to the sclk_cam clocks from the video and media device nodes.
+ * This function should only be called when the graph mutex is held.
+ */
+int fimc_md_set_camclk(struct v4l2_subdev *sd, bool on)
+{
+	struct fimc_sensor_info *s_info = v4l2_get_subdev_hostdata(sd);
+	struct fimc_md *fmd = entity_to_fimc_mdev(&sd->entity);
+
+	return __fimc_md_set_camclk(fmd, s_info, on);
+}
+
+static int fimc_md_link_notify(struct media_pad *source,
+			       struct media_pad *sink, u32 flags)
+{
+	struct video_device *vid_dev;
+	struct fimc_dev *fimc;
+	int ret = 0;
+
+	if (WARN_ON(media_entity_type(sink->entity) != MEDIA_ENT_T_DEVNODE))
+		return 0;
+
+	vid_dev = media_entity_to_video_device(sink->entity);
+	fimc = video_get_drvdata(vid_dev);
+
+	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
+		ret = __fimc_pipeline_shutdown(fimc);
+		fimc->pipeline.sensor = NULL;
+		fimc->pipeline.csis = NULL;
+		return ret;
+	}
+	/*
+	 * Link activation. Enable power of pipeline elements only if the
+	 * pipeline is already in use, i.e. its video node is opened.
+	 */
+	mutex_lock(&fimc->lock);
+	if (fimc->vid_cap.refcnt > 0)
+		ret = __fimc_pipeline_initialize(fimc, source->entity, true);
+	mutex_unlock(&fimc->lock);
+
+	return ret ? -EPIPE : ret;
+}
+
+static ssize_t fimc_md_sysfs_show(struct device *dev,
+				  struct device_attribute *attr, char *buf)
+{
+	struct platform_device *pdev = to_platform_device(dev);
+	struct fimc_md *fmd = platform_get_drvdata(pdev);
+
+	if (fmd->vid_dev_compat)
+		return strlcpy(buf, "V4L2 video node only API (vid-dev)\n",
+			       PAGE_SIZE);
+
+	return strlcpy(buf, "Sub-device API (sub-dev)\n",
+		       PAGE_SIZE);
+}
+
+static ssize_t fimc_md_sysfs_store(struct device *dev,
+				   struct device_attribute *attr,
+				   const char *buf, size_t count)
+{
+	struct platform_device *pdev = to_platform_device(dev);
+	struct fimc_md *fmd = platform_get_drvdata(pdev);
+	int vdev_compat;
+	int i;
+
+	if (!strcmp(buf, "vid-dev\n"))
+		vdev_compat = 1;
+	else if (!strcmp(buf, "sub-dev\n"))
+		vdev_compat = 0;
+	else
+		return count;
+
+	fmd->vid_dev_compat = vdev_compat;
+	for (i = 0; i < FIMC_MAX_DEVS; i++)
+		if (fmd->fimc[i])
+			fmd->fimc[i]->vid_cap.vid_dev_compat = vdev_compat;
+	return count;
+}
+/*
+ * This device attribute is to select video pipeline configuration method.
+ * There are following valid values:
+ *  vid-dev - for V4L2 video node API only, subdevice will be configured
+ *  by the host driver.
+ *  sub-dev - for media controller API, subdevs must be configured in user
+ *  space before starting streaming.
+ */
+static DEVICE_ATTR(subdev_conf_mode, S_IWUSR | S_IRUGO,
+		   fimc_md_sysfs_show, fimc_md_sysfs_store);
+
+static int __devinit fimc_md_probe(struct platform_device *pdev)
+{
+	struct v4l2_device *v4l2_dev;
+	struct fimc_md *fmd;
+	int ret;
+
+	if (WARN(!pdev->dev.platform_data, "Platform data not specified!\n"))
+		return -EINVAL;
+
+	fmd = kzalloc(sizeof(struct fimc_md), GFP_KERNEL);
+	if (!fmd)
+		return -ENOMEM;
+
+	spin_lock_init(&fmd->slock);
+	fmd->pdev = pdev;
+
+	strlcpy(fmd->media_dev.model, "SAMSUNG S5P FIMC",
+		sizeof(fmd->media_dev.model));
+	fmd->media_dev.link_notify = fimc_md_link_notify;
+	fmd->media_dev.dev = &pdev->dev;
+
+	v4l2_dev = &fmd->v4l2_dev;
+	v4l2_dev->mdev = &fmd->media_dev;
+	snprintf(v4l2_dev->name, sizeof(v4l2_dev->name), "%s",
+		 dev_name(&pdev->dev));
+
+	ret = v4l2_device_register(&pdev->dev, &fmd->v4l2_dev);
+	if (ret < 0) {
+		v4l2_err(v4l2_dev, "Failed to register v4l2_device: %d\n", ret);
+		goto err1;
+	}
+	ret = media_device_register(&fmd->media_dev);
+	if (ret < 0) {
+		v4l2_err(v4l2_dev, "Failed to register media device: %d\n", ret);
+		goto err2;
+	}
+	ret = fimc_md_get_clocks(fmd);
+	if (ret)
+		goto err3;
+
+	fmd->vid_dev_compat = 1;
+	ret = fimc_md_register_platform_entities(fmd);
+	if (ret)
+		goto err3;
+
+	ret = fimc_md_register_sensor_entities(fmd);
+	if (ret)
+		goto err3;
+	ret = fimc_md_create_links(fmd);
+	if (ret)
+		goto err3;
+	ret = v4l2_device_register_subdev_nodes(&fmd->v4l2_dev);
+	if (ret)
+		goto err3;
+	ret = device_create_file(&pdev->dev, &dev_attr_subdev_conf_mode);
+	if (!ret) {
+		platform_set_drvdata(pdev, fmd);
+		return 0;
+	}
+err3:
+	media_device_unregister(&fmd->media_dev);
+	fimc_md_put_clocks(fmd);
+	fimc_md_unregister_entities(fmd);
+err2:
+	v4l2_device_unregister(&fmd->v4l2_dev);
+err1:
+	kfree(fmd);
+	return ret;
+}
+
+static int __devexit fimc_md_remove(struct platform_device *pdev)
+{
+	struct fimc_md *fmd = platform_get_drvdata(pdev);
+
+	if (!fmd)
+		return 0;
+	device_remove_file(&pdev->dev, &dev_attr_subdev_conf_mode);
+	fimc_md_unregister_entities(fmd);
+	media_device_unregister(&fmd->media_dev);
+	fimc_md_put_clocks(fmd);
+	kfree(fmd);
+	return 0;
+}
+
+static struct platform_driver fimc_md_driver = {
+	.probe		= fimc_md_probe,
+	.remove		= __devexit_p(fimc_md_remove),
+	.driver = {
+		.name	= "s5p-fimc-md",
+		.owner	= THIS_MODULE,
+	}
+};
+
+int __init fimc_md_init(void)
+{
+	int ret;
+	request_module("s5p-csis");
+	ret = fimc_register_driver();
+	if (ret)
+		return ret;
+	return platform_driver_register(&fimc_md_driver);
+}
+void __exit fimc_md_exit(void)
+{
+	platform_driver_unregister(&fimc_md_driver);
+	fimc_unregister_driver();
+}
+
+module_init(fimc_md_init);
+module_exit(fimc_md_exit);
+
+MODULE_AUTHOR("Sylwester Nawrocki <s.nawrocki@samsung.com>");
+MODULE_DESCRIPTION("S5P FIMC camera host interface/video postprocessor driver");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("2.0.1");
diff --git a/drivers/media/video/s5p-fimc/fimc-mdevice.h b/drivers/media/video/s5p-fimc/fimc-mdevice.h
new file mode 100644
index 0000000..7f39b5a
--- /dev/null
+++ b/drivers/media/video/s5p-fimc/fimc-mdevice.h
@@ -0,0 +1,118 @@
+/*
+ * Copyright (C) 2011 Samsung Electronics Co., Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef FIMC_MDEVICE_H_
+#define FIMC_MDEVICE_H_
+
+#include <linux/clk.h>
+#include <linux/platform_device.h>
+#include <linux/mutex.h>
+#include <media/media-device.h>
+#include <media/media-entity.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+
+#include "fimc-core.h"
+#include "mipi-csis.h"
+
+/* Group IDs of sensor, MIPI CSIS and the writeback subdevs. */
+#define SENSOR_GROUP_ID		(1 << 8)
+#define CSIS_GROUP_ID		(1 << 9)
+#define WRITEBACK_GROUP_ID	(1 << 10)
+
+#define FIMC_MAX_SENSORS	8
+#define FIMC_MAX_CAMCLKS	2
+
+struct fimc_csis_info {
+	struct v4l2_subdev *sd;
+	int id;
+};
+
+struct fimc_camclk_info {
+	struct clk *clock;
+	int use_count;
+	unsigned long frequency;
+};
+
+/**
+ * struct fimc_sensor_info - image data source subdev information
+ * @pdata: sensor's atrributes passed as media device's platform data
+ * @subdev: image sensor v4l2 subdev
+ * @host: fimc device the sensor is currently linked to
+ * @clk_on: sclk_cam clock's state associated with this subdev
+ *
+ * This data structure applies to image sensor and the writeback subdevs.
+ */
+struct fimc_sensor_info {
+	struct s5p_fimc_isp_info *pdata;
+	struct v4l2_subdev *subdev;
+	struct fimc_dev *host;
+	bool clk_on;
+};
+
+/**
+ * struct fimc_md - fimc media device information
+ * @csis: MIPI CSIS subdevs data
+ * @sensor: array of registered sensor subdevs
+ * @num_sensors: actual number of registered sensors
+ * @camclk: external sensor clock information
+ * @fimc: array of registered fimc devices
+ * @media_dev: top level media device
+ * @v4l2_dev: top level v4l2_device holding up the subdevs
+ * @pdev: platform device this media device is hooked up into
+ * @vid_dev_compat: 1 if subdevs are configured by the host driver
+ * @slock: spinlock protecting @sensor array
+ */
+struct fimc_md {
+	struct fimc_csis_info csis[CSIS_MAX_ENTITIES];
+	struct fimc_sensor_info sensor[FIMC_MAX_SENSORS];
+	int num_sensors;
+	struct fimc_camclk_info camclk[FIMC_MAX_CAMCLKS];
+	struct fimc_dev *fimc[FIMC_MAX_DEVS];
+	struct media_device media_dev;
+	struct v4l2_device v4l2_dev;
+	struct platform_device *pdev;
+	int vid_dev_compat;
+	spinlock_t slock;
+};
+
+#define is_subdev_pad(pad) (pad == NULL || \
+	media_entity_type(pad->entity) == MEDIA_ENT_T_V4L2_SUBDEV)
+
+#define me_subtype(me) \
+	((me->type) & (MEDIA_ENT_TYPE_MASK | MEDIA_ENT_SUBTYPE_MASK))
+
+#define subdev_has_devnode(__sd) (__sd->flags & V4L2_SUBDEV_FL_HAS_DEVNODE)
+
+static inline struct fimc_md *entity_to_fimc_mdev(struct media_entity *me)
+{
+	return me->parent == NULL ? NULL :
+		container_of(me->parent, struct fimc_md, media_dev);
+}
+
+static inline void fimc_md_graph_lock(struct fimc_dev *fimc)
+{
+	BUG_ON(fimc->vid_cap.vfd == NULL);
+	mutex_lock(&fimc->vid_cap.vfd->entity.parent->graph_mutex);
+}
+
+static inline void fimc_md_graph_unlock(struct fimc_dev *fimc)
+{
+	BUG_ON(fimc->vid_cap.vfd == NULL);
+	mutex_unlock(&fimc->vid_cap.vfd->entity.parent->graph_mutex);
+}
+
+int fimc_md_set_camclk(struct v4l2_subdev *sd, bool on);
+void fimc_pipeline_prepare(struct fimc_dev *fimc, struct media_entity *me);
+int fimc_pipeline_initialize(struct fimc_dev *fimc, struct media_entity *me,
+			     bool resume);
+int fimc_pipeline_shutdown(struct fimc_dev *fimc);
+int fimc_pipeline_s_power(struct fimc_dev *fimc, int state);
+int fimc_pipeline_s_stream(struct fimc_dev *fimc, int state);
+
+#endif
diff --git a/include/media/s5p_fimc.h b/include/media/s5p_fimc.h
index 9fdff8a..086a7aa 100644
--- a/include/media/s5p_fimc.h
+++ b/include/media/s5p_fimc.h
@@ -36,6 +36,7 @@ struct i2c_board_info;
  * @csi_data_align: MIPI-CSI interface data alignment in bits
  * @i2c_bus_num: i2c control bus id the sensor is attached to
  * @mux_id: FIMC camera interface multiplexer index (separate for MIPI and ITU)
+ * @clk_id: index of the SoC peripheral clock for sensors
  * @flags: flags defining bus signals polarity inversion (High by default)
  */
 struct s5p_fimc_isp_info {
@@ -46,6 +47,7 @@ struct s5p_fimc_isp_info {
 	u16 i2c_bus_num;
 	u16 mux_id;
 	u16 flags;
+	u8 clk_id;
 };
 
 /**
-- 
1.7.5.4

[PATCH v3 09/19] s5p-fimc: Conversion to use struct v4l2_fh

[Sylwester Nawrocki]

This is a prerequisite for the patch converting the driver to use
the control framework. As the capture driver does not use per file
handle contexts, two separate ioctl handlers are created for it
(vidioc_try_fmt_mplane, and vidioc_g_fmt_mplane) so there is no
handlers shared between the memory-to-memory and capture video node.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |  114 +++++++++--------
 drivers/media/video/s5p-fimc/fimc-core.c    |  182 +++++++++++++++------------
 drivers/media/video/s5p-fimc/fimc-core.h    |   10 +-
 3 files changed, 168 insertions(+), 138 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index 70892e4..0ceb16d 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -260,7 +260,10 @@ static struct vb2_ops fimc_capture_qops = {
 static int fimc_capture_open(struct file *file)
 {
 	struct fimc_dev *fimc = video_drvdata(file);
-	int ret = 0;
+	int ret = v4l2_fh_open(file);
+
+	if (ret)
+		return ret;
 
 	dbg("pid: %d, state: 0x%lx", task_pid_nr(current), fimc->state);
 
@@ -269,11 +272,12 @@ static int fimc_capture_open(struct file *file)
 		return -EBUSY;
 
 	ret = pm_runtime_get_sync(&fimc->pdev->dev);
-	if (ret)
+	if (ret < 0) {
+		v4l2_fh_release(file);
 		return ret;
+	}
 
 	++fimc->vid_cap.refcnt;
-	file->private_data = fimc->vid_cap.ctx;
 
 	return 0;
 }
@@ -291,22 +295,20 @@ static int fimc_capture_close(struct file *file)
 
 	pm_runtime_put_sync(&fimc->pdev->dev);
 
-	return 0;
+	return v4l2_fh_release(file);
 }
 
 static unsigned int fimc_capture_poll(struct file *file,
 				      struct poll_table_struct *wait)
 {
-	struct fimc_ctx *ctx = file->private_data;
-	struct fimc_dev *fimc = ctx->fimc_dev;
+	struct fimc_dev *fimc = video_drvdata(file);
 
 	return vb2_poll(&fimc->vid_cap.vbq, file, wait);
 }
 
 static int fimc_capture_mmap(struct file *file, struct vm_area_struct *vma)
 {
-	struct fimc_ctx *ctx = file->private_data;
-	struct fimc_dev *fimc = ctx->fimc_dev;
+	struct fimc_dev *fimc = video_drvdata(file);
 
 	return vb2_mmap(&fimc->vid_cap.vbq, vma);
 }
@@ -324,8 +326,7 @@ static const struct v4l2_file_operations fimc_capture_fops = {
 static int fimc_vidioc_querycap_capture(struct file *file, void *priv,
 					struct v4l2_capability *cap)
 {
-	struct fimc_ctx *ctx = file->private_data;
-	struct fimc_dev *fimc = ctx->fimc_dev;
+	struct fimc_dev *fimc = video_drvdata(file);
 
 	strncpy(cap->driver, fimc->pdev->name, sizeof(cap->driver) - 1);
 	strncpy(cap->card, fimc->pdev->name, sizeof(cap->card) - 1);
@@ -372,20 +373,41 @@ static int sync_capture_fmt(struct fimc_ctx *ctx)
 	return 0;
 }
 
+static int fimc_cap_g_fmt_mplane(struct file *file, void *fh,
+				 struct v4l2_format *f)
+{
+	struct fimc_dev *fimc = video_drvdata(file);
+	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
+
+	if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+		return -EINVAL;
+
+	return fimc_fill_format(&ctx->d_frame, f);
+}
+
+static int fimc_cap_try_fmt_mplane(struct file *file, void *fh,
+				   struct v4l2_format *f)
+{
+	struct fimc_dev *fimc = video_drvdata(file);
+	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
+
+	return fimc_try_fmt_mplane(ctx, f);
+}
+
 static int fimc_cap_s_fmt_mplane(struct file *file, void *priv,
 				 struct v4l2_format *f)
 {
-	struct fimc_ctx *ctx = priv;
-	struct fimc_dev *fimc = ctx->fimc_dev;
-	struct fimc_frame *frame;
+	struct fimc_dev *fimc = video_drvdata(file);
+	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
 	struct v4l2_pix_format_mplane *pix;
+	struct fimc_frame *frame;
 	int ret;
 	int i;
 
 	if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
 		return -EINVAL;
 
-	ret = fimc_vidioc_try_fmt_mplane(file, priv, f);
+	ret = fimc_try_fmt_mplane(ctx, f);
 	if (ret)
 		return ret;
 
@@ -427,7 +449,7 @@ static int fimc_cap_s_fmt_mplane(struct file *file, void *priv,
 static int fimc_cap_enum_input(struct file *file, void *priv,
 			       struct v4l2_input *i)
 {
-	struct fimc_ctx *ctx = priv;
+	struct fimc_dev *fimc = video_drvdata(file);
 
 	if (i->index != 0)
 		return -EINVAL;
@@ -451,8 +473,8 @@ static int fimc_cap_g_input(struct file *file, void *priv, unsigned int *i)
 static int fimc_cap_streamon(struct file *file, void *priv,
 			     enum v4l2_buf_type type)
 {
-	struct fimc_ctx *ctx = priv;
-	struct fimc_dev *fimc = ctx->fimc_dev;
+	struct fimc_dev *fimc = video_drvdata(file);
+	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
 
 	if (fimc_capture_active(fimc) || !fimc->vid_cap.sd)
 		return -EBUSY;
@@ -468,8 +490,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 static int fimc_cap_streamoff(struct file *file, void *priv,
 			    enum v4l2_buf_type type)
 {
-	struct fimc_ctx *ctx = priv;
-	struct fimc_dev *fimc = ctx->fimc_dev;
+	struct fimc_dev *fimc = video_drvdata(file);
 
 	return vb2_streamoff(&fimc->vid_cap.vbq, type);
 }
@@ -477,47 +498,43 @@ static int fimc_cap_streamoff(struct file *file, void *priv,
 static int fimc_cap_reqbufs(struct file *file, void *priv,
 			    struct v4l2_requestbuffers *reqbufs)
 {
-	struct fimc_ctx *ctx = priv;
-	struct fimc_vid_cap *cap = &ctx->fimc_dev->vid_cap;
-	int ret;
-
+	struct fimc_dev *fimc = video_drvdata(file);
+	int ret = vb2_reqbufs(&fimc->vid_cap.vbq, reqbufs);
 
-	ret = vb2_reqbufs(&cap->vbq, reqbufs);
 	if (!ret)
-		cap->reqbufs_count = reqbufs->count;
-
+		fimc->vid_cap.reqbufs_count = reqbufs->count;
 	return ret;
 }
 
 static int fimc_cap_querybuf(struct file *file, void *priv,
 			   struct v4l2_buffer *buf)
 {
-	struct fimc_ctx *ctx = priv;
-	struct fimc_vid_cap *cap = &ctx->fimc_dev->vid_cap;
+	struct fimc_dev *fimc = video_drvdata(file);
 
-	return vb2_querybuf(&cap->vbq, buf);
+	return vb2_querybuf(&fimc->vid_cap.vbq, buf);
 }
 
 static int fimc_cap_qbuf(struct file *file, void *priv,
 			  struct v4l2_buffer *buf)
 {
-	struct fimc_ctx *ctx = priv;
-	struct fimc_vid_cap *cap = &ctx->fimc_dev->vid_cap;
-	return vb2_qbuf(&cap->vbq, buf);
+	struct fimc_dev *fimc = video_drvdata(file);
+
+	return vb2_qbuf(&fimc->vid_cap.vbq, buf);
 }
 
 static int fimc_cap_dqbuf(struct file *file, void *priv,
 			   struct v4l2_buffer *buf)
 {
-	struct fimc_ctx *ctx = priv;
-	return vb2_dqbuf(&ctx->fimc_dev->vid_cap.vbq, buf,
-		file->f_flags & O_NONBLOCK);
+	struct fimc_dev *fimc = video_drvdata(file);
+
+	return vb2_dqbuf(&fimc->vid_cap.vbq, buf, file->f_flags & O_NONBLOCK);
 }
 
 static int fimc_cap_s_ctrl(struct file *file, void *priv,
-			 struct v4l2_control *ctrl)
+			   struct v4l2_control *ctrl)
 {
-	struct fimc_ctx *ctx = priv;
+	struct fimc_dev *fimc = video_drvdata(file);
+	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
 	int ret = -EINVAL;
 
 	/* Allow any controls but 90/270 rotation while streaming */
@@ -540,14 +557,12 @@ static int fimc_cap_s_ctrl(struct file *file, void *priv,
 static int fimc_cap_cropcap(struct file *file, void *fh,
 			    struct v4l2_cropcap *cr)
 {
-	struct fimc_frame *f;
-	struct fimc_ctx *ctx = fh;
+	struct fimc_dev *fimc = video_drvdata(file);
+	struct fimc_frame *f = &fimc->vid_cap.ctx->s_frame;
 
 	if (cr->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
 		return -EINVAL;
 
-	f = &ctx->s_frame;
-
 	cr->bounds.left		= 0;
 	cr->bounds.top		= 0;
 	cr->bounds.width	= f->o_width;
@@ -559,10 +574,8 @@ static int fimc_cap_cropcap(struct file *file, void *fh,
 
 static int fimc_cap_g_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 {
-	struct fimc_frame *f;
-	struct fimc_ctx *ctx = file->private_data;
-
-	f = &ctx->s_frame;
+	struct fimc_dev *fimc = video_drvdata(file);
+	struct fimc_frame *f = &fimc->vid_cap.ctx->s_frame;
 
 	cr->c.left	= f->offs_h;
 	cr->c.top	= f->offs_v;
@@ -572,12 +585,11 @@ static int fimc_cap_g_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 	return 0;
 }
 
-static int fimc_cap_s_crop(struct file *file, void *fh,
-			       struct v4l2_crop *cr)
+static int fimc_cap_s_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 {
+	struct fimc_dev *fimc = video_drvdata(file);
+	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
 	struct fimc_frame *f;
-	struct fimc_ctx *ctx = file->private_data;
-	struct fimc_dev *fimc = ctx->fimc_dev;
 	int ret = -EINVAL;
 
 	if (fimc_capture_active(fimc))
@@ -615,9 +627,9 @@ static const struct v4l2_ioctl_ops fimc_capture_ioctl_ops = {
 	.vidioc_querycap		= fimc_vidioc_querycap_capture,
 
 	.vidioc_enum_fmt_vid_cap_mplane	= fimc_vidioc_enum_fmt_mplane,
-	.vidioc_try_fmt_vid_cap_mplane	= fimc_vidioc_try_fmt_mplane,
+	.vidioc_try_fmt_vid_cap_mplane	= fimc_cap_try_fmt_mplane,
 	.vidioc_s_fmt_vid_cap_mplane	= fimc_cap_s_fmt_mplane,
-	.vidioc_g_fmt_vid_cap_mplane	= fimc_vidioc_g_fmt_mplane,
+	.vidioc_g_fmt_vid_cap_mplane	= fimc_cap_g_fmt_mplane,
 
 	.vidioc_reqbufs			= fimc_cap_reqbufs,
 	.vidioc_querybuf		= fimc_cap_querybuf,
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index 59190aa..1f84cc6 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -777,10 +777,10 @@ static struct vb2_ops fimc_qops = {
 	.start_streaming = start_streaming,
 };
 
-static int fimc_m2m_querycap(struct file *file, void *priv,
-			   struct v4l2_capability *cap)
+static int fimc_m2m_querycap(struct file *file, void *fh,
+			     struct v4l2_capability *cap)
 {
-	struct fimc_ctx *ctx = file->private_data;
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
 	struct fimc_dev *fimc = ctx->fimc_dev;
 
 	strncpy(cap->driver, fimc->pdev->name, sizeof(cap->driver) - 1);
@@ -808,42 +808,41 @@ int fimc_vidioc_enum_fmt_mplane(struct file *file, void *priv,
 	return 0;
 }
 
-int fimc_vidioc_g_fmt_mplane(struct file *file, void *priv,
-			     struct v4l2_format *f)
+int fimc_fill_format(struct fimc_frame *frame, struct v4l2_format *f)
 {
-	struct fimc_ctx *ctx = priv;
-	struct fimc_frame *frame;
-	struct v4l2_pix_format_mplane *pixm;
+	struct v4l2_pix_format_mplane *pixm = &f->fmt.pix_mp;
 	int i;
 
-	frame = ctx_get_frame(ctx, f->type);
-	if (IS_ERR(frame))
-		return PTR_ERR(frame);
-
-	pixm = &f->fmt.pix_mp;
-
-	pixm->width		= frame->width;
-	pixm->height		= frame->height;
-	pixm->field		= V4L2_FIELD_NONE;
-	pixm->pixelformat	= frame->fmt->fourcc;
-	pixm->colorspace	= V4L2_COLORSPACE_JPEG;
-	pixm->num_planes	= frame->fmt->memplanes;
+	pixm->width = frame->o_width;
+	pixm->height = frame->o_height;
+	pixm->field = V4L2_FIELD_NONE;
+	pixm->pixelformat = frame->fmt->fourcc;
+	pixm->colorspace = V4L2_COLORSPACE_JPEG;
+	pixm->num_planes = frame->fmt->memplanes;
 
 	for (i = 0; i < pixm->num_planes; ++i) {
-		int bpl = frame->o_width;
-
+		int bpl = frame->f_width;
 		if (frame->fmt->colplanes == 1) /* packed formats */
 			bpl = (bpl * frame->fmt->depth[0]) / 8;
-
 		pixm->plane_fmt[i].bytesperline = bpl;
-
 		pixm->plane_fmt[i].sizeimage = (frame->o_width *
 			frame->o_height * frame->fmt->depth[i]) / 8;
 	}
-
 	return 0;
 }
 
+static int fimc_m2m_g_fmt_mplane(struct file *file, void *fh,
+				 struct v4l2_format *f)
+{
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
+	struct fimc_frame *frame = ctx_get_frame(ctx, f->type);
+
+	if (IS_ERR(frame))
+		return PTR_ERR(frame);
+
+	return fimc_fill_format(frame, f);
+}
+
 struct fimc_fmt *find_format(struct v4l2_format *f, unsigned int mask)
 {
 	struct fimc_fmt *fmt;
@@ -874,11 +873,8 @@ struct fimc_fmt *find_mbus_format(struct v4l2_mbus_framefmt *f,
 	return (i == ARRAY_SIZE(fimc_formats)) ? NULL : fmt;
 }
 
-
-int fimc_vidioc_try_fmt_mplane(struct file *file, void *priv,
-			       struct v4l2_format *f)
+int fimc_try_fmt_mplane(struct fimc_ctx *ctx, struct v4l2_format *f)
 {
-	struct fimc_ctx *ctx = priv;
 	struct fimc_dev *fimc = ctx->fimc_dev;
 	struct samsung_fimc_variant *variant = fimc->variant;
 	struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp;
@@ -957,17 +953,25 @@ int fimc_vidioc_try_fmt_mplane(struct file *file, void *priv,
 	return 0;
 }
 
-static int fimc_m2m_s_fmt_mplane(struct file *file, void *priv,
+static int fimc_m2m_try_fmt_mplane(struct file *file, void *fh,
+				   struct v4l2_format *f)
+{
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
+
+	return fimc_try_fmt_mplane(ctx, f);
+}
+
+static int fimc_m2m_s_fmt_mplane(struct file *file, void *fh,
 				 struct v4l2_format *f)
 {
-	struct fimc_ctx *ctx = priv;
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
 	struct fimc_dev *fimc = ctx->fimc_dev;
 	struct vb2_queue *vq;
 	struct fimc_frame *frame;
 	struct v4l2_pix_format_mplane *pix;
 	int i, ret = 0;
 
-	ret = fimc_vidioc_try_fmt_mplane(file, priv, f);
+	ret = fimc_try_fmt_mplane(ctx, f);
 	if (ret)
 		return ret;
 
@@ -978,15 +982,10 @@ static int fimc_m2m_s_fmt_mplane(struct file *file, void *priv,
 		return -EBUSY;
 	}
 
-	if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
+	if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
 		frame = &ctx->s_frame;
-	} else if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+	else
 		frame = &ctx->d_frame;
-	} else {
-		v4l2_err(fimc->m2m.vfd,
-			 "Wrong buffer/video queue type (%d)\n", f->type);
-		return -EINVAL;
-	}
 
 	pix = &f->fmt.pix_mp;
 	frame->fmt = find_format(f, FMT_FLAGS_M2M);
@@ -1018,39 +1017,42 @@ static int fimc_m2m_s_fmt_mplane(struct file *file, void *priv,
 	return 0;
 }
 
-static int fimc_m2m_reqbufs(struct file *file, void *priv,
-			  struct v4l2_requestbuffers *reqbufs)
+static int fimc_m2m_reqbufs(struct file *file, void *fh,
+			    struct v4l2_requestbuffers *reqbufs)
 {
-	struct fimc_ctx *ctx = priv;
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
+
 	return v4l2_m2m_reqbufs(file, ctx->m2m_ctx, reqbufs);
 }
 
-static int fimc_m2m_querybuf(struct file *file, void *priv,
-			   struct v4l2_buffer *buf)
+static int fimc_m2m_querybuf(struct file *file, void *fh,
+			     struct v4l2_buffer *buf)
 {
-	struct fimc_ctx *ctx = priv;
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
+
 	return v4l2_m2m_querybuf(file, ctx->m2m_ctx, buf);
 }
 
-static int fimc_m2m_qbuf(struct file *file, void *priv,
-			  struct v4l2_buffer *buf)
+static int fimc_m2m_qbuf(struct file *file, void *fh,
+			 struct v4l2_buffer *buf)
 {
-	struct fimc_ctx *ctx = priv;
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
 
 	return v4l2_m2m_qbuf(file, ctx->m2m_ctx, buf);
 }
 
-static int fimc_m2m_dqbuf(struct file *file, void *priv,
-			   struct v4l2_buffer *buf)
+static int fimc_m2m_dqbuf(struct file *file, void *fh,
+			  struct v4l2_buffer *buf)
 {
-	struct fimc_ctx *ctx = priv;
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
+
 	return v4l2_m2m_dqbuf(file, ctx->m2m_ctx, buf);
 }
 
-static int fimc_m2m_streamon(struct file *file, void *priv,
-			   enum v4l2_buf_type type)
+static int fimc_m2m_streamon(struct file *file, void *fh,
+			     enum v4l2_buf_type type)
 {
-	struct fimc_ctx *ctx = priv;
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
 
 	/* The source and target color format need to be set */
 	if (V4L2_TYPE_IS_OUTPUT(type)) {
@@ -1063,17 +1065,19 @@ static int fimc_m2m_streamon(struct file *file, void *priv,
 	return v4l2_m2m_streamon(file, ctx->m2m_ctx, type);
 }
 
-static int fimc_m2m_streamoff(struct file *file, void *priv,
+static int fimc_m2m_streamoff(struct file *file, void *fh,
 			    enum v4l2_buf_type type)
 {
-	struct fimc_ctx *ctx = priv;
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
+
 	return v4l2_m2m_streamoff(file, ctx->m2m_ctx, type);
 }
 
-int fimc_vidioc_queryctrl(struct file *file, void *priv,
-			    struct v4l2_queryctrl *qc)
+int fimc_vidioc_queryctrl(struct file *file, void *fh,
+			  struct v4l2_queryctrl *qc)
 {
-	struct fimc_ctx *ctx = priv;
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
+	struct fimc_dev *fimc = ctx->fimc_dev;
 	struct v4l2_queryctrl *c;
 	int ret = -EINVAL;
 
@@ -1090,10 +1094,9 @@ int fimc_vidioc_queryctrl(struct file *file, void *priv,
 	return ret;
 }
 
-int fimc_vidioc_g_ctrl(struct file *file, void *priv,
-			 struct v4l2_control *ctrl)
+int fimc_vidioc_g_ctrl(struct file *file, void *fh, struct v4l2_control *ctrl)
 {
-	struct fimc_ctx *ctx = priv;
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
 	struct fimc_dev *fimc = ctx->fimc_dev;
 
 	switch (ctrl->id) {
@@ -1186,10 +1189,10 @@ int fimc_s_ctrl(struct fimc_ctx *ctx, struct v4l2_control *ctrl)
 	return 0;
 }
 
-static int fimc_m2m_s_ctrl(struct file *file, void *priv,
+static int fimc_m2m_s_ctrl(struct file *file, void *fh,
 			   struct v4l2_control *ctrl)
 {
-	struct fimc_ctx *ctx = priv;
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
 	int ret = 0;
 
 	ret = check_ctrl_val(ctx, ctrl);
@@ -1201,10 +1204,10 @@ static int fimc_m2m_s_ctrl(struct file *file, void *priv,
 }
 
 static int fimc_m2m_cropcap(struct file *file, void *fh,
-			struct v4l2_cropcap *cr)
+			    struct v4l2_cropcap *cr)
 {
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
 	struct fimc_frame *frame;
-	struct fimc_ctx *ctx = fh;
 
 	frame = ctx_get_frame(ctx, cr->type);
 	if (IS_ERR(frame))
@@ -1221,8 +1224,8 @@ static int fimc_m2m_cropcap(struct file *file, void *fh,
 
 static int fimc_m2m_g_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 {
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
 	struct fimc_frame *frame;
-	struct fimc_ctx *ctx = file->private_data;
 
 	frame = ctx_get_frame(ctx, cr->type);
 	if (IS_ERR(frame))
@@ -1300,7 +1303,7 @@ int fimc_try_crop(struct fimc_ctx *ctx, struct v4l2_crop *cr)
 
 static int fimc_m2m_s_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 {
-	struct fimc_ctx *ctx = file->private_data;
+	struct fimc_ctx *ctx = fh_to_ctx(fh);
 	struct fimc_dev *fimc = ctx->fimc_dev;
 	struct fimc_frame *f;
 	int ret;
@@ -1347,11 +1350,11 @@ static const struct v4l2_ioctl_ops fimc_m2m_ioctl_ops = {
 	.vidioc_enum_fmt_vid_cap_mplane	= fimc_vidioc_enum_fmt_mplane,
 	.vidioc_enum_fmt_vid_out_mplane	= fimc_vidioc_enum_fmt_mplane,
 
-	.vidioc_g_fmt_vid_cap_mplane	= fimc_vidioc_g_fmt_mplane,
-	.vidioc_g_fmt_vid_out_mplane	= fimc_vidioc_g_fmt_mplane,
+	.vidioc_g_fmt_vid_cap_mplane	= fimc_m2m_g_fmt_mplane,
+	.vidioc_g_fmt_vid_out_mplane	= fimc_m2m_g_fmt_mplane,
 
-	.vidioc_try_fmt_vid_cap_mplane	= fimc_vidioc_try_fmt_mplane,
-	.vidioc_try_fmt_vid_out_mplane	= fimc_vidioc_try_fmt_mplane,
+	.vidioc_try_fmt_vid_cap_mplane	= fimc_m2m_try_fmt_mplane,
+	.vidioc_try_fmt_vid_out_mplane	= fimc_m2m_try_fmt_mplane,
 
 	.vidioc_s_fmt_vid_cap_mplane	= fimc_m2m_s_fmt_mplane,
 	.vidioc_s_fmt_vid_out_mplane	= fimc_m2m_s_fmt_mplane,
@@ -1407,7 +1410,8 @@ static int queue_init(void *priv, struct vb2_queue *src_vq,
 static int fimc_m2m_open(struct file *file)
 {
 	struct fimc_dev *fimc = video_drvdata(file);
-	struct fimc_ctx *ctx = NULL;
+	struct fimc_ctx *ctx;
+	int ret;
 
 	dbg("pid: %d, state: 0x%lx, refcnt: %d",
 		task_pid_nr(current), fimc->state, fimc->vid_cap.refcnt);
@@ -1422,13 +1426,18 @@ static int fimc_m2m_open(struct file *file)
 	ctx = kzalloc(sizeof *ctx, GFP_KERNEL);
 	if (!ctx)
 		return -ENOMEM;
+	ret = v4l2_fh_init(&ctx->fh, fimc->m2m.vfd);
+	if (ret)
+		goto error;
+
+	file->private_data = &ctx->fh;
+	v4l2_fh_add(&ctx->fh);
 
-	file->private_data = ctx;
 	ctx->fimc_dev = fimc;
 	/* Default color format */
 	ctx->s_frame.fmt = &fimc_formats[0];
 	ctx->d_frame.fmt = &fimc_formats[0];
-	/* Setup the device context for mem2mem mode. */
+	/* Setup the device context for memory-to-memory mode */
 	ctx->state = FIMC_CTX_M2M;
 	ctx->flags = 0;
 	ctx->in_path = FIMC_DMA;
@@ -1437,26 +1446,33 @@ static int fimc_m2m_open(struct file *file)
 
 	ctx->m2m_ctx = v4l2_m2m_ctx_init(fimc->m2m.m2m_dev, ctx, queue_init);
 	if (IS_ERR(ctx->m2m_ctx)) {
-		int err = PTR_ERR(ctx->m2m_ctx);
-		kfree(ctx);
-		return err;
+		ret = PTR_ERR(ctx->m2m_ctx);
+		goto error_fh;
 	}
 
 	if (fimc->m2m.refcnt++ == 0)
 		set_bit(ST_M2M_RUN, &fimc->state);
-
 	return 0;
+
+error_fh:
+	v4l2_fh_del(&ctx->fh);
+	v4l2_fh_exit(&ctx->fh);
+error:
+	kfree(ctx);
+	return ret;
 }
 
 static int fimc_m2m_release(struct file *file)
 {
-	struct fimc_ctx *ctx = file->private_data;
+	struct fimc_ctx *ctx = fh_to_ctx(file->private_data);
 	struct fimc_dev *fimc = ctx->fimc_dev;
 
 	dbg("pid: %d, state: 0x%lx, refcnt= %d",
 		task_pid_nr(current), fimc->state, fimc->m2m.refcnt);
 
 	v4l2_m2m_ctx_release(ctx->m2m_ctx);
+	v4l2_fh_del(&ctx->fh);
+	v4l2_fh_exit(&ctx->fh);
 
 	if (--fimc->m2m.refcnt <= 0)
 		clear_bit(ST_M2M_RUN, &fimc->state);
@@ -1465,9 +1481,9 @@ static int fimc_m2m_release(struct file *file)
 }
 
 static unsigned int fimc_m2m_poll(struct file *file,
-				     struct poll_table_struct *wait)
+				  struct poll_table_struct *wait)
 {
-	struct fimc_ctx *ctx = file->private_data;
+	struct fimc_ctx *ctx = fh_to_ctx(file->private_data);
 
 	return v4l2_m2m_poll(file, ctx->m2m_ctx, wait);
 }
@@ -1475,7 +1491,7 @@ static unsigned int fimc_m2m_poll(struct file *file,
 
 static int fimc_m2m_mmap(struct file *file, struct vm_area_struct *vma)
 {
-	struct fimc_ctx *ctx = file->private_data;
+	struct fimc_ctx *ctx = fh_to_ctx(file->private_data);
 
 	return v4l2_m2m_mmap(file, ctx->m2m_ctx, vma);
 }
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 25a8005..2591066 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -460,6 +460,7 @@ struct fimc_dev {
  * @state:		flags to keep track of user configuration
  * @fimc_dev:		the FIMC device this context applies to
  * @m2m_ctx:		memory-to-memory device context
+ * @fh:			v4l2 file handle
  */
 struct fimc_ctx {
 	spinlock_t		slock;
@@ -479,8 +480,11 @@ struct fimc_ctx {
 	u32			state;
 	struct fimc_dev		*fimc_dev;
 	struct v4l2_m2m_ctx	*m2m_ctx;
+	struct v4l2_fh		fh;
 };
 
+#define fh_to_ctx(__fh) container_of(__fh, struct fimc_ctx, fh)
+
 static inline bool fimc_capture_active(struct fimc_dev *fimc)
 {
 	unsigned long flags;
@@ -632,18 +636,16 @@ int fimc_hw_set_camera_type(struct fimc_dev *fimc,
 /* fimc-core.c */
 int fimc_vidioc_enum_fmt_mplane(struct file *file, void *priv,
 				struct v4l2_fmtdesc *f);
-int fimc_vidioc_g_fmt_mplane(struct file *file, void *priv,
-			     struct v4l2_format *f);
-int fimc_vidioc_try_fmt_mplane(struct file *file, void *priv,
-			       struct v4l2_format *f);
 int fimc_vidioc_queryctrl(struct file *file, void *priv,
 			  struct v4l2_queryctrl *qc);
 int fimc_vidioc_g_ctrl(struct file *file, void *priv,
 		       struct v4l2_control *ctrl);
 
+int fimc_try_fmt_mplane(struct fimc_ctx *ctx, struct v4l2_format *f);
 int fimc_try_crop(struct fimc_ctx *ctx, struct v4l2_crop *cr);
 int check_ctrl_val(struct fimc_ctx *ctx,  struct v4l2_control *ctrl);
 int fimc_s_ctrl(struct fimc_ctx *ctx, struct v4l2_control *ctrl);
+int fimc_fill_format(struct fimc_frame *frame, struct v4l2_format *f);
 
 struct fimc_fmt *find_format(struct v4l2_format *f, unsigned int mask);
 struct fimc_fmt *find_mbus_format(struct v4l2_mbus_framefmt *f,
-- 
1.7.5.4

[PATCH v3 10/19] s5p-fimc: Conversion to the control framework

[Sylwester Nawrocki]

The FIMC entity supports rotation, horizontal and vertical flip
in camera capture and memory-to-memory operation mode.
Due to atomic contexts used in mem-to-mem driver the control
values need to be cached in drivers internal data structure.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |   60 +++---
 drivers/media/video/s5p-fimc/fimc-core.c    |  291 +++++++++++----------------
 drivers/media/video/s5p-fimc/fimc-core.h    |   34 ++--
 drivers/media/video/s5p-fimc/fimc-mdevice.c |   11 +-
 drivers/media/video/s5p-fimc/fimc-reg.c     |   32 +---
 5 files changed, 186 insertions(+), 242 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index 0ceb16d..b193b7e 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -27,6 +27,7 @@
 #include <media/videobuf2-core.h>
 #include <media/videobuf2-dma-contig.h>
 
+#include "fimc-mdevice.h"
 #include "fimc-core.h"
 
 static int fimc_stop_capture(struct fimc_dev *fimc)
@@ -257,6 +258,29 @@ static struct vb2_ops fimc_capture_qops = {
 	.stop_streaming		= stop_streaming,
 };
 
+/**
+ * fimc_capture_ctrls_create - initialize the control handler
+ *
+ * Initialize the capture video node control handler and populate it
+ * with FIMC specific controls. If the linked sensor subdevice does
+ * not expose a video node add its controls to the FIMC control
+ * handler. This function must be called with the graph mutex held.
+ */
+int fimc_capture_ctrls_create(struct fimc_dev *fimc)
+{
+	int ret;
+
+	if (WARN_ON(fimc->vid_cap.ctx == NULL))
+		return -ENXIO;
+	if (fimc->vid_cap.ctx->ctrls_rdy)
+		return 0;
+	ret = fimc_ctrls_create(fimc->vid_cap.ctx);
+	if (ret || subdev_has_devnode(fimc->pipeline.sensor))
+		return ret;
+	return v4l2_ctrl_add_handler(&fimc->vid_cap.ctx->ctrl_handler,
+				    fimc->pipeline.sensor->ctrl_handler);
+}
+
 static int fimc_capture_open(struct file *file)
 {
 	struct fimc_dev *fimc = video_drvdata(file);
@@ -277,9 +301,10 @@ static int fimc_capture_open(struct file *file)
 		return ret;
 	}
 
-	++fimc->vid_cap.refcnt;
+	if (++fimc->vid_cap.refcnt == 1)
+		ret = fimc_capture_ctrls_create(fimc);
 
-	return 0;
+	return ret;
 }
 
 static int fimc_capture_close(struct file *file)
@@ -290,11 +315,11 @@ static int fimc_capture_close(struct file *file)
 
 	if (--fimc->vid_cap.refcnt == 0) {
 		fimc_stop_capture(fimc);
+		fimc_ctrls_delete(fimc->vid_cap.ctx);
 		vb2_queue_release(&fimc->vid_cap.vbq);
 	}
 
 	pm_runtime_put_sync(&fimc->pdev->dev);
-
 	return v4l2_fh_release(file);
 }
 
@@ -530,30 +555,6 @@ static int fimc_cap_dqbuf(struct file *file, void *priv,
 	return vb2_dqbuf(&fimc->vid_cap.vbq, buf, file->f_flags & O_NONBLOCK);
 }
 
-static int fimc_cap_s_ctrl(struct file *file, void *priv,
-			   struct v4l2_control *ctrl)
-{
-	struct fimc_dev *fimc = video_drvdata(file);
-	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
-	int ret = -EINVAL;
-
-	/* Allow any controls but 90/270 rotation while streaming */
-	if (!fimc_capture_active(ctx->fimc_dev) ||
-	    ctrl->id != V4L2_CID_ROTATE ||
-	    (ctrl->value != 90 && ctrl->value != 270)) {
-		ret = check_ctrl_val(ctx, ctrl);
-		if (!ret) {
-			ret = fimc_s_ctrl(ctx, ctrl);
-			if (!ret)
-				ctx->state |= FIMC_PARAMS;
-		}
-	}
-	if (ret == -EINVAL)
-		ret = v4l2_subdev_call(ctx->fimc_dev->vid_cap.sd,
-				       core, s_ctrl, ctrl);
-	return ret;
-}
-
 static int fimc_cap_cropcap(struct file *file, void *fh,
 			    struct v4l2_cropcap *cr)
 {
@@ -640,10 +641,6 @@ static const struct v4l2_ioctl_ops fimc_capture_ioctl_ops = {
 	.vidioc_streamon		= fimc_cap_streamon,
 	.vidioc_streamoff		= fimc_cap_streamoff,
 
-	.vidioc_queryctrl		= fimc_vidioc_queryctrl,
-	.vidioc_g_ctrl			= fimc_vidioc_g_ctrl,
-	.vidioc_s_ctrl			= fimc_cap_s_ctrl,
-
 	.vidioc_g_crop			= fimc_cap_g_crop,
 	.vidioc_s_crop			= fimc_cap_s_crop,
 	.vidioc_cropcap			= fimc_cap_cropcap,
@@ -727,6 +724,7 @@ int fimc_register_capture_device(struct fimc_dev *fimc,
 	if (ret)
 		goto err_ent;
 
+	vfd->ctrl_handler = &ctx->ctrl_handler;
 	ret = video_register_device(vfd, VFL_TYPE_GRABBER, -1);
 	if (ret) {
 		v4l2_err(v4l2_dev, "Failed to register video device\n");
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index 1f84cc6..cb89146 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -162,43 +162,6 @@ static struct fimc_fmt fimc_formats[] = {
 	},
 };
 
-static struct v4l2_queryctrl fimc_ctrls[] = {
-	{
-		.id		= V4L2_CID_HFLIP,
-		.type		= V4L2_CTRL_TYPE_BOOLEAN,
-		.name		= "Horizontal flip",
-		.minimum	= 0,
-		.maximum	= 1,
-		.default_value	= 0,
-	}, {
-		.id		= V4L2_CID_VFLIP,
-		.type		= V4L2_CTRL_TYPE_BOOLEAN,
-		.name		= "Vertical flip",
-		.minimum	= 0,
-		.maximum	= 1,
-		.default_value	= 0,
-	}, {
-		.id		= V4L2_CID_ROTATE,
-		.type		= V4L2_CTRL_TYPE_INTEGER,
-		.name		= "Rotation (CCW)",
-		.minimum	= 0,
-		.maximum	= 270,
-		.step		= 90,
-		.default_value	= 0,
-	},
-};
-
-
-static struct v4l2_queryctrl *get_ctrl(int id)
-{
-	int i;
-
-	for (i = 0; i < ARRAY_SIZE(fimc_ctrls); ++i)
-		if (id == fimc_ctrls[i].id)
-			return &fimc_ctrls[i];
-	return NULL;
-}
-
 int fimc_check_scaler_ratio(int sw, int sh, int dw, int dh, int rot)
 {
 	int tx, ty;
@@ -777,6 +740,116 @@ static struct vb2_ops fimc_qops = {
 	.start_streaming = start_streaming,
 };
 
+/*
+ * V4L2 controls handling
+ */
+#define ctrl_to_ctx(__ctrl) \
+	container_of((__ctrl)->handler, struct fimc_ctx, ctrl_handler)
+
+static int fimc_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct fimc_ctx *ctx = ctrl_to_ctx(ctrl);
+	struct fimc_dev *fimc = ctx->fimc_dev;
+	struct samsung_fimc_variant *variant = fimc->variant;
+	unsigned long flags;
+	int ret = 0;
+
+	if (ctrl->flags & V4L2_CTRL_FLAG_INACTIVE)
+		return 0;
+
+	switch (ctrl->id) {
+	case V4L2_CID_HFLIP:
+		spin_lock_irqsave(&ctx->slock, flags);
+		ctx->hflip = ctrl->val;
+		break;
+
+	case V4L2_CID_VFLIP:
+		spin_lock_irqsave(&ctx->slock, flags);
+		ctx->vflip = ctrl->val;
+		break;
+
+	case V4L2_CID_ROTATE:
+		if (fimc_capture_pending(fimc) ||
+		    fimc_ctx_state_is_set(FIMC_DST_FMT | FIMC_SRC_FMT, ctx)) {
+			ret = fimc_check_scaler_ratio(ctx->s_frame.width,
+					ctx->s_frame.height, ctx->d_frame.width,
+					ctx->d_frame.height, ctrl->val);
+		}
+		if (ret) {
+			v4l2_err(fimc->m2m.vfd, "Out of scaler range\n");
+			return -EINVAL;
+		}
+		if ((ctrl->val == 90 || ctrl->val == 270) &&
+		    !variant->has_out_rot)
+			return -EINVAL;
+		spin_lock_irqsave(&ctx->slock, flags);
+		ctx->rotation = ctrl->val;
+		break;
+
+	default:
+		v4l2_err(fimc->v4l2_dev, "Invalid control: 0x%X\n", ctrl->id);
+		return -EINVAL;
+	}
+	ctx->state |= FIMC_PARAMS;
+	set_bit(ST_CAPT_APPLY_CFG, &fimc->state);
+	spin_unlock_irqrestore(&ctx->slock, flags);
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops fimc_ctrl_ops = {
+	.s_ctrl = fimc_s_ctrl,
+};
+
+int fimc_ctrls_create(struct fimc_ctx *ctx)
+{
+	if (ctx->ctrls_rdy)
+		return 0;
+	v4l2_ctrl_handler_init(&ctx->ctrl_handler, 3);
+
+	ctx->ctrl_rotate = v4l2_ctrl_new_std(&ctx->ctrl_handler, &fimc_ctrl_ops,
+				     V4L2_CID_HFLIP, 0, 1, 1, 0);
+	ctx->ctrl_hflip = v4l2_ctrl_new_std(&ctx->ctrl_handler, &fimc_ctrl_ops,
+				    V4L2_CID_VFLIP, 0, 1, 1, 0);
+	ctx->ctrl_vflip = v4l2_ctrl_new_std(&ctx->ctrl_handler, &fimc_ctrl_ops,
+				    V4L2_CID_ROTATE, 0, 270, 90, 0);
+	ctx->ctrls_rdy = ctx->ctrl_handler.error == 0;
+
+	return ctx->ctrl_handler.error;
+}
+
+void fimc_ctrls_delete(struct fimc_ctx *ctx)
+{
+	if (ctx->ctrls_rdy) {
+		v4l2_ctrl_handler_free(&ctx->ctrl_handler);
+		ctx->ctrls_rdy = false;
+	}
+}
+
+void fimc_ctrls_activate(struct fimc_ctx *ctx, bool active)
+{
+	if (!ctx->ctrls_rdy)
+		return;
+
+	mutex_lock(&ctx->ctrl_handler.lock);
+	v4l2_ctrl_activate(ctx->ctrl_rotate, active);
+	v4l2_ctrl_activate(ctx->ctrl_hflip, active);
+	v4l2_ctrl_activate(ctx->ctrl_vflip, active);
+
+	if (active) {
+		ctx->rotation = ctx->ctrl_rotate->val;
+		ctx->hflip    = ctx->ctrl_hflip->val;
+		ctx->vflip    = ctx->ctrl_vflip->val;
+	} else {
+		ctx->rotation = 0;
+		ctx->hflip    = 0;
+		ctx->vflip    = 0;
+	}
+	mutex_unlock(&ctx->ctrl_handler.lock);
+}
+
+/*
+ * V4L2 ioctl handlers
+ */
 static int fimc_m2m_querycap(struct file *file, void *fh,
 			     struct v4l2_capability *cap)
 {
@@ -1073,136 +1146,6 @@ static int fimc_m2m_streamoff(struct file *file, void *fh,
 	return v4l2_m2m_streamoff(file, ctx->m2m_ctx, type);
 }
 
-int fimc_vidioc_queryctrl(struct file *file, void *fh,
-			  struct v4l2_queryctrl *qc)
-{
-	struct fimc_ctx *ctx = fh_to_ctx(fh);
-	struct fimc_dev *fimc = ctx->fimc_dev;
-	struct v4l2_queryctrl *c;
-	int ret = -EINVAL;
-
-	c = get_ctrl(qc->id);
-	if (c) {
-		*qc = *c;
-		return 0;
-	}
-
-	if (fimc_ctx_state_is_set(FIMC_CTX_CAP, ctx)) {
-		return v4l2_subdev_call(ctx->fimc_dev->vid_cap.sd,
-					core, queryctrl, qc);
-	}
-	return ret;
-}
-
-int fimc_vidioc_g_ctrl(struct file *file, void *fh, struct v4l2_control *ctrl)
-{
-	struct fimc_ctx *ctx = fh_to_ctx(fh);
-	struct fimc_dev *fimc = ctx->fimc_dev;
-
-	switch (ctrl->id) {
-	case V4L2_CID_HFLIP:
-		ctrl->value = (FLIP_X_AXIS & ctx->flip) ? 1 : 0;
-		break;
-	case V4L2_CID_VFLIP:
-		ctrl->value = (FLIP_Y_AXIS & ctx->flip) ? 1 : 0;
-		break;
-	case V4L2_CID_ROTATE:
-		ctrl->value = ctx->rotation;
-		break;
-	default:
-		if (fimc_ctx_state_is_set(FIMC_CTX_CAP, ctx)) {
-			return v4l2_subdev_call(fimc->vid_cap.sd, core,
-						g_ctrl, ctrl);
-		} else {
-			v4l2_err(fimc->m2m.vfd, "Invalid control\n");
-			return -EINVAL;
-		}
-	}
-	dbg("ctrl->value= %d", ctrl->value);
-
-	return 0;
-}
-
-int check_ctrl_val(struct fimc_ctx *ctx,  struct v4l2_control *ctrl)
-{
-	struct v4l2_queryctrl *c;
-	c = get_ctrl(ctrl->id);
-	if (!c)
-		return -EINVAL;
-
-	if (ctrl->value < c->minimum || ctrl->value > c->maximum
-		|| (c->step != 0 && ctrl->value % c->step != 0)) {
-		v4l2_err(ctx->fimc_dev->m2m.vfd, "Invalid control value\n");
-		return -ERANGE;
-	}
-
-	return 0;
-}
-
-int fimc_s_ctrl(struct fimc_ctx *ctx, struct v4l2_control *ctrl)
-{
-	struct samsung_fimc_variant *variant = ctx->fimc_dev->variant;
-	struct fimc_dev *fimc = ctx->fimc_dev;
-	int ret = 0;
-
-	switch (ctrl->id) {
-	case V4L2_CID_HFLIP:
-		if (ctrl->value)
-			ctx->flip |= FLIP_X_AXIS;
-		else
-			ctx->flip &= ~FLIP_X_AXIS;
-		break;
-
-	case V4L2_CID_VFLIP:
-		if (ctrl->value)
-			ctx->flip |= FLIP_Y_AXIS;
-		else
-			ctx->flip &= ~FLIP_Y_AXIS;
-		break;
-
-	case V4L2_CID_ROTATE:
-		if (fimc_ctx_state_is_set(FIMC_DST_FMT | FIMC_SRC_FMT, ctx)) {
-			ret = fimc_check_scaler_ratio(ctx->s_frame.width,
-					ctx->s_frame.height, ctx->d_frame.width,
-					ctx->d_frame.height, ctrl->value);
-		}
-
-		if (ret) {
-			v4l2_err(fimc->m2m.vfd, "Out of scaler range\n");
-			return -EINVAL;
-		}
-
-		/* Check for the output rotator availability */
-		if ((ctrl->value == 90 || ctrl->value == 270) &&
-		    (ctx->in_path == FIMC_DMA && !variant->has_out_rot))
-			return -EINVAL;
-		ctx->rotation = ctrl->value;
-		break;
-
-	default:
-		v4l2_err(fimc->v4l2_dev, "Invalid control\n");
-		return -EINVAL;
-	}
-
-	fimc_ctx_state_lock_set(FIMC_PARAMS, ctx);
-
-	return 0;
-}
-
-static int fimc_m2m_s_ctrl(struct file *file, void *fh,
-			   struct v4l2_control *ctrl)
-{
-	struct fimc_ctx *ctx = fh_to_ctx(fh);
-	int ret = 0;
-
-	ret = check_ctrl_val(ctx, ctrl);
-	if (ret)
-		return ret;
-
-	ret = fimc_s_ctrl(ctx, ctrl);
-	return 0;
-}
-
 static int fimc_m2m_cropcap(struct file *file, void *fh,
 			    struct v4l2_cropcap *cr)
 {
@@ -1368,10 +1311,6 @@ static const struct v4l2_ioctl_ops fimc_m2m_ioctl_ops = {
 	.vidioc_streamon		= fimc_m2m_streamon,
 	.vidioc_streamoff		= fimc_m2m_streamoff,
 
-	.vidioc_queryctrl		= fimc_vidioc_queryctrl,
-	.vidioc_g_ctrl			= fimc_vidioc_g_ctrl,
-	.vidioc_s_ctrl			= fimc_m2m_s_ctrl,
-
 	.vidioc_g_crop			= fimc_m2m_g_crop,
 	.vidioc_s_crop			= fimc_m2m_s_crop,
 	.vidioc_cropcap			= fimc_m2m_cropcap
@@ -1429,7 +1368,12 @@ static int fimc_m2m_open(struct file *file)
 	ret = v4l2_fh_init(&ctx->fh, fimc->m2m.vfd);
 	if (ret)
 		goto error;
+	ret = fimc_ctrls_create(ctx);
+	if (ret)
+		goto error_fh;
 
+	/* Use separate control handler per file handle */
+	ctx->fh.ctrl_handler = &ctx->ctrl_handler;
 	file->private_data = &ctx->fh;
 	v4l2_fh_add(&ctx->fh);
 
@@ -1447,13 +1391,15 @@ static int fimc_m2m_open(struct file *file)
 	ctx->m2m_ctx = v4l2_m2m_ctx_init(fimc->m2m.m2m_dev, ctx, queue_init);
 	if (IS_ERR(ctx->m2m_ctx)) {
 		ret = PTR_ERR(ctx->m2m_ctx);
-		goto error_fh;
+		goto error_c;
 	}
 
 	if (fimc->m2m.refcnt++ == 0)
 		set_bit(ST_M2M_RUN, &fimc->state);
 	return 0;
 
+error_c:
+	fimc_ctrls_delete(ctx);
 error_fh:
 	v4l2_fh_del(&ctx->fh);
 	v4l2_fh_exit(&ctx->fh);
@@ -1471,6 +1417,7 @@ static int fimc_m2m_release(struct file *file)
 		task_pid_nr(current), fimc->state, fimc->m2m.refcnt);
 
 	v4l2_m2m_ctx_release(ctx->m2m_ctx);
+	fimc_ctrls_delete(ctx);
 	v4l2_fh_del(&ctx->fh);
 	v4l2_fh_exit(&ctx->fh);
 
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 2591066..3606369 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -20,6 +20,7 @@
 
 #include <media/media-entity.h>
 #include <media/videobuf2-core.h>
+#include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
 #include <media/v4l2-mem2mem.h>
 #include <media/v4l2-mediabus.h>
@@ -62,6 +63,7 @@ enum fimc_dev_flags {
 	ST_CAPT_STREAM,
 	ST_CAPT_SHUT,
 	ST_CAPT_INUSE,
+	ST_CAPT_APPLY_CFG,
 };
 
 #define fimc_m2m_active(dev) test_bit(ST_M2M_RUN, &(dev)->state)
@@ -128,11 +130,6 @@ enum fimc_color_fmt {
 /* Y (16 ~ 235), Cb/Cr (16 ~ 240) */
 #define	FIMC_COLOR_RANGE_NARROW		(1 << 3)
 
-#define	FLIP_NONE			0
-#define	FLIP_X_AXIS			1
-#define	FLIP_Y_AXIS			2
-#define	FLIP_XY_AXIS			(FLIP_X_AXIS | FLIP_Y_AXIS)
-
 /**
  * struct fimc_fmt - the driver's internal color format data
  * @mbus_code: Media Bus pixel code, -1 if not applicable
@@ -455,12 +452,18 @@ struct fimc_dev {
  * @scaler:		image scaler properties
  * @effect:		image effect
  * @rotation:		image clockwise rotation in degrees
- * @flip:		image flip mode
+ * @hflip:		indicates image horizontal flip if set
+ * @vflip:		indicates image vertical flip if set
  * @flags:		additional flags for image conversion
  * @state:		flags to keep track of user configuration
  * @fimc_dev:		the FIMC device this context applies to
  * @m2m_ctx:		memory-to-memory device context
  * @fh:			v4l2 file handle
+ * @ctrl_handler:	v4l2 controls handler
+ * @ctrl_rotate		image rotation control
+ * @ctrl_hflip		horizontal flip control
+ * @ctrl_vflip		vartical flip control
+ * @ctrls_rdy:		true if the control handler is initialized
  */
 struct fimc_ctx {
 	spinlock_t		slock;
@@ -475,12 +478,18 @@ struct fimc_ctx {
 	struct fimc_scaler	scaler;
 	struct fimc_effect	effect;
 	int			rotation;
-	u32			flip;
+	unsigned int		hflip:1;
+	unsigned int		vflip:1;
 	u32			flags;
 	u32			state;
 	struct fimc_dev		*fimc_dev;
 	struct v4l2_m2m_ctx	*m2m_ctx;
 	struct v4l2_fh		fh;
+	struct v4l2_ctrl_handler ctrl_handler;
+	struct v4l2_ctrl	*ctrl_rotate;
+	struct v4l2_ctrl	*ctrl_hflip;
+	struct v4l2_ctrl	*ctrl_vflip;
+	bool			ctrls_rdy;
 };
 
 #define fh_to_ctx(__fh) container_of(__fh, struct fimc_ctx, fh)
@@ -636,15 +645,11 @@ int fimc_hw_set_camera_type(struct fimc_dev *fimc,
 /* fimc-core.c */
 int fimc_vidioc_enum_fmt_mplane(struct file *file, void *priv,
 				struct v4l2_fmtdesc *f);
-int fimc_vidioc_queryctrl(struct file *file, void *priv,
-			  struct v4l2_queryctrl *qc);
-int fimc_vidioc_g_ctrl(struct file *file, void *priv,
-		       struct v4l2_control *ctrl);
-
 int fimc_try_fmt_mplane(struct fimc_ctx *ctx, struct v4l2_format *f);
 int fimc_try_crop(struct fimc_ctx *ctx, struct v4l2_crop *cr);
-int check_ctrl_val(struct fimc_ctx *ctx,  struct v4l2_control *ctrl);
-int fimc_s_ctrl(struct fimc_ctx *ctx, struct v4l2_control *ctrl);
+int fimc_ctrls_create(struct fimc_ctx *ctx);
+void fimc_ctrls_delete(struct fimc_ctx *ctx);
+void fimc_ctrls_activate(struct fimc_ctx *ctx, bool active);
 int fimc_fill_format(struct fimc_frame *frame, struct v4l2_format *f);
 
 struct fimc_fmt *find_format(struct v4l2_format *f, unsigned int mask);
@@ -667,6 +672,7 @@ void fimc_unregister_driver(void);
 int fimc_register_capture_device(struct fimc_dev *fimc,
 				 struct v4l2_device *v4l2_dev);
 void fimc_unregister_capture_device(struct fimc_dev *fimc);
+int fimc_capture_ctrls_create(struct fimc_dev *fimc);
 int fimc_vid_cap_buf_queue(struct fimc_dev *fimc,
 			     struct fimc_vid_buffer *fimc_vb);
 int fimc_capture_suspend(struct fimc_dev *fimc);
diff --git a/drivers/media/video/s5p-fimc/fimc-mdevice.c b/drivers/media/video/s5p-fimc/fimc-mdevice.c
index 10c8d5d..54e889b 100644
--- a/drivers/media/video/s5p-fimc/fimc-mdevice.c
+++ b/drivers/media/video/s5p-fimc/fimc-mdevice.c
@@ -22,6 +22,7 @@
 #include <linux/types.h>
 #include <linux/slab.h>
 #include <linux/version.h>
+#include <media/v4l2-ctrls.h>
 #include <media/media-device.h>
 
 #include "fimc-core.h"
@@ -626,15 +627,23 @@ static int fimc_md_link_notify(struct media_pad *source,
 		ret = __fimc_pipeline_shutdown(fimc);
 		fimc->pipeline.sensor = NULL;
 		fimc->pipeline.csis = NULL;
+
+		mutex_lock(&fimc->lock);
+		fimc_ctrls_delete(fimc->vid_cap.ctx);
+		mutex_unlock(&fimc->lock);
 		return ret;
 	}
 	/*
 	 * Link activation. Enable power of pipeline elements only if the
 	 * pipeline is already in use, i.e. its video node is opened.
+	 * Recreate the controls destroyed during the link deactivation.
 	 */
 	mutex_lock(&fimc->lock);
-	if (fimc->vid_cap.refcnt > 0)
+	if (fimc->vid_cap.refcnt > 0) {
 		ret = __fimc_pipeline_initialize(fimc, source->entity, true);
+		if (!ret)
+			ret = fimc_capture_ctrls_create(fimc);
+	}
 	mutex_unlock(&fimc->lock);
 
 	return ret ? -EPIPE : ret;
diff --git a/drivers/media/video/s5p-fimc/fimc-reg.c b/drivers/media/video/s5p-fimc/fimc-reg.c
index c688263..50937b4 100644
--- a/drivers/media/video/s5p-fimc/fimc-reg.c
+++ b/drivers/media/video/s5p-fimc/fimc-reg.c
@@ -41,19 +41,11 @@ static u32 fimc_hw_get_in_flip(struct fimc_ctx *ctx)
 {
 	u32 flip = S5P_MSCTRL_FLIP_NORMAL;
 
-	switch (ctx->flip) {
-	case FLIP_X_AXIS:
+	if (ctx->hflip)
 		flip = S5P_MSCTRL_FLIP_X_MIRROR;
-		break;
-	case FLIP_Y_AXIS:
+	if (ctx->vflip)
 		flip = S5P_MSCTRL_FLIP_Y_MIRROR;
-		break;
-	case FLIP_XY_AXIS:
-		flip = S5P_MSCTRL_FLIP_180;
-		break;
-	default:
-		break;
-	}
+
 	if (ctx->rotation <= 90)
 		return flip;
 
@@ -64,19 +56,11 @@ static u32 fimc_hw_get_target_flip(struct fimc_ctx *ctx)
 {
 	u32 flip = S5P_CITRGFMT_FLIP_NORMAL;
 
-	switch (ctx->flip) {
-	case FLIP_X_AXIS:
-		flip = S5P_CITRGFMT_FLIP_X_MIRROR;
-		break;
-	case FLIP_Y_AXIS:
-		flip = S5P_CITRGFMT_FLIP_Y_MIRROR;
-		break;
-	case FLIP_XY_AXIS:
-		flip = S5P_CITRGFMT_FLIP_180;
-		break;
-	default:
-		break;
-	}
+	if (ctx->hflip)
+		flip |= S5P_CITRGFMT_FLIP_X_MIRROR;
+	if (ctx->vflip)
+		flip |= S5P_CITRGFMT_FLIP_Y_MIRROR;
+
 	if (ctx->rotation <= 90)
 		return flip;
 
-- 
1.7.5.4

[PATCH v3 11/19] s5p-fimc: Add media operations in the capture entity driver

[Sylwester Nawrocki]

Add the link_setup handler for the camera capture video node.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |   32 +++++++++++++++++++++++++++
 drivers/media/video/s5p-fimc/fimc-core.h    |    2 +
 2 files changed, 34 insertions(+), 0 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index b193b7e..7c6c420 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -650,6 +650,37 @@ static const struct v4l2_ioctl_ops fimc_capture_ioctl_ops = {
 	.vidioc_g_input			= fimc_cap_g_input,
 };
 
+/* Media operations */
+static int fimc_link_setup(struct media_entity *entity,
+			   const struct media_pad *local,
+			   const struct media_pad *remote, u32 flags)
+{
+	struct video_device *vd = media_entity_to_video_device(entity);
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(remote->entity);
+	struct fimc_dev *fimc = video_get_drvdata(vd);
+
+	if (WARN_ON(fimc == NULL))
+		return 0;
+
+	dbg("%s --> %s, flags: 0x%x. input: 0x%x",
+	    local->entity->name, remote->entity->name, flags,
+	    fimc->vid_cap.input);
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (fimc->vid_cap.input != 0)
+			return -EBUSY;
+		fimc->vid_cap.input = sd->grp_id;
+		return 0;
+	}
+
+	fimc->vid_cap.input = 0;
+	return 0;
+}
+
+static const struct media_entity_operations fimc_media_ops = {
+	.link_setup = fimc_link_setup,
+};
+
 /* fimc->lock must be already initialized */
 int fimc_register_capture_device(struct fimc_dev *fimc,
 				 struct v4l2_device *v4l2_dev)
@@ -724,6 +755,7 @@ int fimc_register_capture_device(struct fimc_dev *fimc,
 	if (ret)
 		goto err_ent;
 
+	vfd->entity.ops = &fimc_media_ops;
 	vfd->ctrl_handler = &ctx->ctrl_handler;
 	ret = video_register_device(vfd, VFL_TYPE_GRABBER, -1);
 	if (ret) {
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 3606369..86288c8 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -305,6 +305,7 @@ struct fimc_m2m_device {
  * @reqbufs_count: the number of buffers requested in REQBUFS ioctl
  * @input_index: input (camera sensor) index
  * @refcnt: driver's private reference counter
+ * @input: capture input type, grp_id of attached subdev
  * @vid_dev_compat: true to enable full pipeline setup in the driver
  */
 struct fimc_vid_cap {
@@ -323,6 +324,7 @@ struct fimc_vid_cap {
 	unsigned int			reqbufs_count;
 	int				input_index;
 	int				refcnt;
+	u32				input;
 	bool				vid_dev_compat;
 };
 
-- 
1.7.5.4

[PATCH v3 12/19] s5p-fimc: Add PM helper function for streaming control

[Sylwester Nawrocki]

Create a helper function for (re)starting streaming in PM resume calls.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |   80 ++++++++++++++++-----------
 drivers/media/video/s5p-fimc/fimc-core.c    |    4 +-
 drivers/media/video/s5p-fimc/fimc-core.h    |    3 +
 3 files changed, 52 insertions(+), 35 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index 7c6c420..7af4ac5 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -30,6 +30,48 @@
 #include "fimc-mdevice.h"
 #include "fimc-core.h"
 
+/**
+ * fimc_start_capture - initialize the H/W for camera capture operation
+ *
+ * Initialize the camera capture datapath so when this function successfuly
+ * completes all what is needed to start the capture pipeline is asserting
+ * the global capture bit ImgCptEn.
+ */
+static int fimc_start_capture(struct fimc_dev *fimc)
+{
+	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
+	struct fimc_sensor_info *sensor;
+	unsigned long flags;
+	int ret = 0;
+
+	if (fimc->pipeline.sensor == NULL || ctx == NULL)
+		return -EIO;
+	sensor = v4l2_get_subdev_hostdata(fimc->pipeline.sensor);
+
+	spin_lock_irqsave(&fimc->slock, flags);
+	fimc_prepare_dma_offset(ctx, &ctx->d_frame);
+	fimc_set_yuv_order(ctx);
+
+	fimc_hw_set_camera_polarity(fimc, sensor->pdata);
+	fimc_hw_set_camera_type(fimc, sensor->pdata);
+	fimc_hw_set_camera_source(fimc, sensor->pdata);
+	fimc_hw_set_camera_offset(fimc, &ctx->s_frame);
+
+	ret = fimc_set_scaler_info(ctx);
+	if (!ret) {
+		fimc_hw_set_input_path(ctx);
+		fimc_hw_set_prescaler(ctx);
+		fimc_hw_set_mainscaler(ctx);
+		fimc_hw_set_target_format(ctx);
+		fimc_hw_set_rotation(ctx);
+		fimc_hw_set_effect(ctx);
+		fimc_hw_set_output_path(ctx);
+		fimc_hw_set_out_dma(ctx);
+	}
+	spin_unlock_irqrestore(&fimc->slock, flags);
+	return ret;
+}
+
 static int fimc_stop_capture(struct fimc_dev *fimc)
 {
 	unsigned long flags;
@@ -89,47 +131,19 @@ static int start_streaming(struct vb2_queue *q)
 {
 	struct fimc_ctx *ctx = q->drv_priv;
 	struct fimc_dev *fimc = ctx->fimc_dev;
-	struct s5p_fimc_isp_info *isp_info;
+	struct fimc_vid_cap *vid_cap = &fimc->vid_cap;
 	int ret;
 
 	fimc_hw_reset(fimc);
 
-	ret = v4l2_subdev_call(fimc->vid_cap.sd, video, s_stream, 1);
-	if (ret && ret != -ENOIOCTLCMD)
-		return ret;
+	vid_cap->active_buf_cnt = 0;
+	vid_cap->frame_count = 0;
+	vid_cap->buf_index = 0;
 
-	ret = fimc_prepare_config(ctx, ctx->state);
+	ret = fimc_start_capture(fimc);
 	if (ret)
 		return ret;
 
-	isp_info = &fimc->pdata->isp_info[fimc->vid_cap.input_index];
-	fimc_hw_set_camera_type(fimc, isp_info);
-	fimc_hw_set_camera_source(fimc, isp_info);
-	fimc_hw_set_camera_offset(fimc, &ctx->s_frame);
-
-	if (ctx->state & FIMC_PARAMS) {
-		ret = fimc_set_scaler_info(ctx);
-		if (ret) {
-			err("Scaler setup error");
-			return ret;
-		}
-		fimc_hw_set_input_path(ctx);
-		fimc_hw_set_prescaler(ctx);
-		fimc_hw_set_mainscaler(ctx);
-		fimc_hw_set_target_format(ctx);
-		fimc_hw_set_rotation(ctx);
-		fimc_hw_set_effect(ctx);
-	}
-
-	fimc_hw_set_output_path(ctx);
-	fimc_hw_set_out_dma(ctx);
-
-	INIT_LIST_HEAD(&fimc->vid_cap.pending_buf_q);
-	INIT_LIST_HEAD(&fimc->vid_cap.active_buf_q);
-	fimc->vid_cap.active_buf_cnt = 0;
-	fimc->vid_cap.frame_count = 0;
-	fimc->vid_cap.buf_index = 0;
-
 	set_bit(ST_CAPT_PEND, &fimc->state);
 	return 0;
 }
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index cb89146..a3edaf8 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -476,7 +476,7 @@ int fimc_prepare_addr(struct fimc_ctx *ctx, struct vb2_buffer *vb,
 }
 
 /* Set order for 1 and 2 plane YCBCR 4:2:2 formats. */
-static void fimc_set_yuv_order(struct fimc_ctx *ctx)
+void fimc_set_yuv_order(struct fimc_ctx *ctx)
 {
 	/* The one only mode supported in SoC. */
 	ctx->in_order_2p = S5P_FIMC_LSB_CRCB;
@@ -518,7 +518,7 @@ static void fimc_set_yuv_order(struct fimc_ctx *ctx)
 	dbg("ctx->out_order_1p= %d", ctx->out_order_1p);
 }
 
-static void fimc_prepare_dma_offset(struct fimc_ctx *ctx, struct fimc_frame *f)
+void fimc_prepare_dma_offset(struct fimc_ctx *ctx, struct fimc_frame *f)
 {
 	struct samsung_fimc_variant *variant = ctx->fimc_dev->variant;
 	u32 i, depth = 0;
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 86288c8..ab911be 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -663,6 +663,9 @@ int fimc_set_scaler_info(struct fimc_ctx *ctx);
 int fimc_prepare_config(struct fimc_ctx *ctx, u32 flags);
 int fimc_prepare_addr(struct fimc_ctx *ctx, struct vb2_buffer *vb,
 		      struct fimc_frame *frame, struct fimc_addr *paddr);
+void fimc_prepare_dma_offset(struct fimc_ctx *ctx, struct fimc_frame *f);
+void fimc_set_yuv_order(struct fimc_ctx *ctx);
+
 int fimc_register_m2m_device(struct fimc_dev *fimc,
 			     struct v4l2_device *v4l2_dev);
 void fimc_unregister_m2m_device(struct fimc_dev *fimc);
-- 
1.7.5.4

[PATCH v3 13/19] s5p-fimc: Correct color format enumeration

[Sylwester Nawrocki]

Create separate VIDIOC_ENUM_FMT ioctl handlers for video capture
and mem-to-mem video node. This is needed as some formats are
valid only for the video capture and some only for the mem-to-mem
video node. Create single function for pixel format lookup out of
find_mbus_format() and find_format().

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |   32 ++++++++++----
 drivers/media/video/s5p-fimc/fimc-core.c    |   61 +++++++++++++--------------
 drivers/media/video/s5p-fimc/fimc-core.h    |    5 +-
 drivers/media/video/s5p-fimc/fimc-mdevice.c |    2 +-
 4 files changed, 55 insertions(+), 45 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index 7af4ac5..fb0d89a 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -376,6 +376,22 @@ static int fimc_vidioc_querycap_capture(struct file *file, void *priv,
 	return 0;
 }
 
+static int fimc_cap_enum_fmt_mplane(struct file *file, void *priv,
+				    struct v4l2_fmtdesc *f)
+{
+	struct fimc_fmt *fmt;
+
+	fmt = fimc_find_format(NULL, NULL, FMT_FLAGS_CAM | FMT_FLAGS_M2M,
+			       f->index);
+	if (!fmt)
+		return -EINVAL;
+	strncpy(f->description, fmt->name, sizeof(f->description) - 1);
+	f->pixelformat = fmt->fourcc;
+	if (fmt->fourcc == V4L2_MBUS_FMT_JPEG_1X8)
+		f->flags |= V4L2_FMT_FLAG_COMPRESSED;
+	return 0;
+}
+
 /* Synchronize formats of the camera interface input and attached  sensor. */
 static int sync_capture_fmt(struct fimc_ctx *ctx)
 {
@@ -394,7 +410,7 @@ static int sync_capture_fmt(struct fimc_ctx *ctx)
 	}
 	dbg("w: %d, h: %d, code= %d", fmt->width, fmt->height, fmt->code);
 
-	frame->fmt = find_mbus_format(fmt, FMT_FLAGS_CAM);
+	frame->fmt = fimc_find_format(NULL, &fmt->code, FMT_FLAGS_CAM, -1);
 	if (!frame->fmt) {
 		err("fimc source format not found\n");
 		return -EINVAL;
@@ -456,12 +472,10 @@ static int fimc_cap_s_fmt_mplane(struct file *file, void *priv,
 	frame = &ctx->d_frame;
 
 	pix = &f->fmt.pix_mp;
-	frame->fmt = find_format(f, FMT_FLAGS_M2M | FMT_FLAGS_CAM);
-	if (!frame->fmt) {
-		v4l2_err(fimc->vid_cap.vfd,
-			 "Not supported capture (FIMC target) color format\n");
+	frame->fmt = fimc_find_format(&pix->pixelformat, NULL,
+				      FMT_FLAGS_M2M | FMT_FLAGS_CAM, 0);
+	if (WARN(frame->fmt == NULL, "Pixel format lookup failed\n"))
 		return -EINVAL;
-	}
 
 	for (i = 0; i < frame->fmt->colplanes; i++) {
 		frame->payload[i] =
@@ -641,7 +655,7 @@ static int fimc_cap_s_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 static const struct v4l2_ioctl_ops fimc_capture_ioctl_ops = {
 	.vidioc_querycap		= fimc_vidioc_querycap_capture,
 
-	.vidioc_enum_fmt_vid_cap_mplane	= fimc_vidioc_enum_fmt_mplane,
+	.vidioc_enum_fmt_vid_cap_mplane	= fimc_cap_enum_fmt_mplane,
 	.vidioc_try_fmt_vid_cap_mplane	= fimc_cap_try_fmt_mplane,
 	.vidioc_s_fmt_vid_cap_mplane	= fimc_cap_s_fmt_mplane,
 	.vidioc_g_fmt_vid_cap_mplane	= fimc_cap_g_fmt_mplane,
@@ -702,7 +716,6 @@ int fimc_register_capture_device(struct fimc_dev *fimc,
 	struct video_device *vfd;
 	struct fimc_vid_cap *vid_cap;
 	struct fimc_ctx *ctx;
-	struct v4l2_format f;
 	struct fimc_frame *fr;
 	struct vb2_queue *q;
 	int ret = -ENOMEM;
@@ -717,9 +730,8 @@ int fimc_register_capture_device(struct fimc_dev *fimc,
 	ctx->state	 = FIMC_CTX_CAP;
 
 	/* Default format of the output frames */
-	f.fmt.pix.pixelformat = V4L2_PIX_FMT_RGB32;
 	fr = &ctx->d_frame;
-	fr->fmt = find_format(&f, FMT_FLAGS_M2M);
+	fr->fmt = fimc_find_format(NULL, NULL, FMT_FLAGS_CAM, 0);
 	fr->width = fr->f_width = fr->o_width = 640;
 	fr->height = fr->f_height = fr->o_height = 480;
 
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index a3edaf8..8bd8d30 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -866,18 +866,17 @@ static int fimc_m2m_querycap(struct file *file, void *fh,
 	return 0;
 }
 
-int fimc_vidioc_enum_fmt_mplane(struct file *file, void *priv,
-				struct v4l2_fmtdesc *f)
+static int fimc_m2m_enum_fmt_mplane(struct file *file, void *priv,
+				    struct v4l2_fmtdesc *f)
 {
 	struct fimc_fmt *fmt;
 
-	if (f->index >= ARRAY_SIZE(fimc_formats))
+	fmt = fimc_find_format(NULL, NULL, FMT_FLAGS_M2M, f->index);
+	if (!fmt)
 		return -EINVAL;
 
-	fmt = &fimc_formats[f->index];
 	strncpy(f->description, fmt->name, sizeof(f->description) - 1);
 	f->pixelformat = fmt->fourcc;
-
 	return 0;
 }
 
@@ -916,34 +915,33 @@ static int fimc_m2m_g_fmt_mplane(struct file *file, void *fh,
 	return fimc_fill_format(frame, f);
 }
 
-struct fimc_fmt *find_format(struct v4l2_format *f, unsigned int mask)
-{
-	struct fimc_fmt *fmt;
-	unsigned int i;
-
-	for (i = 0; i < ARRAY_SIZE(fimc_formats); ++i) {
-		fmt = &fimc_formats[i];
-		if (fmt->fourcc == f->fmt.pix_mp.pixelformat &&
-		   (fmt->flags & mask))
-			break;
-	}
-
-	return (i == ARRAY_SIZE(fimc_formats)) ? NULL : fmt;
-}
-
-struct fimc_fmt *find_mbus_format(struct v4l2_mbus_framefmt *f,
-				  unsigned int mask)
+/**
+ * fimc_find_format - lookup fimc color format by fourcc or media bus format
+ * @pixelformat: fourcc to match, ignored if null
+ * @mbus_code: media bus code to match, ignored if null
+ * @mask: the color flags to match
+ * @index: offset in the fimc_formats array, ignored if negative
+ */
+struct fimc_fmt *fimc_find_format(u32 *pixelformat, u32 *mbus_code,
+				  unsigned int mask, int index)
 {
-	struct fimc_fmt *fmt;
+	struct fimc_fmt *fmt, *def_fmt = NULL;
 	unsigned int i;
+	int id = 0;
 
 	for (i = 0; i < ARRAY_SIZE(fimc_formats); ++i) {
 		fmt = &fimc_formats[i];
-		if (fmt->mbus_code == f->code && (fmt->flags & mask))
-			break;
+		if (!(fmt->flags & mask))
+			continue;
+		if (pixelformat && fmt->fourcc == *pixelformat)
+			return fmt;
+		if (mbus_code && fmt->mbus_code == *mbus_code)
+			return fmt;
+		if (index == id)
+			def_fmt = fmt;
+		id++;
 	}
-
-	return (i == ARRAY_SIZE(fimc_formats)) ? NULL : fmt;
+	return def_fmt;
 }
 
 int fimc_try_fmt_mplane(struct fimc_ctx *ctx, struct v4l2_format *f)
@@ -966,7 +964,7 @@ int fimc_try_fmt_mplane(struct fimc_ctx *ctx, struct v4l2_format *f)
 	dbg("w: %d, h: %d", pix->width, pix->height);
 
 	mask = is_output ? FMT_FLAGS_M2M : FMT_FLAGS_M2M | FMT_FLAGS_CAM;
-	fmt = find_format(f, mask);
+	fmt = fimc_find_format(&pix->pixelformat, NULL, mask, -1);
 	if (!fmt) {
 		v4l2_err(fimc->v4l2_dev, "Fourcc format (0x%X) invalid.\n",
 			 pix->pixelformat);
@@ -1061,7 +1059,8 @@ static int fimc_m2m_s_fmt_mplane(struct file *file, void *fh,
 		frame = &ctx->d_frame;
 
 	pix = &f->fmt.pix_mp;
-	frame->fmt = find_format(f, FMT_FLAGS_M2M);
+	frame->fmt = fimc_find_format(&pix->pixelformat, NULL,
+				      FMT_FLAGS_M2M, 0);
 	if (!frame->fmt)
 		return -EINVAL;
 
@@ -1290,8 +1289,8 @@ static int fimc_m2m_s_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 static const struct v4l2_ioctl_ops fimc_m2m_ioctl_ops = {
 	.vidioc_querycap		= fimc_m2m_querycap,
 
-	.vidioc_enum_fmt_vid_cap_mplane	= fimc_vidioc_enum_fmt_mplane,
-	.vidioc_enum_fmt_vid_out_mplane	= fimc_vidioc_enum_fmt_mplane,
+	.vidioc_enum_fmt_vid_cap_mplane	= fimc_m2m_enum_fmt_mplane,
+	.vidioc_enum_fmt_vid_out_mplane	= fimc_m2m_enum_fmt_mplane,
 
 	.vidioc_g_fmt_vid_cap_mplane	= fimc_m2m_g_fmt_mplane,
 	.vidioc_g_fmt_vid_out_mplane	= fimc_m2m_g_fmt_mplane,
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index ab911be..18e8c02 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -654,9 +654,8 @@ void fimc_ctrls_delete(struct fimc_ctx *ctx);
 void fimc_ctrls_activate(struct fimc_ctx *ctx, bool active);
 int fimc_fill_format(struct fimc_frame *frame, struct v4l2_format *f);
 
-struct fimc_fmt *find_format(struct v4l2_format *f, unsigned int mask);
-struct fimc_fmt *find_mbus_format(struct v4l2_mbus_framefmt *f,
-				  unsigned int mask);
+struct fimc_fmt *fimc_find_format(u32 *pixelformat, u32 *mbus_code,
+				  unsigned int mask, int index);
 
 int fimc_check_scaler_ratio(int sw, int sh, int dw, int dh, int rot);
 int fimc_set_scaler_info(struct fimc_ctx *ctx);
diff --git a/drivers/media/video/s5p-fimc/fimc-mdevice.c b/drivers/media/video/s5p-fimc/fimc-mdevice.c
index 54e889b..91f57d6 100644
--- a/drivers/media/video/s5p-fimc/fimc-mdevice.c
+++ b/drivers/media/video/s5p-fimc/fimc-mdevice.c
@@ -194,7 +194,7 @@ int fimc_pipeline_s_stream(struct fimc_dev *fimc, int on)
 	if ((on && p->csis) || !on)
 		ret = v4l2_subdev_call(on ? p->csis : p->sensor,
 				       video, s_stream, on);
-	if (ret && ret != -ENOIOCTLCMD)
+	if (ret < 0 && ret != -ENOIOCTLCMD)
 		return ret;
 	if ((!on && p->csis) || on)
 		ret = v4l2_subdev_call(on ? p->sensor : p->csis,
-- 
1.7.5.4

[PATCH v3 14/19] s5p-fimc: Convert to use media pipeline operations

[Sylwester Nawrocki]

Add format negotiation routine for sensor subdevs not exposing
a device node.
TRY_FMT ioctl is completed by a subsequent patch adding the
capture subdev. This way the try_fmt routines can be common
for the subdev and the video node.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |  109 +++++++++--------------
 drivers/media/video/s5p-fimc/fimc-core.c    |  125 +++++++++++++++------------
 drivers/media/video/s5p-fimc/fimc-core.h    |   14 ++-
 drivers/media/video/s5p-fimc/fimc-reg.c     |    8 +-
 4 files changed, 125 insertions(+), 131 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index fb0d89a..ecb0246 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -74,11 +74,9 @@ static int fimc_start_capture(struct fimc_dev *fimc)
 
 static int fimc_stop_capture(struct fimc_dev *fimc)
 {
-	unsigned long flags;
-	struct fimc_vid_cap *cap;
+	struct fimc_vid_cap *cap = &fimc->vid_cap;
 	struct fimc_vid_buffer *buf;
-
-	cap = &fimc->vid_cap;
+	unsigned long flags;
 
 	if (!fimc_capture_active(fimc))
 		return 0;
@@ -92,11 +90,10 @@ static int fimc_stop_capture(struct fimc_dev *fimc)
 			   !test_bit(ST_CAPT_SHUT, &fimc->state),
 			   FIMC_SHUTDOWN_TIMEOUT);
 
-	v4l2_subdev_call(cap->sd, video, s_stream, 0);
-
 	spin_lock_irqsave(&fimc->slock, flags);
 	fimc->state &= ~(1 << ST_CAPT_RUN | 1 << ST_CAPT_PEND |
-			 1 << ST_CAPT_SHUT | 1 << ST_CAPT_STREAM);
+			 1 << ST_CAPT_SHUT | 1 << ST_CAPT_STREAM |
+			 1 << ST_CAPT_ISP_STREAM);
 
 	fimc->vid_cap.active_buf_cnt = 0;
 
@@ -112,9 +109,9 @@ static int fimc_stop_capture(struct fimc_dev *fimc)
 	}
 
 	spin_unlock_irqrestore(&fimc->slock, flags);
-
 	dbg("state: 0x%lx", fimc->state);
-	return 0;
+
+	return fimc_pipeline_s_stream(fimc, 0);
 }
 
 int fimc_capture_suspend(struct fimc_dev *fimc)
@@ -197,15 +194,13 @@ static int buffer_prepare(struct vb2_buffer *vb)
 		return -EINVAL;
 
 	for (i = 0; i < ctx->d_frame.fmt->memplanes; i++) {
-		unsigned long size = get_plane_size(&ctx->d_frame, i);
-
+		unsigned long size = ctx->d_frame.payload[i];
 		if (vb2_plane_size(vb, i) < size) {
 			v4l2_err(ctx->fimc_dev->vid_cap.vfd,
 				 "User buffer too small (%ld < %ld)\n",
 				 vb2_plane_size(vb, i), size);
 			return -EINVAL;
 		}
-
 		vb2_set_plane_payload(vb, i, size);
 	}
 
@@ -214,10 +209,10 @@ static int buffer_prepare(struct vb2_buffer *vb)
 
 static void buffer_queue(struct vb2_buffer *vb)
 {
-	struct fimc_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue);
-	struct fimc_dev *fimc = ctx->fimc_dev;
 	struct fimc_vid_buffer *buf
 		= container_of(vb, struct fimc_vid_buffer, vb);
+	struct fimc_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue);
+	struct fimc_dev *fimc = ctx->fimc_dev;
 	struct fimc_vid_cap *vid_cap = &fimc->vid_cap;
 	unsigned long flags;
 	int min_bufs;
@@ -244,9 +239,14 @@ static void buffer_queue(struct vb2_buffer *vb)
 	min_bufs = vid_cap->reqbufs_count > 1 ? 2 : 1;
 
 	if (vid_cap->active_buf_cnt >= min_bufs &&
-	    !test_and_set_bit(ST_CAPT_STREAM, &fimc->state))
+	    !test_and_set_bit(ST_CAPT_STREAM, &fimc->state)) {
 		fimc_activate_capture(ctx);
+		spin_unlock_irqrestore(&fimc->slock, flags);
 
+		if (!test_and_set_bit(ST_CAPT_ISP_STREAM, &fimc->state))
+			fimc_pipeline_s_stream(fimc, 1);
+		return;
+	}
 	spin_unlock_irqrestore(&fimc->slock, flags);
 }
 
@@ -309,15 +309,21 @@ static int fimc_capture_open(struct file *file)
 	if (fimc_m2m_active(fimc))
 		return -EBUSY;
 
-	ret = pm_runtime_get_sync(&fimc->pdev->dev);
-	if (ret < 0) {
-		v4l2_fh_release(file);
-		return ret;
-	}
-
-	if (++fimc->vid_cap.refcnt == 1)
+	pm_runtime_get_sync(&fimc->pdev->dev);
+
+	if (++fimc->vid_cap.refcnt == 1) {
+		ret = fimc_pipeline_initialize(fimc,
+			       &fimc->vid_cap.vfd->entity, true);
+		if (ret < 0) {
+			dev_err(&fimc->pdev->dev,
+				"Video pipeline initialization failed\n");
+			pm_runtime_put_sync(&fimc->pdev->dev);
+			fimc->vid_cap.refcnt--;
+			v4l2_fh_release(file);
+			return ret;
+		}
 		ret = fimc_capture_ctrls_create(fimc);
-
+	}
 	return ret;
 }
 
@@ -329,10 +335,10 @@ static int fimc_capture_close(struct file *file)
 
 	if (--fimc->vid_cap.refcnt == 0) {
 		fimc_stop_capture(fimc);
+		fimc_pipeline_shutdown(fimc);
 		fimc_ctrls_delete(fimc->vid_cap.ctx);
 		vb2_queue_release(&fimc->vid_cap.vbq);
 	}
-
 	pm_runtime_put_sync(&fimc->pdev->dev);
 	return v4l2_fh_release(file);
 }
@@ -392,41 +398,11 @@ static int fimc_cap_enum_fmt_mplane(struct file *file, void *priv,
 	return 0;
 }
 
-/* Synchronize formats of the camera interface input and attached  sensor. */
-static int sync_capture_fmt(struct fimc_ctx *ctx)
-{
-	struct fimc_frame *frame = &ctx->s_frame;
-	struct fimc_dev *fimc = ctx->fimc_dev;
-	struct v4l2_mbus_framefmt *fmt = &fimc->vid_cap.fmt;
-	int ret;
 
-	fmt->width  = ctx->d_frame.o_width;
-	fmt->height = ctx->d_frame.o_height;
 
-	ret = v4l2_subdev_call(fimc->vid_cap.sd, video, s_mbus_fmt, fmt);
-	if (ret == -ENOIOCTLCMD) {
-		err("s_mbus_fmt failed");
-		return ret;
-	}
-	dbg("w: %d, h: %d, code= %d", fmt->width, fmt->height, fmt->code);
 
-	frame->fmt = fimc_find_format(NULL, &fmt->code, FMT_FLAGS_CAM, -1);
-	if (!frame->fmt) {
-		err("fimc source format not found\n");
-		return -EINVAL;
-	}
 
-	frame->f_width	= fmt->width;
-	frame->f_height = fmt->height;
-	frame->width	= fmt->width;
-	frame->height	= fmt->height;
-	frame->o_width	= fmt->width;
-	frame->o_height = fmt->height;
-	frame->offs_h	= 0;
-	frame->offs_v	= 0;
 
-	return 0;
-}
 
 static int fimc_cap_g_fmt_mplane(struct file *file, void *fh,
 				 struct v4l2_format *f)
@@ -446,7 +422,7 @@ static int fimc_cap_try_fmt_mplane(struct file *file, void *fh,
 	struct fimc_dev *fimc = video_drvdata(file);
 	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
 
-	return fimc_try_fmt_mplane(ctx, f);
+	return 0;
 }
 
 static int fimc_cap_s_fmt_mplane(struct file *file, void *priv,
@@ -462,10 +438,6 @@ static int fimc_cap_s_fmt_mplane(struct file *file, void *priv,
 	if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
 		return -EINVAL;
 
-	ret = fimc_try_fmt_mplane(ctx, f);
-	if (ret)
-		return ret;
-
 	if (vb2_is_busy(&fimc->vid_cap.vbq) || fimc_capture_active(fimc))
 		return -EBUSY;
 
@@ -495,7 +467,6 @@ static int fimc_cap_s_fmt_mplane(struct file *file, void *priv,
 
 	ctx->state |= (FIMC_PARAMS | FIMC_DST_FMT);
 
-	ret = sync_capture_fmt(ctx);
 	return ret;
 }
 
@@ -503,12 +474,14 @@ static int fimc_cap_enum_input(struct file *file, void *priv,
 			       struct v4l2_input *i)
 {
 	struct fimc_dev *fimc = video_drvdata(file);
+	struct v4l2_subdev *sd = fimc->pipeline.sensor;
 
 	if (i->index != 0)
 		return -EINVAL;
 
-
 	i->type = V4L2_INPUT_TYPE_CAMERA;
+	if (sd)
+		strlcpy(i->name, sd->name, sizeof(i->name));
 	return 0;
 }
 
@@ -528,14 +501,16 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 {
 	struct fimc_dev *fimc = video_drvdata(file);
 	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
+	struct fimc_pipeline *p = &fimc->pipeline;
 
-	if (fimc_capture_active(fimc) || !fimc->vid_cap.sd)
+	if (fimc_capture_active(fimc))
 		return -EBUSY;
 
 	if (!(ctx->state & FIMC_DST_FMT)) {
 		v4l2_err(fimc->vid_cap.vfd, "Format is not set\n");
 		return -EINVAL;
 	}
+	media_entity_pipeline_start(&p->sensor->entity, p->pipe);
 
 	return vb2_streamon(&fimc->vid_cap.vbq, type);
 }
@@ -544,8 +519,13 @@ static int fimc_cap_streamoff(struct file *file, void *priv,
 			    enum v4l2_buf_type type)
 {
 	struct fimc_dev *fimc = video_drvdata(file);
+	struct v4l2_subdev *sd = fimc->pipeline.sensor;
+	int ret;
 
-	return vb2_streamoff(&fimc->vid_cap.vbq, type);
+	ret = vb2_streamoff(&fimc->vid_cap.vbq, type);
+	if (ret == 0)
+		media_entity_pipeline_stop(&sd->entity);
+	return ret;
 }
 
 static int fimc_cap_reqbufs(struct file *file, void *priv,
@@ -651,7 +631,6 @@ static int fimc_cap_s_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 	return 0;
 }
 
-
 static const struct v4l2_ioctl_ops fimc_capture_ioctl_ops = {
 	.vidioc_querycap		= fimc_vidioc_querycap_capture,
 
@@ -757,8 +736,6 @@ int fimc_register_capture_device(struct fimc_dev *fimc,
 	vid_cap->active_buf_cnt = 0;
 	vid_cap->reqbufs_count  = 0;
 	vid_cap->refcnt = 0;
-	/* Default color format for image sensor */
-	vid_cap->fmt.code = V4L2_MBUS_FMT_YUYV8_2X8;
 
 	INIT_LIST_HEAD(&vid_cap->pending_buf_q);
 	INIT_LIST_HEAD(&vid_cap->active_buf_q);
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index 8bd8d30..7a10fea 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -903,6 +903,60 @@ int fimc_fill_format(struct fimc_frame *frame, struct v4l2_format *f)
 	return 0;
 }
 
+void fimc_fill_frame(struct fimc_frame *frame, struct v4l2_format *f)
+{
+	struct v4l2_pix_format_mplane *pixm = &f->fmt.pix_mp;
+
+	frame->f_width  = pixm->plane_fmt[0].bytesperline;
+	if (frame->fmt->colplanes == 1)
+		frame->f_width = (frame->f_width * 8) / frame->fmt->depth[0];
+	frame->f_height	= pixm->height;
+	frame->width    = pixm->width;
+	frame->height   = pixm->height;
+	frame->o_width  = pixm->width;
+	frame->o_height = pixm->height;
+	frame->offs_h   = 0;
+	frame->offs_v   = 0;
+}
+
+/**
+ * fimc_adjust_mplane_desc - adjust bytesperline/sizeimage for each plane
+ * @fmt: fimc pixel format description (input)
+ * @width: requested pixel width
+ * @height: requested pixel height
+ * @pix: multi-plane format to adjust
+ */
+void fimc_adjust_mplane_format(struct fimc_fmt *fmt, u32 width, u32 height,
+			       struct v4l2_pix_format_mplane *pix)
+{
+	u32 bytesperline = 0;
+	int i;
+
+	pix->colorspace	= V4L2_COLORSPACE_JPEG;
+	pix->field = V4L2_FIELD_NONE;
+	pix->num_planes = fmt->memplanes;
+	pix->height = height;
+	pix->width = width;
+
+	for (i = 0; i < pix->num_planes; ++i) {
+		u32 bpl = pix->plane_fmt[i].bytesperline;
+		u32 *sizeimage = &pix->plane_fmt[i].sizeimage;
+
+		if (fmt->colplanes > 1 && (bpl == 0 || bpl < pix->width))
+			bpl = pix->width; /* Planar */
+
+		if (fmt->colplanes == 1 && /* Packed */
+		    (bpl == 0 || ((bpl * 8) / fmt->depth[i]) < pix->width))
+			bpl = (pix->width * fmt->depth[0]) / 8;
+
+		if (i == 0) /* Same bytesperline for each plane. */
+			bytesperline = bpl;
+
+		pix->plane_fmt[i].bytesperline = bytesperline;
+		*sizeimage = (pix->width * pix->height * fmt->depth[i]) / 8;
+	}
+}
+
 static int fimc_m2m_g_fmt_mplane(struct file *file, void *fh,
 				 struct v4l2_format *f)
 {
@@ -944,43 +998,33 @@ struct fimc_fmt *fimc_find_format(u32 *pixelformat, u32 *mbus_code,
 	return def_fmt;
 }
 
-int fimc_try_fmt_mplane(struct fimc_ctx *ctx, struct v4l2_format *f)
+static int fimc_try_fmt_mplane(struct fimc_ctx *ctx, struct v4l2_format *f)
 {
 	struct fimc_dev *fimc = ctx->fimc_dev;
 	struct samsung_fimc_variant *variant = fimc->variant;
 	struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp;
 	struct fimc_fmt *fmt;
-	u32 max_width, mod_x, mod_y, mask;
-	int i, is_output = 0;
+	u32 max_w, mod_x, mod_y;
 
-	if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
-		if (fimc_ctx_state_is_set(FIMC_CTX_CAP, ctx))
-			return -EINVAL;
-		is_output = 1;
-	} else if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+	if (!IS_M2M(f->type))
 		return -EINVAL;
-	}
 
 	dbg("w: %d, h: %d", pix->width, pix->height);
 
-	mask = is_output ? FMT_FLAGS_M2M : FMT_FLAGS_M2M | FMT_FLAGS_CAM;
-	fmt = fimc_find_format(&pix->pixelformat, NULL, mask, -1);
-	if (!fmt) {
-		v4l2_err(fimc->v4l2_dev, "Fourcc format (0x%X) invalid.\n",
-			 pix->pixelformat);
+	fmt = fimc_find_format(&pix->pixelformat, NULL, FMT_FLAGS_M2M, 0);
+	if (WARN(fmt == NULL, "Pixel format lookup failed"))
 		return -EINVAL;
-	}
 
 	if (pix->field == V4L2_FIELD_ANY)
 		pix->field = V4L2_FIELD_NONE;
-	else if (V4L2_FIELD_NONE != pix->field)
+	else if (pix->field != V4L2_FIELD_NONE)
 		return -EINVAL;
 
-	if (is_output) {
-		max_width = variant->pix_limit->scaler_dis_w;
+	if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
+		max_w = variant->pix_limit->scaler_dis_w;
 		mod_x = ffs(variant->min_inp_pixsize) - 1;
 	} else {
-		max_width = variant->pix_limit->out_rot_dis_w;
+		max_w = variant->pix_limit->out_rot_dis_w;
 		mod_x = ffs(variant->min_out_pixsize) - 1;
 	}
 
@@ -993,34 +1037,12 @@ int fimc_try_fmt_mplane(struct fimc_ctx *ctx, struct v4l2_format *f)
 		else
 			mod_y = mod_x;
 	}
+	dbg("mod_x: %d, mod_y: %d, max_w: %d", mod_x, mod_y, max_w);
 
-	dbg("mod_x: %d, mod_y: %d, max_w: %d", mod_x, mod_y, max_width);
-
-	v4l_bound_align_image(&pix->width, 16, max_width, mod_x,
+	v4l_bound_align_image(&pix->width, 16, max_w, mod_x,
 		&pix->height, 8, variant->pix_limit->scaler_dis_w, mod_y, 0);
 
-	pix->num_planes = fmt->memplanes;
-	pix->colorspace	= V4L2_COLORSPACE_JPEG;
-
-
-	for (i = 0; i < pix->num_planes; ++i) {
-		u32 bpl = pix->plane_fmt[i].bytesperline;
-		u32 *sizeimage = &pix->plane_fmt[i].sizeimage;
-
-		if (fmt->colplanes > 1 && (bpl == 0 || bpl < pix->width))
-			bpl = pix->width; /* Planar */
-
-		if (fmt->colplanes == 1 && /* Packed */
-		    (bpl == 0 || ((bpl * 8) / fmt->depth[i]) < pix->width))
-			bpl = (pix->width * fmt->depth[0]) / 8;
-
-		if (i == 0) /* Same bytesperline for each plane. */
-			mod_x = bpl;
-
-		pix->plane_fmt[i].bytesperline = mod_x;
-		*sizeimage = (pix->width * pix->height * fmt->depth[i]) / 8;
-	}
-
+	fimc_adjust_mplane_format(fmt, pix->width, pix->height, &f->fmt.pix_mp);
 	return 0;
 }
 
@@ -1069,15 +1091,7 @@ static int fimc_m2m_s_fmt_mplane(struct file *file, void *fh,
 			(pix->width * pix->height * frame->fmt->depth[i]) / 8;
 	}
 
-	frame->f_width	= pix->plane_fmt[0].bytesperline * 8 /
-		frame->fmt->depth[0];
-	frame->f_height	= pix->height;
-	frame->width	= pix->width;
-	frame->height	= pix->height;
-	frame->o_width	= pix->width;
-	frame->o_height = pix->height;
-	frame->offs_h	= 0;
-	frame->offs_v	= 0;
+	fimc_fill_frame(frame, f);
 
 	if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
 		fimc_ctx_state_lock_set(FIMC_PARAMS | FIMC_DST_FMT, ctx);
@@ -1157,8 +1171,8 @@ static int fimc_m2m_cropcap(struct file *file, void *fh,
 
 	cr->bounds.left		= 0;
 	cr->bounds.top		= 0;
-	cr->bounds.width	= frame->f_width;
-	cr->bounds.height	= frame->f_height;
+	cr->bounds.width	= frame->o_width;
+	cr->bounds.height	= frame->o_height;
 	cr->defrect		= cr->bounds;
 
 	return 0;
@@ -1622,7 +1636,6 @@ static int fimc_probe(struct platform_device *pdev)
 
 	init_waitqueue_head(&fimc->irq_queue);
 	spin_lock_init(&fimc->slock);
-
 	mutex_init(&fimc->lock);
 
 	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 18e8c02..36752f7 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -43,6 +43,7 @@
 #define SCALER_MAX_HRATIO	64
 #define SCALER_MAX_VRATIO	64
 #define DMA_MIN_SIZE		8
+#define FIMC_CAMIF_MAX_HEIGHT	0x2000
 
 /* indices to the clocks array */
 enum {
@@ -61,6 +62,7 @@ enum fimc_dev_flags {
 	ST_CAPT_PEND,
 	ST_CAPT_RUN,
 	ST_CAPT_STREAM,
+	ST_CAPT_ISP_STREAM,
 	ST_CAPT_SHUT,
 	ST_CAPT_INUSE,
 	ST_CAPT_APPLY_CFG,
@@ -95,6 +97,9 @@ enum fimc_color_fmt {
 
 #define fimc_fmt_is_rgb(x) ((x) & 0x10)
 
+#define IS_M2M(__strt) ((__strt) == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE || \
+			__strt == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+
 /* Cb/Cr chrominance components order for 2 plane Y/CbCr 4:2:2 formats. */
 #define	S5P_FIMC_LSB_CRCB	S5P_CIOCTRL_ORDER422_2P_LSB_CRCB
 
@@ -293,7 +298,6 @@ struct fimc_m2m_device {
  * struct fimc_vid_cap - camera capture device information
  * @ctx: hardware context data
  * @vfd: video device node for camera capture mode
- * @sd: pointer to camera sensor subdevice currently in use
  * @vd_pad: fimc video capture node pad
  * @fmt: Media Bus format configured at selected image sensor
  * @pending_buf_q: the pending buffer queue head
@@ -312,9 +316,8 @@ struct fimc_vid_cap {
 	struct fimc_ctx			*ctx;
 	struct vb2_alloc_ctx		*alloc_ctx;
 	struct video_device		*vfd;
-	struct v4l2_subdev		*sd;;
 	struct media_pad		vd_pad;
-	struct v4l2_mbus_framefmt	fmt;
+	struct v4l2_mbus_framefmt	mf;
 	struct list_head		pending_buf_q;
 	struct list_head		active_buf_q;
 	struct vb2_queue		vbq;
@@ -647,13 +650,13 @@ int fimc_hw_set_camera_type(struct fimc_dev *fimc,
 /* fimc-core.c */
 int fimc_vidioc_enum_fmt_mplane(struct file *file, void *priv,
 				struct v4l2_fmtdesc *f);
-int fimc_try_fmt_mplane(struct fimc_ctx *ctx, struct v4l2_format *f);
 int fimc_try_crop(struct fimc_ctx *ctx, struct v4l2_crop *cr);
 int fimc_ctrls_create(struct fimc_ctx *ctx);
 void fimc_ctrls_delete(struct fimc_ctx *ctx);
 void fimc_ctrls_activate(struct fimc_ctx *ctx, bool active);
 int fimc_fill_format(struct fimc_frame *frame, struct v4l2_format *f);
-
+void fimc_adjust_mplane_format(struct fimc_fmt *fmt, u32 width, u32 height,
+			       struct v4l2_pix_format_mplane *pix);
 struct fimc_fmt *fimc_find_format(u32 *pixelformat, u32 *mbus_code,
 				  unsigned int mask, int index);
 
@@ -664,6 +667,7 @@ int fimc_prepare_addr(struct fimc_ctx *ctx, struct vb2_buffer *vb,
 		      struct fimc_frame *frame, struct fimc_addr *paddr);
 void fimc_prepare_dma_offset(struct fimc_ctx *ctx, struct fimc_frame *f);
 void fimc_set_yuv_order(struct fimc_ctx *ctx);
+void fimc_fill_frame(struct fimc_frame *frame, struct v4l2_format *f);
 
 int fimc_register_m2m_device(struct fimc_dev *fimc,
 			     struct v4l2_device *v4l2_dev);
diff --git a/drivers/media/video/s5p-fimc/fimc-reg.c b/drivers/media/video/s5p-fimc/fimc-reg.c
index 50937b4..a1fff02 100644
--- a/drivers/media/video/s5p-fimc/fimc-reg.c
+++ b/drivers/media/video/s5p-fimc/fimc-reg.c
@@ -572,7 +572,7 @@ int fimc_hw_set_camera_source(struct fimc_dev *fimc,
 
 	if (cam->bus_type == FIMC_ITU_601 || cam->bus_type == FIMC_ITU_656) {
 		for (i = 0; i < ARRAY_SIZE(pix_desc); i++) {
-			if (fimc->vid_cap.fmt.code == pix_desc[i].pixelcode) {
+			if (fimc->vid_cap.mf.code == pix_desc[i].pixelcode) {
 				cfg = pix_desc[i].cisrcfmt;
 				bus_width = pix_desc[i].bus_width;
 				break;
@@ -582,7 +582,7 @@ int fimc_hw_set_camera_source(struct fimc_dev *fimc,
 		if (i == ARRAY_SIZE(pix_desc)) {
 			v4l2_err(fimc->vid_cap.vfd,
 				 "Camera color format not supported: %d\n",
-				 fimc->vid_cap.fmt.code);
+				 fimc->vid_cap.mf.code);
 			return -EINVAL;
 		}
 
@@ -642,12 +642,12 @@ int fimc_hw_set_camera_type(struct fimc_dev *fimc,
 			cfg |= S5P_CIGCTRL_SELCAM_MIPI_A;
 
 		/* TODO: add remaining supported formats. */
-		if (vid_cap->fmt.code == V4L2_MBUS_FMT_VYUY8_2X8) {
+		if (vid_cap->mf.code == V4L2_MBUS_FMT_VYUY8_2X8) {
 			tmp = S5P_CSIIMGFMT_YCBCR422_8BIT;
 		} else {
 			v4l2_err(fimc->vid_cap.vfd,
 				 "Not supported camera pixel format: %d",
-				 vid_cap->fmt.code);
+				 vid_cap->mf.code);
 			return -EINVAL;
 		}
 		tmp |= (cam->csi_data_align == 32) << 8;
-- 
1.7.5.4

[PATCH v3 15/19] s5p-fimc: Add subdev for the FIMC processing block

[Sylwester Nawrocki]

This subdev interface exposes the internal scaler and color converter
functionality to user space. Resolution and media bus format can now
be configured explicitly by applications. Camera frame composition
onto the output buffer can be configured through set/get_crop at FIMC.{n}
source pad. Additionally crop, controls and composition may be
reconfigured while streaming.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |  688 ++++++++++++++++++++++++---
 drivers/media/video/s5p-fimc/fimc-core.c    |   32 +-
 drivers/media/video/s5p-fimc/fimc-core.h    |   51 ++-
 drivers/media/video/s5p-fimc/fimc-mdevice.c |   37 ++-
 4 files changed, 709 insertions(+), 99 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index ecb0246..4d7b279 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -67,6 +67,7 @@ static int fimc_start_capture(struct fimc_dev *fimc)
 		fimc_hw_set_effect(ctx);
 		fimc_hw_set_output_path(ctx);
 		fimc_hw_set_out_dma(ctx);
+		clear_bit(ST_CAPT_APPLY_CFG, &fimc->state);
 	}
 	spin_unlock_irqrestore(&fimc->slock, flags);
 	return ret;
@@ -124,6 +125,33 @@ int fimc_capture_resume(struct fimc_dev *fimc)
 	return 0;
 }
 
+/**
+ * fimc_capture_config_update - apply the camera interface configuration
+ *
+ * To be called from within the interrupt handler with fimc.slock
+ * spinlock held. Camera pixel crop, rotation and image flip state will
+ * be updated in the H/W.
+ */
+int fimc_capture_config_update(struct fimc_ctx *ctx)
+{
+	struct fimc_dev *fimc = ctx->fimc_dev;
+	int ret;
+
+	spin_lock(&ctx->slock);
+	fimc_hw_set_camera_offset(fimc, &ctx->s_frame);
+	ret = fimc_set_scaler_info(ctx);
+	if (ret == 0) {
+		fimc_hw_set_prescaler(ctx);
+		fimc_hw_set_mainscaler(ctx);
+		fimc_hw_set_target_format(ctx);
+		fimc_hw_set_rotation(ctx);
+		fimc_prepare_dma_offset(ctx, &ctx->d_frame);
+		fimc_hw_set_out_dma(ctx);
+	}
+	spin_unlock(&ctx->slock);
+	return ret;
+}
+
 static int start_streaming(struct vb2_queue *q)
 {
 	struct fimc_ctx *ctx = q->drv_priv;
@@ -289,7 +317,7 @@ int fimc_capture_ctrls_create(struct fimc_dev *fimc)
 	if (fimc->vid_cap.ctx->ctrls_rdy)
 		return 0;
 	ret = fimc_ctrls_create(fimc->vid_cap.ctx);
-	if (ret || subdev_has_devnode(fimc->pipeline.sensor))
+	if (ret || !fimc->vid_cap.vid_dev_compat)
 		return ret;
 	return v4l2_ctrl_add_handler(&fimc->vid_cap.ctx->ctrl_handler,
 				    fimc->pipeline.sensor->ctrl_handler);
@@ -358,7 +386,6 @@ static int fimc_capture_mmap(struct file *file, struct vm_area_struct *vma)
 	return vb2_mmap(&fimc->vid_cap.vbq, vma);
 }
 
-/* video device file operations */
 static const struct v4l2_file_operations fimc_capture_fops = {
 	.owner		= THIS_MODULE,
 	.open		= fimc_capture_open,
@@ -368,6 +395,143 @@ static const struct v4l2_file_operations fimc_capture_fops = {
 	.mmap		= fimc_capture_mmap,
 };
 
+/*
+ * Format and crop negotiation helpers
+ */
+
+static struct fimc_fmt *fimc_capture_try_format(struct fimc_ctx *ctx,
+						u32 *width, u32 *height,
+						u32 *code, u32 *fourcc, int pad)
+{
+	bool rotation = ctx->rotation == 90 || ctx->rotation == 270;
+	struct fimc_dev *fimc = ctx->fimc_dev;
+	struct samsung_fimc_variant *var = fimc->variant;
+	struct fimc_pix_limit *pl = var->pix_limit;
+	struct fimc_frame *dst = &ctx->d_frame;
+	u32 depth, min_w, max_w, min_h, align_h = 4;
+	u32 mask = FMT_FLAGS_CAM;
+	struct fimc_fmt *ffmt;
+
+	/* Color conversion from/to JPEG is not supported */
+	if (pad == FIMC_SD_PAD_SOURCE &&
+	    fimc_fmt_is_jpeg(ctx->s_frame.fmt->color))
+		*code = V4L2_MBUS_FMT_JPEG_1X8;
+	if (fourcc)
+		mask |= FMT_FLAGS_M2M;
+	ffmt = fimc_find_format(fourcc, code, mask, 0);
+	if (WARN_ON(!ffmt))
+		return NULL;
+	if (code)
+		*code = ffmt->mbus_code;
+	if (fourcc)
+		*fourcc = ffmt->fourcc;
+
+	if (pad == FIMC_SD_PAD_SINK) {
+		max_w = fimc_fmt_is_jpeg(ffmt->color) ?
+			pl->scaler_dis_w : pl->scaler_en_w;
+		/* Apply the camera input interface pixel constraints */
+		v4l_bound_align_image(width, 32, max_w, 4,
+				      height, 32, FIMC_CAMIF_MAX_HEIGHT, 1,
+				      0);
+		return ffmt;
+	}
+	/* Can't scale or crop in transparent (JPEG) transfer mode */
+	if (fimc_fmt_is_jpeg(ffmt->color)) {
+		*width  = ctx->s_frame.f_width;
+		*height = ctx->s_frame.f_height;
+		return ffmt;
+	}
+	/* Apply the scaler and the output DMA constraints */
+	max_w = rotation ? pl->out_rot_en_w : pl->out_rot_dis_w;
+	min_w = ctx->state & FIMC_DST_CROP ? dst->width : var->min_out_pixsize;
+	min_h = ctx->state & FIMC_DST_CROP ? dst->height : var->min_out_pixsize;
+	if (fimc->id == 1 && var->pix_hoff)
+		align_h = fimc_fmt_is_rgb(ffmt->color) ? 0 : 1;
+
+	depth = fimc_get_format_depth(ffmt);
+	v4l_bound_align_image(width, min_w, max_w,
+			      ffs(var->min_out_pixsize) - 1,
+			      height, min_h, FIMC_CAMIF_MAX_HEIGHT,
+			      align_h,
+			      64/(ALIGN(depth, 8)));
+
+	dbg("pad%d: code: 0x%x, %dx%d. dst fmt: %dx%d",
+	    pad, code ? *code : 0, *width, *height,
+	    dst->f_width, dst->f_height);
+
+	return ffmt;
+}
+
+static void fimc_capture_try_crop(struct fimc_ctx *ctx, struct v4l2_rect *r,
+				  int pad)
+{
+	bool rotate = ctx->rotation == 90 || ctx->rotation == 270;
+	struct fimc_dev *fimc = ctx->fimc_dev;
+	struct samsung_fimc_variant *var = fimc->variant;
+	struct fimc_pix_limit *pl = var->pix_limit;
+	struct fimc_frame *sink = &ctx->s_frame;
+	u32 max_w, max_h, min_w = 0, min_h = 0, min_sz;
+	u32 align_sz = 0, align_h = 4;
+	u32 max_sc_h, max_sc_v;
+
+	/* In JPEG transparent transfer mode cropping is not supported */
+	if (fimc_fmt_is_jpeg(ctx->d_frame.fmt->color)) {
+		r->width  = sink->f_width;
+		r->height = sink->f_height;
+		r->left   = r->top = 0;
+		return;
+	}
+	if (pad == FIMC_SD_PAD_SOURCE) {
+		if (ctx->rotation != 90 && ctx->rotation != 270)
+			align_h = 1;
+		max_sc_h = min(SCALER_MAX_HRATIO, 1 << (ffs(sink->width) - 3));
+		max_sc_v = min(SCALER_MAX_VRATIO, 1 << (ffs(sink->height) - 1));
+		min_sz = var->min_out_pixsize;
+	} else {
+		u32 depth = fimc_get_format_depth(sink->fmt);
+		align_sz = 64/ALIGN(depth, 8);
+		min_sz = var->min_inp_pixsize;
+		min_w = min_h = min_sz;
+		max_sc_h = max_sc_v = 1;
+	}
+	/*
+	 * For the crop rectangle at source pad the following constraints
+	 * must be met:
+	 * - it must fit in the sink pad format rectangle (f_width/f_height);
+	 * - maximum downscaling ratio is 64;
+	 * - maximum crop size depends if the rotator is used or not;
+	 * - the sink pad format width/height must be 4 multiple of the
+	 *   prescaler ratios determined by sink pad size and source pad crop,
+	 *   the prescaler ratio is returned by fimc_get_scaler_factor().
+	 */
+	max_w = min_t(u32,
+		      rotate ? pl->out_rot_en_w : pl->out_rot_dis_w,
+		      rotate ? sink->f_height : sink->f_width);
+	max_h = min_t(u32, FIMC_CAMIF_MAX_HEIGHT, sink->f_height);
+	if (pad == FIMC_SD_PAD_SOURCE) {
+		min_w = min_t(u32, max_w, sink->f_width / max_sc_h);
+		min_h = min_t(u32, max_h, sink->f_height / max_sc_v);
+		if (rotate) {
+			swap(max_sc_h, max_sc_v);
+			swap(min_w, min_h);
+		}
+	}
+	v4l_bound_align_image(&r->width, min_w, max_w, ffs(min_sz) - 1,
+			      &r->height, min_h, max_h, align_h,
+			      align_sz);
+	/* Adjust left/top if cropping rectangle is out of bounds */
+	r->left = clamp_t(u32, r->left, 0, sink->f_width - r->width);
+	r->top  = clamp_t(u32, r->top, 0, sink->f_height - r->height);
+	r->left = round_down(r->left, var->hor_offs_align);
+
+	dbg("pad%d: (%d,%d)/%dx%d, sink fmt: %dx%d",
+	    pad, r->left, r->top, r->width, r->height,
+	    sink->f_width, sink->f_height);
+}
+
+/*
+ * The video node ioctl operations
+ */
 static int fimc_vidioc_querycap_capture(struct file *file, void *priv,
 					struct v4l2_capability *cap)
 {
@@ -398,11 +562,86 @@ static int fimc_cap_enum_fmt_mplane(struct file *file, void *priv,
 	return 0;
 }
 
+/**
+ * fimc_pipeline_try_format - negotiate and/or set formats at pipeline
+ *                            elements
+ * @ctx: FIMC capture context
+ * @tfmt: media bus format to try/set on subdevs
+ * @fmt_id: fimc pixel format id corresponding to returned @tfmt (output)
+ * @set: true to set format on subdevs, false to try only
+ */
+static int fimc_pipeline_try_format(struct fimc_ctx *ctx,
+				    struct v4l2_mbus_framefmt *tfmt,
+				    struct fimc_fmt **fmt_id,
+				    bool set)
+{
+	struct fimc_dev *fimc = ctx->fimc_dev;
+	struct v4l2_subdev *sd = fimc->pipeline.sensor;
+	struct v4l2_subdev *csis = fimc->pipeline.csis;
+	struct v4l2_subdev_format sfmt;
+	struct v4l2_mbus_framefmt *mf = &sfmt.format;
+	struct fimc_fmt *ffmt = NULL;
+	int ret, i = 0;
 
+	if (WARN_ON(!sd || !tfmt))
+		return -EINVAL;
 
+	if (!fimc->vid_cap.vid_dev_compat)
+		return 0;
 
+	memset(&sfmt, 0, sizeof(sfmt));
+	sfmt.format = *tfmt;
+
+	sfmt.which = set ? V4L2_SUBDEV_FORMAT_ACTIVE : V4L2_SUBDEV_FORMAT_TRY;
+	while (1) {
+		if (mf->code == 0) {
+			ffmt = fimc_find_format(NULL, NULL, FMT_FLAGS_CAM, i++);
+			if (WARN_ON(!ffmt)) {
+				/*
+				 * Notify user-space if common pixel code for
+				 * host and sensor does not exist.
+				 */
+				return -EINVAL;
+			}
+			mf->code = tfmt->code = ffmt->mbus_code;
+		}
+		ret = v4l2_subdev_call(sd, pad, set_fmt, NULL, &sfmt);
+		if (ret)
+			return ret;
+		if (mf->code != tfmt->code) {
+			mf->code = 0;
+			continue;
+		}
+		if (mf->width != tfmt->width || mf->width != tfmt->width) {
+			u32 pfcc = ffmt ? ffmt->fourcc :
+				fmt_id ? (*fmt_id)->fourcc : 0;
+			tfmt->width  = mf->width;
+			tfmt->height = mf->height;
+			ffmt = fimc_capture_try_format(ctx,
+					       &tfmt->width, &tfmt->height,
+					       NULL, &pfcc, FIMC_SD_PAD_SOURCE);
+			if (ffmt && ffmt->mbus_code)
+				mf->code = ffmt->mbus_code;
+			if (mf->width != tfmt->width || mf->width != tfmt->width)
+				continue;
+		}
+		tfmt->code = mf->code;
+
+		if (csis) {
+			ret = v4l2_subdev_call(csis, pad, set_fmt, NULL, &sfmt);
+		}
+		if (mf->code == tfmt->code &&
+		    mf->width == tfmt->width && mf->width == tfmt->width)
+			break;
+	}
 
+	if (fmt_id && ffmt)
+		*fmt_id = ffmt;
+	*tfmt = *mf;
 
+	dbg("code: 0x%x, %dx%d", mf->code, mf->width, mf->height);
+	return 0;
+}
 
 static int fimc_cap_g_fmt_mplane(struct file *file, void *fh,
 				 struct v4l2_format *f)
@@ -419,53 +658,98 @@ static int fimc_cap_g_fmt_mplane(struct file *file, void *fh,
 static int fimc_cap_try_fmt_mplane(struct file *file, void *fh,
 				   struct v4l2_format *f)
 {
+	struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp;
 	struct fimc_dev *fimc = video_drvdata(file);
 	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
+	struct v4l2_mbus_framefmt mf;
+	struct fimc_fmt *ffmt;
+
+	if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+		return -EINVAL;
+	ffmt = fimc_capture_try_format(ctx, &pix->width, &pix->height,
+				       NULL, &pix->pixelformat,
+				       FIMC_SD_PAD_SOURCE);
+	if (!ffmt)
+		return -EIO;
+
+	mf.width  = pix->width;
+	mf.height = pix->height;
+	mf.code   = ffmt->mbus_code;
+
+	fimc_md_graph_lock(fimc);
+	fimc_pipeline_try_format(ctx, &mf, &ffmt, false);
+	fimc_md_graph_unlock(fimc);
+
+	if (pix->width != mf.width || pix->height != mf.height) {
+		/* Pixel resolution has been changed be sensor subdev */
+		pix->width  = mf.width;
+		pix->height = mf.height;
+		ffmt = fimc_capture_try_format(ctx, &mf.width, &mf.height,
+					       NULL, &pix->pixelformat,
+					       FIMC_SD_PAD_SOURCE);
+		if (pix->width != mf.width || pix->height != mf.height)
+			v4l2_err(fimc->vid_cap.vfd,
+				 "Not supported format returned from sensor");
+	}
 
+	fimc_adjust_mplane_format(ffmt, pix->width, pix->height, pix);
 	return 0;
 }
 
+
 static int fimc_cap_s_fmt_mplane(struct file *file, void *priv,
 				 struct v4l2_format *f)
 {
 	struct fimc_dev *fimc = video_drvdata(file);
 	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
-	struct v4l2_pix_format_mplane *pix;
-	struct fimc_frame *frame;
-	int ret;
-	int i;
+	struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp;
+	struct v4l2_mbus_framefmt *mf = &fimc->vid_cap.mf;
+	struct fimc_frame *ff = &ctx->d_frame;
+	struct fimc_fmt *s_fmt;
+	int ret, i;
 
 	if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
 		return -EINVAL;
 
-	if (vb2_is_busy(&fimc->vid_cap.vbq) || fimc_capture_active(fimc))
+	if (vb2_is_busy(&fimc->vid_cap.vbq))
 		return -EBUSY;
 
-	frame = &ctx->d_frame;
-
-	pix = &f->fmt.pix_mp;
-	frame->fmt = fimc_find_format(&pix->pixelformat, NULL,
-				      FMT_FLAGS_M2M | FMT_FLAGS_CAM, 0);
-	if (WARN(frame->fmt == NULL, "Pixel format lookup failed\n"))
-		return -EINVAL;
-
-	for (i = 0; i < frame->fmt->colplanes; i++) {
-		frame->payload[i] =
-			(pix->width * pix->height * frame->fmt->depth[i]) >> 3;
-	}
-
-	/* Output DMA frame pixel size and offsets. */
-	frame->f_width = pix->plane_fmt[0].bytesperline * 8
-			/ frame->fmt->depth[0];
-	frame->f_height = pix->height;
-	frame->width	= pix->width;
-	frame->height	= pix->height;
-	frame->o_width	= pix->width;
-	frame->o_height = pix->height;
-	frame->offs_h	= 0;
-	frame->offs_v	= 0;
-
-	ctx->state |= (FIMC_PARAMS | FIMC_DST_FMT);
+	/* Try the format at the scaler and DMA output */
+	ff->fmt = fimc_capture_try_format(ctx, &pix->width, &pix->height,
+					  NULL, &pix->pixelformat,
+					  FIMC_SD_PAD_SOURCE);
+	if (!ff->fmt)
+		return -EIO;
+	/* Try the format at the camera interface input */
+	mf->code   = ff->fmt->mbus_code;
+	mf->width  = pix->width;
+	mf->height = pix->height;
+
+	fimc_md_graph_lock(fimc);
+	ret = fimc_pipeline_try_format(ctx, mf, &s_fmt, true);
+	fimc_md_graph_unlock(fimc);
+	if (ret)
+		return ret;
+	pix->width  = mf->width;
+	pix->height = mf->height;
+
+	fimc_adjust_mplane_format(ff->fmt, pix->width, pix->height, pix);
+	for (i = 0; i < ff->fmt->colplanes; i++)
+		ff->payload[i] =
+			(pix->width * pix->height * ff->fmt->depth[i]) / 8;
+
+	ctx->state |= FIMC_DST_FMT;
+	set_frame_bounds(ff, pix->width, pix->height);
+	/* Reset the composition rectangle if not yet configured */
+	if (!(ctx->state & FIMC_DST_CROP))
+		set_frame_crop(ff, 0, 0, mf->width, mf->height);
+
+	ff = &ctx->s_frame;
+	ff->fmt = s_fmt;
+	set_frame_bounds(ff, pix->width, pix->height);
+	/* Reset cropping at the host interface if required */
+	if (fimc->vid_cap.vid_dev_compat)
+		set_frame_crop(ff, 0, 0, mf->width, mf->height);
 
 	return ret;
 }
@@ -496,12 +780,70 @@ static int fimc_cap_g_input(struct file *file, void *priv, unsigned int *i)
 	return 0;
 }
 
+/**
+ * fimc_pipeline_validate - check for formats inconsistencies
+ *                          beween source and sink pad of each link
+ *
+ * Return 0 if all formats match or -EPIPE otherwise.
+ */
+static int fimc_pipeline_validate(struct fimc_dev *fimc)
+{
+	struct v4l2_subdev_format sink_fmt, src_fmt;
+	struct fimc_vid_cap *vid_cap = &fimc->vid_cap;
+	struct v4l2_subdev *sd;
+	struct media_pad *pad;
+	int ret;
+
+	/* Start with the video capture node pad */
+	pad = media_entity_remote_source(&vid_cap->vd_pad);
+	if (pad == NULL)
+		return -EPIPE;
+	/* FIMC.{N} subdevice */
+	sd = media_entity_to_v4l2_subdev(pad->entity);
+
+	while (1) {
+		/* Retrieve format at the sink pad */
+		pad = &sd->entity.pads[0];
+		if (!(pad->flags & MEDIA_PAD_FL_SINK))
+			break;
+		/* Don't call FIMC subdev operation to avoid a deadlock */
+		if (sd == fimc->vid_cap.subdev) {
+			sink_fmt.format.width = vid_cap->ctx->s_frame.f_width;
+			sink_fmt.format.height = vid_cap->ctx->s_frame.f_height;
+		} else {
+			sink_fmt.pad = pad->index;
+			sink_fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+			ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &sink_fmt);
+			if (ret < 0 && ret != -ENOIOCTLCMD)
+				return -EPIPE;
+		}
+		/* Retrieve format at the source pad */
+		pad = media_entity_remote_source(pad);
+		if (pad == NULL ||
+		    media_entity_type(pad->entity) != MEDIA_ENT_T_V4L2_SUBDEV)
+			break;
+
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		src_fmt.pad = pad->index;
+		src_fmt.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+		ret = v4l2_subdev_call(sd, pad, get_fmt, NULL, &src_fmt);
+		if (ret < 0 && ret != -ENOIOCTLCMD)
+			return -EPIPE;
+
+		if (src_fmt.format.width != sink_fmt.format.width ||
+		    src_fmt.format.height != sink_fmt.format.height)
+			return -EPIPE;
+	}
+	return 0;
+}
+
 static int fimc_cap_streamon(struct file *file, void *priv,
 			     enum v4l2_buf_type type)
 {
 	struct fimc_dev *fimc = video_drvdata(file);
 	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
 	struct fimc_pipeline *p = &fimc->pipeline;
+	int ret;
 
 	if (fimc_capture_active(fimc))
 		return -EBUSY;
@@ -512,6 +854,11 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	}
 	media_entity_pipeline_start(&p->sensor->entity, p->pipe);
 
+	if (!fimc->vid_cap.vid_dev_compat) {
+		ret = fimc_pipeline_validate(fimc);
+		if (ret)
+			return ret;
+	}
 	return vb2_streamon(&fimc->vid_cap.vbq, type);
 }
 
@@ -598,35 +945,16 @@ static int fimc_cap_s_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 {
 	struct fimc_dev *fimc = video_drvdata(file);
 	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
-	struct fimc_frame *f;
-	int ret = -EINVAL;
-
-	if (fimc_capture_active(fimc))
-		return -EBUSY;
-
-	ret = fimc_try_crop(ctx, cr);
-	if (ret)
-		return ret;
-
-	if (!(ctx->state & FIMC_DST_FMT)) {
-		v4l2_err(fimc->vid_cap.vfd, "Capture format is not set\n");
-		return -EINVAL;
-	}
+	struct fimc_frame *ff;
+	unsigned long flags;
 
-	f = &ctx->s_frame;
-	/* Check for the pixel scaling ratio when cropping input image. */
-	ret = fimc_check_scaler_ratio(cr->c.width, cr->c.height,
-				      ctx->d_frame.width, ctx->d_frame.height,
-				      ctx->rotation);
-	if (ret) {
-		v4l2_err(fimc->vid_cap.vfd, "Out of the scaler range\n");
-		return ret;
-	}
+	fimc_capture_try_crop(ctx, &cr->c, FIMC_SD_PAD_SINK);
+	ff = &ctx->s_frame;
 
-	f->offs_h = cr->c.left;
-	f->offs_v = cr->c.top;
-	f->width  = cr->c.width;
-	f->height = cr->c.height;
+	spin_lock_irqsave(&fimc->slock, flags);
+	set_frame_crop(ff, cr->c.left, cr->c.top, cr->c.width, cr->c.height);
+	set_bit(ST_CAPT_APPLY_CFG, &fimc->state);
+	spin_unlock_irqrestore(&fimc->slock, flags);
 
 	return 0;
 }
@@ -688,6 +1016,235 @@ static const struct media_entity_operations fimc_media_ops = {
 	.link_setup = fimc_link_setup,
 };
 
+static int fimc_subdev_enum_mbus_code(struct v4l2_subdev *sd,
+				      struct v4l2_subdev_fh *fh,
+				      struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct fimc_fmt *fmt;
+
+	fmt = fimc_find_format(NULL, NULL, FMT_FLAGS_CAM, code->index);
+	if (!fmt)
+		return -EINVAL;
+	code->code = fmt->mbus_code;
+	return 0;
+}
+
+static int fimc_subdev_get_fmt(struct v4l2_subdev *sd,
+			       struct v4l2_subdev_fh *fh,
+			       struct v4l2_subdev_format *fmt)
+{
+	struct fimc_dev *fimc = v4l2_get_subdevdata(sd);
+	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
+	struct v4l2_mbus_framefmt *mf;
+	struct fimc_frame *ff;
+
+	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
+		mf = v4l2_subdev_get_try_format(fh, fmt->pad);
+		fmt->format = *mf;
+		return 0;
+	}
+	mf = &fmt->format;
+	mf->colorspace = V4L2_COLORSPACE_JPEG;
+	ff = fmt->pad == FIMC_SD_PAD_SINK ? &ctx->s_frame : &ctx->d_frame;
+
+	mutex_lock(&fimc->lock);
+	/* The pixel code is same on both input and output pad */
+	if (!WARN_ON(ctx->s_frame.fmt == NULL))
+		mf->code = ctx->s_frame.fmt->mbus_code;
+	mf->width  = ff->f_width;
+	mf->height = ff->f_height;
+	mutex_unlock(&fimc->lock);
+
+	return 0;
+}
+
+static int fimc_subdev_set_fmt(struct v4l2_subdev *sd,
+			       struct v4l2_subdev_fh *fh,
+			       struct v4l2_subdev_format *fmt)
+{
+	struct fimc_dev *fimc = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *mf = &fmt->format;
+	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
+	struct fimc_frame *ff;
+	struct fimc_fmt *ffmt;
+
+	dbg("pad%d: code: 0x%x, %dx%d",
+	    fmt->pad, mf->code, mf->width, mf->height);
+
+	if (fmt->pad == FIMC_SD_PAD_SOURCE &&
+	    vb2_is_busy(&fimc->vid_cap.vbq))
+		return -EBUSY;
+
+	mutex_lock(&fimc->lock);
+	ffmt = fimc_capture_try_format(ctx, &mf->width, &mf->height,
+				       &mf->code, NULL, fmt->pad);
+	mutex_unlock(&fimc->lock);
+	mf->colorspace = V4L2_COLORSPACE_JPEG;
+
+	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
+		mf = v4l2_subdev_get_try_format(fh, fmt->pad);
+		*mf = fmt->format;
+		return 0;
+	}
+	ff = fmt->pad == FIMC_SD_PAD_SINK ?
+		&ctx->s_frame : &ctx->d_frame;
+
+	mutex_lock(&fimc->lock);
+	set_frame_bounds(ff, mf->width, mf->height);
+	ff->fmt = ffmt;
+
+	/* Reset the crop rectangle if required. */
+	if (!(fmt->pad == FIMC_SD_PAD_SOURCE && (ctx->state & FIMC_DST_CROP)))
+		set_frame_crop(ff, 0, 0, mf->width, mf->height);
+
+	if (fmt->pad == FIMC_SD_PAD_SINK)
+		ctx->state &= ~FIMC_DST_CROP;
+	mutex_unlock(&fimc->lock);
+	return 0;
+}
+
+static int fimc_subdev_get_crop(struct v4l2_subdev *sd,
+				struct v4l2_subdev_fh *fh,
+				struct v4l2_subdev_crop *crop)
+{
+	struct fimc_dev *fimc = v4l2_get_subdevdata(sd);
+	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
+	struct v4l2_rect *r = &crop->rect;
+	struct fimc_frame *ff;
+
+	if (crop->which == V4L2_SUBDEV_FORMAT_TRY) {
+		crop->rect = *v4l2_subdev_get_try_crop(fh, crop->pad);
+		return 0;
+	}
+	ff = crop->pad == FIMC_SD_PAD_SINK ?
+		&ctx->s_frame : &ctx->d_frame;
+
+	mutex_lock(&fimc->lock);
+	r->left	  = ff->offs_h;
+	r->top	  = ff->offs_v;
+	r->width  = ff->width;
+	r->height = ff->height;
+	mutex_unlock(&fimc->lock);
+
+	dbg("ff:%p, pad%d: l:%d, t:%d, %dx%d, f_w: %d, f_h: %d",
+	    ff, crop->pad, r->left, r->top, r->width, r->height,
+	    ff->f_width, ff->f_height);
+
+	return 0;
+}
+
+static int fimc_subdev_set_crop(struct v4l2_subdev *sd,
+				struct v4l2_subdev_fh *fh,
+				struct v4l2_subdev_crop *crop)
+{
+	struct fimc_dev *fimc = v4l2_get_subdevdata(sd);
+	struct fimc_ctx *ctx = fimc->vid_cap.ctx;
+	struct v4l2_rect *r = &crop->rect;
+	struct fimc_frame *ff;
+	unsigned long flags;
+
+	dbg("(%d,%d)/%dx%d", r->left, r->top, r->width, r->height);
+
+	ff = crop->pad == FIMC_SD_PAD_SOURCE ?
+		&ctx->d_frame : &ctx->s_frame;
+
+	mutex_lock(&fimc->lock);
+	fimc_capture_try_crop(ctx, r, crop->pad);
+
+	if (crop->which == V4L2_SUBDEV_FORMAT_TRY) {
+		mutex_lock(&fimc->lock);
+		*v4l2_subdev_get_try_crop(fh, crop->pad) = *r;
+		return 0;
+	}
+	spin_lock_irqsave(&fimc->slock, flags);
+	set_frame_crop(ff, r->left, r->top, r->width, r->height);
+	if (crop->pad == FIMC_SD_PAD_SOURCE)
+		ctx->state |= FIMC_DST_CROP;
+
+	set_bit(ST_CAPT_APPLY_CFG, &fimc->state);
+	spin_unlock_irqrestore(&fimc->slock, flags);
+
+	dbg("pad%d: (%d,%d)/%dx%d", crop->pad, r->left, r->top,
+	    r->width, r->height);
+
+	mutex_unlock(&fimc->lock);
+	return 0;
+}
+
+static struct v4l2_subdev_pad_ops fimc_subdev_pad_ops = {
+	.enum_mbus_code = fimc_subdev_enum_mbus_code,
+	.get_fmt = fimc_subdev_get_fmt,
+	.set_fmt = fimc_subdev_set_fmt,
+	.get_crop = fimc_subdev_get_crop,
+	.set_crop = fimc_subdev_set_crop,
+};
+
+static struct v4l2_subdev_ops fimc_subdev_ops = {
+	.pad = &fimc_subdev_pad_ops,
+};
+
+static int fimc_create_capture_subdev(struct fimc_dev *fimc,
+				      struct v4l2_device *v4l2_dev)
+{
+	struct v4l2_subdev *sd;
+	int ret;
+
+	sd = kzalloc(sizeof(*sd), GFP_KERNEL);
+	if (!sd)
+		return -ENOMEM;
+
+	v4l2_subdev_init(sd, &fimc_subdev_ops);
+	sd->flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+	snprintf(sd->name, sizeof(sd->name), "FIMC.%d", fimc->pdev->id);
+
+	fimc->vid_cap.sd_pads[FIMC_SD_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
+	fimc->vid_cap.sd_pads[FIMC_SD_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
+	ret = media_entity_init(&sd->entity, FIMC_SD_PADS_NUM,
+				fimc->vid_cap.sd_pads, 0);
+	if (ret)
+		goto error;
+	ret = v4l2_device_register_subdev(v4l2_dev, sd);
+	if (!ret) {
+		fimc->vid_cap.subdev = sd;
+		v4l2_set_subdevdata(sd, fimc);
+		return 0;
+	}
+	media_entity_cleanup(&sd->entity);
+error:
+	kfree(sd);
+	return ret;
+}
+
+static void fimc_destroy_capture_subdev(struct fimc_dev *fimc)
+{
+	struct v4l2_subdev *sd = fimc->vid_cap.subdev;
+
+	if (!sd)
+		return;
+	media_entity_cleanup(&sd->entity);
+	v4l2_device_unregister_subdev(sd);
+	kfree(sd);
+	sd = NULL;
+}
+
+/* Set default format at the camera input, scaler and the output DMA */
+static void fimc_cap_set_default_fmt(struct fimc_ctx *ctx)
+{
+	struct fimc_fmt *fmt = fimc_find_format(NULL, NULL, FMT_FLAGS_CAM, 0);
+	struct fimc_frame *ff = &ctx->s_frame;
+
+	ff->fmt = fmt;
+	set_frame_bounds(ff, 640, 480);
+	ff->width = 640;
+	ff->height = 480;
+
+	ff = &ctx->d_frame;
+	ff->fmt = fmt;
+	set_frame_bounds(ff, 640, 480);
+	ff->width = 640;
+	ff->height = 480;
+}
+
 /* fimc->lock must be already initialized */
 int fimc_register_capture_device(struct fimc_dev *fimc,
 				 struct v4l2_device *v4l2_dev)
@@ -695,7 +1252,6 @@ int fimc_register_capture_device(struct fimc_dev *fimc,
 	struct video_device *vfd;
 	struct fimc_vid_cap *vid_cap;
 	struct fimc_ctx *ctx;
-	struct fimc_frame *fr;
 	struct vb2_queue *q;
 	int ret = -ENOMEM;
 
@@ -708,11 +1264,7 @@ int fimc_register_capture_device(struct fimc_dev *fimc,
 	ctx->out_path	 = FIMC_DMA;
 	ctx->state	 = FIMC_CTX_CAP;
 
-	/* Default format of the output frames */
-	fr = &ctx->d_frame;
-	fr->fmt = fimc_find_format(NULL, NULL, FMT_FLAGS_CAM, 0);
-	fr->width = fr->f_width = fr->o_width = 640;
-	fr->height = fr->f_height = fr->o_height = 480;
+	fimc_cap_set_default_fmt(ctx);
 
 	vfd = video_device_alloc();
 	if (!vfd) {
@@ -757,6 +1309,9 @@ int fimc_register_capture_device(struct fimc_dev *fimc,
 	ret = media_entity_init(&vfd->entity, 1, &fimc->vid_cap.vd_pad, 0);
 	if (ret)
 		goto err_ent;
+	ret = fimc_create_capture_subdev(fimc, v4l2_dev);
+	if (ret)
+		goto err_sd_reg;
 
 	vfd->entity.ops = &fimc_media_ops;
 	vfd->ctrl_handler = &ctx->ctrl_handler;
@@ -771,6 +1326,8 @@ int fimc_register_capture_device(struct fimc_dev *fimc,
 	return 0;
 
 err_vd_reg:
+	fimc_destroy_capture_subdev(fimc);
+err_sd_reg:
 	media_entity_cleanup(&vfd->entity);
 err_ent:
 	video_device_release(vfd);
@@ -787,6 +1344,7 @@ void fimc_unregister_capture_device(struct fimc_dev *fimc)
 		media_entity_cleanup(&vfd->entity);
 		video_unregister_device(vfd);
 	}
+	fimc_destroy_capture_subdev(fimc);
 	kfree(fimc->vid_cap.ctx);
 	fimc->vid_cap.ctx = NULL;
 }
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index 7a10fea..4b9f471 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -371,6 +371,10 @@ void fimc_capture_irq_handler(struct fimc_dev *fimc)
 	} else {
 		set_bit(ST_CAPT_RUN, &fimc->state);
 	}
+	if (test_bit(ST_CAPT_APPLY_CFG, &fimc->state)) {
+		if (fimc_capture_config_update(cap->ctx) == 0)
+			clear_bit(ST_CAPT_APPLY_CFG, &fimc->state);
+	}
 
 	dbg("frame: %d, active_buf_cnt: %d",
 	    fimc_hw_get_frame_index(fimc), cap->active_buf_cnt);
@@ -1195,12 +1199,11 @@ static int fimc_m2m_g_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 	return 0;
 }
 
-int fimc_try_crop(struct fimc_ctx *ctx, struct v4l2_crop *cr)
+static int fimc_m2m_try_crop(struct fimc_ctx *ctx, struct v4l2_crop *cr)
 {
 	struct fimc_dev *fimc = ctx->fimc_dev;
 	struct fimc_frame *f;
 	u32 min_size, halign, depth = 0;
-	bool is_capture_ctx;
 	int i;
 
 	if (cr->c.top < 0 || cr->c.left < 0) {
@@ -1208,13 +1211,9 @@ int fimc_try_crop(struct fimc_ctx *ctx, struct v4l2_crop *cr)
 			"doesn't support negative values for top & left\n");
 		return -EINVAL;
 	}
-
-	is_capture_ctx = fimc_ctx_state_is_set(FIMC_CTX_CAP, ctx);
-
 	if (cr->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
-		f = is_capture_ctx ? &ctx->s_frame : &ctx->d_frame;
-	else if (cr->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE &&
-		 !is_capture_ctx)
+		f = &ctx->d_frame;
+	else if (cr->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
 		f = &ctx->s_frame;
 	else
 		return -EINVAL;
@@ -1223,15 +1222,10 @@ int fimc_try_crop(struct fimc_ctx *ctx, struct v4l2_crop *cr)
 		fimc->variant->min_inp_pixsize : fimc->variant->min_out_pixsize;
 
 	/* Get pixel alignment constraints. */
-	if (is_capture_ctx) {
-		min_size = 16;
-		halign = 4;
-	} else {
-		if (fimc->id == 1 && fimc->variant->pix_hoff)
-			halign = fimc_fmt_is_rgb(f->fmt->color) ? 0 : 1;
-		else
-			halign = ffs(min_size) - 1;
-	}
+	if (fimc->id == 1 && fimc->variant->pix_hoff)
+		halign = fimc_fmt_is_rgb(f->fmt->color) ? 0 : 1;
+	else
+		halign = ffs(min_size) - 1;
 
 	for (i = 0; i < f->fmt->colplanes; i++)
 		depth += f->fmt->depth[i];
@@ -1248,7 +1242,7 @@ int fimc_try_crop(struct fimc_ctx *ctx, struct v4l2_crop *cr)
 		cr->c.top = f->o_height - cr->c.height;
 
 	cr->c.left = round_down(cr->c.left, min_size);
-	cr->c.top  = round_down(cr->c.top, is_capture_ctx ? 16 : 8);
+	cr->c.top  = round_down(cr->c.top, fimc->variant->hor_offs_align);
 
 	dbg("l:%d, t:%d, w:%d, h:%d, f_w: %d, f_h: %d",
 	    cr->c.left, cr->c.top, cr->c.width, cr->c.height,
@@ -1264,7 +1258,7 @@ static int fimc_m2m_s_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 	struct fimc_frame *f;
 	int ret;
 
-	ret = fimc_try_crop(ctx, cr);
+	ret = fimc_m2m_try_crop(ctx, cr);
 	if (ret)
 		return ret;
 
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 36752f7..385a18a 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -93,9 +93,11 @@ enum fimc_color_fmt {
 	S5P_FIMC_CBYCRY422,
 	S5P_FIMC_CRYCBY422,
 	S5P_FIMC_YCBCR444_LOCAL,
+	S5P_FIMC_JPEG = 0x40,
 };
 
-#define fimc_fmt_is_rgb(x) ((x) & 0x10)
+#define fimc_fmt_is_rgb(x) (!!((x) & 0x10))
+#define fimc_fmt_is_jpeg(x) (!!((x) & 0x40))
 
 #define IS_M2M(__strt) ((__strt) == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE || \
 			__strt == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
@@ -117,9 +119,11 @@ enum fimc_color_fmt {
 #define	FIMC_DST_ADDR		(1 << 2)
 #define	FIMC_SRC_FMT		(1 << 3)
 #define	FIMC_DST_FMT		(1 << 4)
-#define	FIMC_CTX_M2M		(1 << 5)
-#define	FIMC_CTX_CAP		(1 << 6)
-#define	FIMC_CTX_SHUT		(1 << 7)
+#define	FIMC_DST_CROP		(1 << 5)
+#define	FIMC_CTX_M2M		(1 << 16)
+#define	FIMC_CTX_CAP		(1 << 17)
+#define	FIMC_CTX_SHUT		(1 << 18)
+
 
 /* Image conversion flags */
 #define	FIMC_IN_DMA_ACCESS_TILED	(1 << 0)
@@ -294,12 +298,18 @@ struct fimc_m2m_device {
 	int			refcnt;
 };
 
+#define FIMC_SD_PAD_SINK	0
+#define FIMC_SD_PAD_SOURCE	1
+#define FIMC_SD_PADS_NUM	2
+
 /**
  * struct fimc_vid_cap - camera capture device information
  * @ctx: hardware context data
  * @vfd: video device node for camera capture mode
+ * @subdev: subdev exposing the FIMC processing block
  * @vd_pad: fimc video capture node pad
- * @fmt: Media Bus format configured at selected image sensor
+ * @sd_pads: fimc video processing block pads
+ * @mf: media bus format at the FIMC camera input (and the scaler output) pad
  * @pending_buf_q: the pending buffer queue head
  * @active_buf_q: the queue head of buffers scheduled in hardware
  * @vbq: the capture am video buffer queue
@@ -316,8 +326,10 @@ struct fimc_vid_cap {
 	struct fimc_ctx			*ctx;
 	struct vb2_alloc_ctx		*alloc_ctx;
 	struct video_device		*vfd;
+	struct v4l2_subdev		*subdev;
 	struct media_pad		vd_pad;
 	struct v4l2_mbus_framefmt	mf;
+	struct media_pad		sd_pads[FIMC_SD_PADS_NUM];
 	struct list_head		pending_buf_q;
 	struct list_head		active_buf_q;
 	struct vb2_queue		vbq;
@@ -499,6 +511,33 @@ struct fimc_ctx {
 
 #define fh_to_ctx(__fh) container_of(__fh, struct fimc_ctx, fh)
 
+static inline void set_frame_bounds(struct fimc_frame *f, u32 width, u32 height)
+{
+	f->o_width  = width;
+	f->o_height = height;
+	f->f_width  = width;
+	f->f_height = height;
+}
+
+static inline void set_frame_crop(struct fimc_frame *f,
+				  u32 left, u32 top, u32 width, u32 height)
+{
+	f->offs_h = left;
+	f->offs_v = top;
+	f->width  = width;
+	f->height = height;
+}
+
+static inline u32 fimc_get_format_depth(struct fimc_fmt *ff)
+{
+	u32 i, depth = 0;
+
+	if (ff != NULL)
+		for (i = 0; i < ff->colplanes; i++)
+			depth += ff->depth[i];
+	return depth;
+}
+
 static inline bool fimc_capture_active(struct fimc_dev *fimc)
 {
 	unsigned long flags;
@@ -650,7 +689,6 @@ int fimc_hw_set_camera_type(struct fimc_dev *fimc,
 /* fimc-core.c */
 int fimc_vidioc_enum_fmt_mplane(struct file *file, void *priv,
 				struct v4l2_fmtdesc *f);
-int fimc_try_crop(struct fimc_ctx *ctx, struct v4l2_crop *cr);
 int fimc_ctrls_create(struct fimc_ctx *ctx);
 void fimc_ctrls_delete(struct fimc_ctx *ctx);
 void fimc_ctrls_activate(struct fimc_ctx *ctx, bool active);
@@ -685,6 +723,7 @@ int fimc_vid_cap_buf_queue(struct fimc_dev *fimc,
 			     struct fimc_vid_buffer *fimc_vb);
 int fimc_capture_suspend(struct fimc_dev *fimc);
 int fimc_capture_resume(struct fimc_dev *fimc);
+int fimc_capture_config_update(struct fimc_ctx *ctx);
 
 /* Locking: the caller holds fimc->slock */
 static inline void fimc_activate_capture(struct fimc_ctx *ctx)
diff --git a/drivers/media/video/s5p-fimc/fimc-mdevice.c b/drivers/media/video/s5p-fimc/fimc-mdevice.c
index 91f57d6..d425b65 100644
--- a/drivers/media/video/s5p-fimc/fimc-mdevice.c
+++ b/drivers/media/video/s5p-fimc/fimc-mdevice.c
@@ -408,8 +408,9 @@ static int __fimc_md_create_fimc_links(struct fimc_md *fmd,
 			continue;
 
 		flags = (i == fimc_id) ? MEDIA_LNK_FL_ENABLED : 0;
-		sink = &fmd->fimc[i]->vid_cap.vfd->entity;
-		ret = media_entity_create_link(source, 0, sink, 0, flags);
+		sink = &fmd->fimc[i]->vid_cap.subdev->entity;
+		ret = media_entity_create_link(source, pad, sink,
+					      FIMC_SD_PAD_SINK, flags);
 		if (ret)
 			return ret;
 
@@ -446,10 +447,10 @@ static int fimc_md_create_links(struct fimc_md *fmd)
 	struct v4l2_subdev *sensor, *csis;
 	struct s5p_fimc_isp_info *pdata;
 	struct fimc_sensor_info *s_info;
-	struct media_entity *source;
-	int fimc_id = 0;
-	int i, pad;
+	struct media_entity *source, *sink;
+	int i, pad, fimc_id = 0;
 	int ret = 0;
+	u32 flags;
 
 	for (i = 0; i < fmd->num_sensors; i++) {
 		if (fmd->sensor[i].subdev == NULL)
@@ -505,6 +506,24 @@ static int fimc_md_create_links(struct fimc_md *fmd)
 		ret = __fimc_md_create_fimc_links(fmd, source, sensor, pad,
 						 fimc_id++);
 	}
+	/* Create immutable links between each FIMC's subdev and video node */
+	flags = MEDIA_LNK_FL_IMMUTABLE | MEDIA_LNK_FL_ENABLED;
+	for (i = 0; i < FIMC_MAX_DEVS; i++) {
+		if (!fmd->fimc[i])
+			continue;
+		source = &fmd->fimc[i]->vid_cap.subdev->entity;
+		sink = &fmd->fimc[i]->vid_cap.vfd->entity;
+		ret = media_entity_create_link(source, FIMC_SD_PAD_SOURCE,
+					      sink, 0, flags);
+		if (ret)
+			break;
+		/* Notify FIMC capture video node entity */
+		ret = media_entity_call(sink, link_setup, &sink->pads[0],
+				&source->pads[FIMC_SD_PAD_SOURCE], flags);
+		if (ret)
+			break;
+	}
+
 	return ret;
 }
 
@@ -613,15 +632,15 @@ int fimc_md_set_camclk(struct v4l2_subdev *sd, bool on)
 static int fimc_md_link_notify(struct media_pad *source,
 			       struct media_pad *sink, u32 flags)
 {
-	struct video_device *vid_dev;
+	struct v4l2_subdev *sd;
 	struct fimc_dev *fimc;
 	int ret = 0;
 
-	if (WARN_ON(media_entity_type(sink->entity) != MEDIA_ENT_T_DEVNODE))
+	if (media_entity_type(sink->entity) != MEDIA_ENT_T_V4L2_SUBDEV)
 		return 0;
 
-	vid_dev = media_entity_to_video_device(sink->entity);
-	fimc = video_get_drvdata(vid_dev);
+	sd = media_entity_to_v4l2_subdev(sink->entity);
+	fimc = v4l2_get_subdevdata(sd);
 
 	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
 		ret = __fimc_pipeline_shutdown(fimc);
-- 
1.7.5.4

[PATCH v3 16/19] s5p-fimc: Add support for camera capture in JPEG format

[Sylwester Nawrocki]

Add support for transparent DMA transfer of JPEG data with MIPI-CSI2
USER1 format. In JPEG mode the color effect, scaling and cropping
is not supported. Same applies to image rotation and flip thus those
controls are marked as inactive when V4L2_PIX_FMT_JPEG pixelformat
is selected.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |   17 ++++++-
 drivers/media/video/s5p-fimc/fimc-core.c    |   70 ++++++++++++++-------------
 drivers/media/video/s5p-fimc/fimc-core.h    |    7 ++-
 drivers/media/video/s5p-fimc/fimc-reg.c     |   31 ++++++++----
 drivers/media/video/s5p-fimc/regs-fimc.h    |    8 +--
 5 files changed, 81 insertions(+), 52 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index 4d7b279..2bfc050 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -64,7 +64,7 @@ static int fimc_start_capture(struct fimc_dev *fimc)
 		fimc_hw_set_mainscaler(ctx);
 		fimc_hw_set_target_format(ctx);
 		fimc_hw_set_rotation(ctx);
-		fimc_hw_set_effect(ctx);
+		fimc_hw_set_effect(ctx, false);
 		fimc_hw_set_output_path(ctx);
 		fimc_hw_set_out_dma(ctx);
 		clear_bit(ST_CAPT_APPLY_CFG, &fimc->state);
@@ -696,6 +696,16 @@ static int fimc_cap_try_fmt_mplane(struct file *file, void *fh,
 	return 0;
 }
 
+static void fimc_capture_mark_jpeg_xfer(struct fimc_ctx *ctx, bool jpeg)
+{
+	ctx->scaler.enabled = !jpeg;
+	fimc_ctrls_activate(ctx, !jpeg);
+
+	if (jpeg)
+		set_bit(ST_CAPT_JPEG, &ctx->fimc_dev->state);
+	else
+		clear_bit(ST_CAPT_JPEG, &ctx->fimc_dev->state);
+}
 
 static int fimc_cap_s_fmt_mplane(struct file *file, void *priv,
 				 struct v4l2_format *f)
@@ -751,6 +761,9 @@ static int fimc_cap_s_fmt_mplane(struct file *file, void *priv,
 	if (fimc->vid_cap.vid_dev_compat)
 		set_frame_crop(ff, 0, 0, mf->width, mf->height);
 
+	/* Switch to the transparent DMA transfer mode if required */
+	fimc_capture_mark_jpeg_xfer(ctx, fimc_fmt_is_jpeg(ff->fmt->color));
+
 	return ret;
 }
 
@@ -1086,6 +1099,8 @@ static int fimc_subdev_set_fmt(struct v4l2_subdev *sd,
 		*mf = fmt->format;
 		return 0;
 	}
+	fimc_capture_mark_jpeg_xfer(ctx, fimc_fmt_is_jpeg(ffmt->color));
+
 	ff = fmt->pad == FIMC_SD_PAD_SINK ?
 		&ctx->s_frame : &ctx->d_frame;
 
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index 4b9f471..9538ab7 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -159,22 +159,28 @@ static struct fimc_fmt fimc_formats[] = {
 		.memplanes	= 2,
 		.colplanes	= 2,
 		.flags		= FMT_FLAGS_M2M,
+	}, {
+		.name		= "JPEG encoded data",
+		.fourcc		= V4L2_PIX_FMT_JPEG,
+		.color		= S5P_FIMC_JPEG,
+		.depth		= { 8 },
+		.memplanes	= 1,
+		.colplanes	= 1,
+		.mbus_code	= V4L2_MBUS_FMT_JPEG_1X8,
+		.flags		= FMT_FLAGS_CAM,
 	},
 };
 
-int fimc_check_scaler_ratio(int sw, int sh, int dw, int dh, int rot)
+int fimc_check_scaler_ratio(struct fimc_ctx *ctx, int sw, int sh,
+			    int dw, int dh, int rotation)
 {
-	int tx, ty;
+	if (rotation == 90 || rotation == 270)
+		swap(dw, dh);
 
-	if (rot == 90 || rot == 270) {
-		ty = dw;
-		tx = dh;
-	} else {
-		tx = dw;
-		ty = dh;
-	}
+	if (!ctx->scaler.enabled)
+		return (sw == dw && sh == dh) ? 0 : -EINVAL;
 
-	if ((sw >= SCALER_MAX_HRATIO * tx) || (sh >= SCALER_MAX_VRATIO * ty))
+	if ((sw >= SCALER_MAX_HRATIO * dw) || (sh >= SCALER_MAX_VRATIO * dh))
 		return -EINVAL;
 
 	return 0;
@@ -321,7 +327,7 @@ static int stop_streaming(struct vb2_queue *q)
 	return 0;
 }
 
-void fimc_capture_irq_handler(struct fimc_dev *fimc)
+void fimc_capture_irq_handler(struct fimc_dev *fimc, bool rel_buf)
 {
 	struct fimc_vid_cap *cap = &fimc->vid_cap;
 	struct fimc_vid_buffer *v_buf;
@@ -329,7 +335,7 @@ void fimc_capture_irq_handler(struct fimc_dev *fimc)
 	struct timespec ts;
 
 	if (!list_empty(&cap->active_buf_q) &&
-	    test_bit(ST_CAPT_RUN, &fimc->state)) {
+	    test_bit(ST_CAPT_RUN, &fimc->state) && rel_buf) {
 		ktime_get_real_ts(&ts);
 
 		v_buf = active_queue_pop(cap);
@@ -364,7 +370,8 @@ void fimc_capture_irq_handler(struct fimc_dev *fimc)
 	}
 
 	if (cap->active_buf_cnt == 0) {
-		clear_bit(ST_CAPT_RUN, &fimc->state);
+		if (rel_buf)
+			clear_bit(ST_CAPT_RUN, &fimc->state);
 
 		if (++cap->buf_index >= FIMC_MAX_OUT_BUFS)
 			cap->buf_index = 0;
@@ -409,14 +416,12 @@ static irqreturn_t fimc_irq_handler(int irq, void *priv)
 			spin_unlock(&ctx->slock);
 		}
 		return IRQ_HANDLED;
-	} else {
-		if (test_bit(ST_CAPT_PEND, &fimc->state)) {
-			fimc_capture_irq_handler(fimc);
-
-			if (cap->active_buf_cnt == 1) {
-				fimc_deactivate_capture(fimc);
-				clear_bit(ST_CAPT_STREAM, &fimc->state);
-			}
+	} else if (test_bit(ST_CAPT_PEND, &fimc->state)) {
+		fimc_capture_irq_handler(fimc,
+				 !test_bit(ST_CAPT_JPEG, &fimc->state));
+		if (cap->active_buf_cnt == 1) {
+			fimc_deactivate_capture(fimc);
+			clear_bit(ST_CAPT_STREAM, &fimc->state);
 		}
 	}
 out:
@@ -588,9 +593,6 @@ int fimc_prepare_config(struct fimc_ctx *ctx, u32 flags)
 		fimc_set_yuv_order(ctx);
 	}
 
-	/* Input DMA mode is not allowed when the scaler is disabled. */
-	ctx->scaler.enabled = 1;
-
 	if (flags & FIMC_SRC_ADDR) {
 		vb = v4l2_m2m_next_src_buf(ctx->m2m_ctx);
 		ret = fimc_prepare_addr(ctx, vb, s_frame, &s_frame->paddr);
@@ -645,7 +647,7 @@ static void fimc_dma_run(void *priv)
 		fimc_hw_set_mainscaler(ctx);
 		fimc_hw_set_target_format(ctx);
 		fimc_hw_set_rotation(ctx);
-		fimc_hw_set_effect(ctx);
+		fimc_hw_set_effect(ctx, false);
 	}
 
 	fimc_hw_set_output_path(ctx);
@@ -775,7 +777,7 @@ static int fimc_s_ctrl(struct v4l2_ctrl *ctrl)
 	case V4L2_CID_ROTATE:
 		if (fimc_capture_pending(fimc) ||
 		    fimc_ctx_state_is_set(FIMC_DST_FMT | FIMC_SRC_FMT, ctx)) {
-			ret = fimc_check_scaler_ratio(ctx->s_frame.width,
+			ret = fimc_check_scaler_ratio(ctx, ctx->s_frame.width,
 					ctx->s_frame.height, ctx->d_frame.width,
 					ctx->d_frame.height, ctrl->val);
 		}
@@ -1097,6 +1099,8 @@ static int fimc_m2m_s_fmt_mplane(struct file *file, void *fh,
 
 	fimc_fill_frame(frame, f);
 
+	ctx->scaler.enabled = 1;
+
 	if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
 		fimc_ctx_state_lock_set(FIMC_PARAMS | FIMC_DST_FMT, ctx);
 	else
@@ -1268,15 +1272,13 @@ static int fimc_m2m_s_crop(struct file *file, void *fh, struct v4l2_crop *cr)
 	/* Check to see if scaling ratio is within supported range */
 	if (fimc_ctx_state_is_set(FIMC_DST_FMT | FIMC_SRC_FMT, ctx)) {
 		if (cr->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
-			ret = fimc_check_scaler_ratio(cr->c.width, cr->c.height,
-						      ctx->d_frame.width,
-						      ctx->d_frame.height,
-						      ctx->rotation);
+			ret = fimc_check_scaler_ratio(ctx, cr->c.width,
+					cr->c.height, ctx->d_frame.width,
+					ctx->d_frame.height, ctx->rotation);
 		} else {
-			ret = fimc_check_scaler_ratio(ctx->s_frame.width,
-						      ctx->s_frame.height,
-						      cr->c.width, cr->c.height,
-						      ctx->rotation);
+			ret = fimc_check_scaler_ratio(ctx, ctx->s_frame.width,
+					ctx->s_frame.height, cr->c.width,
+					cr->c.height, ctx->rotation);
 		}
 		if (ret) {
 			v4l2_err(fimc->m2m.vfd, "Out of scaler range\n");
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 385a18a..a0d13e4 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -66,6 +66,7 @@ enum fimc_dev_flags {
 	ST_CAPT_SHUT,
 	ST_CAPT_INUSE,
 	ST_CAPT_APPLY_CFG,
+	ST_CAPT_JPEG,
 };
 
 #define fimc_m2m_active(dev) test_bit(ST_M2M_RUN, &(dev)->state)
@@ -670,7 +671,7 @@ void fimc_hw_en_irq(struct fimc_dev *fimc, int enable);
 void fimc_hw_set_prescaler(struct fimc_ctx *ctx);
 void fimc_hw_set_mainscaler(struct fimc_ctx *ctx);
 void fimc_hw_en_capture(struct fimc_ctx *ctx);
-void fimc_hw_set_effect(struct fimc_ctx *ctx);
+void fimc_hw_set_effect(struct fimc_ctx *ctx, bool active);
 void fimc_hw_set_in_dma(struct fimc_ctx *ctx);
 void fimc_hw_set_input_path(struct fimc_ctx *ctx);
 void fimc_hw_set_output_path(struct fimc_ctx *ctx);
@@ -698,7 +699,8 @@ void fimc_adjust_mplane_format(struct fimc_fmt *fmt, u32 width, u32 height,
 struct fimc_fmt *fimc_find_format(u32 *pixelformat, u32 *mbus_code,
 				  unsigned int mask, int index);
 
-int fimc_check_scaler_ratio(int sw, int sh, int dw, int dh, int rot);
+int fimc_check_scaler_ratio(struct fimc_ctx *ctx, int sw, int sh,
+			    int dw, int dh, int rotation);
 int fimc_set_scaler_info(struct fimc_ctx *ctx);
 int fimc_prepare_config(struct fimc_ctx *ctx, u32 flags);
 int fimc_prepare_addr(struct fimc_ctx *ctx, struct vb2_buffer *vb,
@@ -706,6 +708,7 @@ int fimc_prepare_addr(struct fimc_ctx *ctx, struct vb2_buffer *vb,
 void fimc_prepare_dma_offset(struct fimc_ctx *ctx, struct fimc_frame *f);
 void fimc_set_yuv_order(struct fimc_ctx *ctx);
 void fimc_fill_frame(struct fimc_frame *frame, struct v4l2_format *f);
+void fimc_capture_irq_handler(struct fimc_dev *fimc, bool done);
 
 int fimc_register_m2m_device(struct fimc_dev *fimc,
 			     struct v4l2_device *v4l2_dev);
diff --git a/drivers/media/video/s5p-fimc/fimc-reg.c b/drivers/media/video/s5p-fimc/fimc-reg.c
index a1fff02..2a1ae51 100644
--- a/drivers/media/video/s5p-fimc/fimc-reg.c
+++ b/drivers/media/video/s5p-fimc/fimc-reg.c
@@ -352,17 +352,19 @@ void fimc_hw_en_capture(struct fimc_ctx *ctx)
 	writel(cfg | S5P_CIIMGCPT_IMGCPTEN, dev->regs + S5P_CIIMGCPT);
 }
 
-void fimc_hw_set_effect(struct fimc_ctx *ctx)
+void fimc_hw_set_effect(struct fimc_ctx *ctx, bool active)
 {
 	struct fimc_dev *dev = ctx->fimc_dev;
 	struct fimc_effect *effect = &ctx->effect;
-	u32 cfg = (S5P_CIIMGEFF_IE_ENABLE | S5P_CIIMGEFF_IE_SC_AFTER);
-
-	cfg |= effect->type;
+	u32 cfg = 0;
 
-	if (effect->type == S5P_FIMC_EFFECT_ARBITRARY) {
-		cfg |= S5P_CIIMGEFF_PAT_CB(effect->pat_cb);
-		cfg |= S5P_CIIMGEFF_PAT_CR(effect->pat_cr);
+	if (active) {
+		cfg |= S5P_CIIMGEFF_IE_SC_AFTER | S5P_CIIMGEFF_IE_ENABLE;
+		cfg |= effect->type;
+		if (effect->type == S5P_FIMC_EFFECT_ARBITRARY) {
+			cfg |= S5P_CIIMGEFF_PAT_CB(effect->pat_cb);
+			cfg |= S5P_CIIMGEFF_PAT_CR(effect->pat_cr);
+		}
 	}
 
 	writel(cfg, dev->regs + S5P_CIIMGEFF);
@@ -592,6 +594,9 @@ int fimc_hw_set_camera_source(struct fimc_dev *fimc,
 			else if (bus_width == 16)
 				cfg |= S5P_CISRCFMT_ITU601_16BIT;
 		} /* else defaults to ITU-R BT.656 8-bit */
+	} else if (cam->bus_type == FIMC_MIPI_CSI2) {
+		if (fimc_fmt_is_jpeg(f->fmt->color))
+			cfg |= S5P_CISRCFMT_ITU601_8BIT;
 	}
 
 	cfg |= S5P_CISRCFMT_HSIZE(f->o_width) | S5P_CISRCFMT_VSIZE(f->o_height);
@@ -633,7 +638,7 @@ int fimc_hw_set_camera_type(struct fimc_dev *fimc,
 	/* Select ITU B interface, disable Writeback path and test pattern. */
 	cfg &= ~(S5P_CIGCTRL_TESTPAT_MASK | S5P_CIGCTRL_SELCAM_ITU_A |
 		S5P_CIGCTRL_SELCAM_MIPI | S5P_CIGCTRL_CAMIF_SELWB |
-		S5P_CIGCTRL_SELCAM_MIPI_A);
+		S5P_CIGCTRL_SELCAM_MIPI_A | S5P_CIGCTRL_CAM_JPEG);
 
 	if (cam->bus_type == FIMC_MIPI_CSI2) {
 		cfg |= S5P_CIGCTRL_SELCAM_MIPI;
@@ -642,9 +647,15 @@ int fimc_hw_set_camera_type(struct fimc_dev *fimc,
 			cfg |= S5P_CIGCTRL_SELCAM_MIPI_A;
 
 		/* TODO: add remaining supported formats. */
-		if (vid_cap->mf.code == V4L2_MBUS_FMT_VYUY8_2X8) {
+		switch (vid_cap->mf.code) {
+		case V4L2_MBUS_FMT_VYUY8_2X8:
 			tmp = S5P_CSIIMGFMT_YCBCR422_8BIT;
-		} else {
+			break;
+		case V4L2_MBUS_FMT_JPEG_1X8:
+			tmp = S5P_CSIIMGFMT_USER(1);
+			cfg |= S5P_CIGCTRL_CAM_JPEG;
+			break;
+		default:
 			v4l2_err(fimc->vid_cap.vfd,
 				 "Not supported camera pixel format: %d",
 				 vid_cap->mf.code);
diff --git a/drivers/media/video/s5p-fimc/regs-fimc.h b/drivers/media/video/s5p-fimc/regs-fimc.h
index 0fea3e6..94d2302 100644
--- a/drivers/media/video/s5p-fimc/regs-fimc.h
+++ b/drivers/media/video/s5p-fimc/regs-fimc.h
@@ -54,6 +54,7 @@
 #define S5P_CIGCTRL_IRQ_CLR		(1 << 19)
 #define S5P_CIGCTRL_IRQ_ENABLE		(1 << 16)
 #define S5P_CIGCTRL_SHDW_DISABLE	(1 << 12)
+#define S5P_CIGCTRL_CAM_JPEG		(1 << 8)
 #define S5P_CIGCTRL_SELCAM_MIPI_A	(1 << 7)
 #define S5P_CIGCTRL_CAMIF_SELWB		(1 << 6)
 /* 0 - ITU601; 1 - ITU709 */
@@ -184,7 +185,6 @@
 
 /* Image effect */
 #define S5P_CIIMGEFF			0xd0
-#define S5P_CIIMGEFF_IE_DISABLE		(0 << 30)
 #define S5P_CIIMGEFF_IE_ENABLE		(1 << 30)
 #define S5P_CIIMGEFF_IE_SC_BEFORE	(0 << 29)
 #define S5P_CIIMGEFF_IE_SC_AFTER	(1 << 29)
@@ -286,10 +286,8 @@
 #define S5P_CSIIMGFMT_RAW8		0x2a
 #define S5P_CSIIMGFMT_RAW10		0x2b
 #define S5P_CSIIMGFMT_RAW12		0x2c
-#define S5P_CSIIMGFMT_USER1		0x30
-#define S5P_CSIIMGFMT_USER2		0x31
-#define S5P_CSIIMGFMT_USER3		0x32
-#define S5P_CSIIMGFMT_USER4		0x33
+/* User defined formats. x = 0...16. */
+#define S5P_CSIIMGFMT_USER(x)		(0x30 + x - 1)
 
 /* Output frame buffer sequence mask */
 #define S5P_CIFCNTSEQ			0x1FC
-- 
1.7.5.4

[PATCH v3 17/19] s5p-fimc: Add v4l2_device notification support for single frame capture

[Sylwester Nawrocki]

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |   47 +++++++++++++++++++++++++++
 drivers/media/video/s5p-fimc/fimc-core.h    |    2 +
 drivers/media/video/s5p-fimc/fimc-mdevice.c |    1 +
 include/media/s5p_fimc.h                    |    9 +++++
 4 files changed, 59 insertions(+), 0 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index 2bfc050..c9420db 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -1029,6 +1029,53 @@ static const struct media_entity_operations fimc_media_ops = {
 	.link_setup = fimc_link_setup,
 };
 
+/**
+ * fimc_sensor_notify - v4l2_device notification from a sensor subdev
+ * @sd: pointer to a subdev generating the notification
+ * @notification: the notification type, must be S5P_FIMC_TX_END_NOTIFY
+ * @arg: pointer to an u32 type integer that stores the frame payload value
+ *
+ * The End Of Frame notification sent by sensor subdev in its still capture
+ * mode. If there is only a single VSYNC generated by the sensor at the
+ * beginning of a frame transmission, FIMC does not issue the LastIrq
+ * (end of frame) interrupt. And this notification is used to complete the
+ * frame capture and returning a buffer to user-space. Subdev drivers should
+ * call this notification from their last 'End of frame capture' interrupt.
+ */
+void fimc_sensor_notify(struct v4l2_subdev *sd, unsigned int notification,
+			void *arg)
+{
+	struct fimc_sensor_info	*sensor;
+	struct fimc_vid_buffer *buf;
+	struct fimc_md *fmd;
+	struct fimc_dev *fimc;
+	unsigned long flags;
+
+	if (sd == NULL)
+		return;
+
+	sensor = v4l2_get_subdev_hostdata(sd);
+	fmd = entity_to_fimc_mdev(&sd->entity);
+
+	spin_lock_irqsave(&fmd->slock, flags);
+	fimc = sensor ? sensor->host : NULL;
+
+	if (fimc && arg && notification == S5P_FIMC_TX_END_NOTIFY &&
+	    test_bit(ST_CAPT_PEND, &fimc->state)) {
+		unsigned long irq_flags;
+		spin_lock_irqsave(&fimc->slock, irq_flags);
+		if (!list_empty(&fimc->vid_cap.active_buf_q)) {
+			buf = list_entry(fimc->vid_cap.active_buf_q.next,
+					 struct fimc_vid_buffer, list);
+			vb2_set_plane_payload(&buf->vb, 0, *((u32 *)arg));
+		}
+		fimc_capture_irq_handler(fimc, true);
+		fimc_deactivate_capture(fimc);
+		spin_unlock_irqrestore(&fimc->slock, irq_flags);
+	}
+	spin_unlock_irqrestore(&fmd->slock, flags);
+}
+
 static int fimc_subdev_enum_mbus_code(struct v4l2_subdev *sd,
 				      struct v4l2_subdev_fh *fh,
 				      struct v4l2_subdev_mbus_code_enum *code)
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index a0d13e4..3759250 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -724,6 +724,8 @@ void fimc_unregister_capture_device(struct fimc_dev *fimc);
 int fimc_capture_ctrls_create(struct fimc_dev *fimc);
 int fimc_vid_cap_buf_queue(struct fimc_dev *fimc,
 			     struct fimc_vid_buffer *fimc_vb);
+void fimc_sensor_notify(struct v4l2_subdev *sd, unsigned int notification,
+			void *arg);
 int fimc_capture_suspend(struct fimc_dev *fimc);
 int fimc_capture_resume(struct fimc_dev *fimc);
 int fimc_capture_config_update(struct fimc_ctx *ctx);
diff --git a/drivers/media/video/s5p-fimc/fimc-mdevice.c b/drivers/media/video/s5p-fimc/fimc-mdevice.c
index d425b65..a050f91 100644
--- a/drivers/media/video/s5p-fimc/fimc-mdevice.c
+++ b/drivers/media/video/s5p-fimc/fimc-mdevice.c
@@ -738,6 +738,7 @@ static int __devinit fimc_md_probe(struct platform_device *pdev)
 
 	v4l2_dev = &fmd->v4l2_dev;
 	v4l2_dev->mdev = &fmd->media_dev;
+	v4l2_dev->notify = fimc_sensor_notify;
 	snprintf(v4l2_dev->name, sizeof(v4l2_dev->name), "%s",
 		 dev_name(&pdev->dev));
 
diff --git a/include/media/s5p_fimc.h b/include/media/s5p_fimc.h
index 086a7aa..2b58904 100644
--- a/include/media/s5p_fimc.h
+++ b/include/media/s5p_fimc.h
@@ -60,4 +60,13 @@ struct s5p_platform_fimc {
 	struct s5p_fimc_isp_info *isp_info;
 	int num_clients;
 };
+
+/*
+ * v4l2_device notification id. This is only for internal use in the kernel.
+ * Sensor subdevs should issue S5P_FIMC_TX_END_NOTIFY notification in single
+ * frame capture mode when there is only one VSYNC pulse issued by the sensor
+ * at begining of the frame transmission.
+ */
+#define S5P_FIMC_TX_END_NOTIFY _IO('e', 0)
+
 #endif /* S5P_FIMC_H_ */
-- 
1.7.5.4

[PATCH v3 18/19] s5p-fimc: Use consistent names for the buffer list functions

[Sylwester Nawrocki]

Also correct and improve *_queue_add/pop functions description.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |    6 ++--
 drivers/media/video/s5p-fimc/fimc-core.c    |    6 ++--
 drivers/media/video/s5p-fimc/fimc-core.h    |   38 +++++++++++++++++---------
 3 files changed, 31 insertions(+), 19 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index c9420db..3b64788 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -100,12 +100,12 @@ static int fimc_stop_capture(struct fimc_dev *fimc)
 
 	/* Release buffers that were enqueued in the driver by videobuf2. */
 	while (!list_empty(&cap->pending_buf_q)) {
-		buf = pending_queue_pop(cap);
+		buf = fimc_pending_queue_pop(cap);
 		vb2_buffer_done(&buf->vb, VB2_BUF_STATE_ERROR);
 	}
 
 	while (!list_empty(&cap->active_buf_q)) {
-		buf = active_queue_pop(cap);
+		buf = fimc_active_queue_pop(cap);
 		vb2_buffer_done(&buf->vb, VB2_BUF_STATE_ERROR);
 	}
 
@@ -256,7 +256,7 @@ static void buffer_queue(struct vb2_buffer *vb)
 
 		fimc_hw_set_output_addr(fimc, &buf->paddr, buf_id);
 		buf->index = vid_cap->buf_index;
-		active_queue_add(vid_cap, buf);
+		fimc_active_queue_add(vid_cap, buf);
 
 		if (++vid_cap->buf_index >= FIMC_MAX_OUT_BUFS)
 			vid_cap->buf_index = 0;
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index 9538ab7..66d1af5 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -338,7 +338,7 @@ void fimc_capture_irq_handler(struct fimc_dev *fimc, bool rel_buf)
 	    test_bit(ST_CAPT_RUN, &fimc->state) && rel_buf) {
 		ktime_get_real_ts(&ts);
 
-		v_buf = active_queue_pop(cap);
+		v_buf = fimc_active_queue_pop(cap);
 
 		tv = &v_buf->vb.v4l2_buf.timestamp;
 		tv->tv_sec = ts.tv_sec;
@@ -355,12 +355,12 @@ void fimc_capture_irq_handler(struct fimc_dev *fimc, bool rel_buf)
 
 	if (!list_empty(&cap->pending_buf_q)) {
 
-		v_buf = pending_queue_pop(cap);
+		v_buf = fimc_pending_queue_pop(cap);
 		fimc_hw_set_output_addr(fimc, &v_buf->paddr, cap->buf_index);
 		v_buf->index = cap->buf_index;
 
 		/* Move the buffer to the capture active queue */
-		active_queue_add(cap, v_buf);
+		fimc_active_queue_add(cap, v_buf);
 
 		dbg("next frame: %d, done frame: %d",
 		    fimc_hw_get_frame_index(fimc), v_buf->index);
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 3759250..879d5dd 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -746,22 +746,27 @@ static inline void fimc_deactivate_capture(struct fimc_dev *fimc)
 }
 
 /*
- * Add buf to the capture active buffers queue.
- * Locking: Need to be called with fimc_dev::slock held.
+ * Buffer list manipulation functions. Must be called with fimc.slock held.
  */
-static inline void active_queue_add(struct fimc_vid_cap *vid_cap,
-				    struct fimc_vid_buffer *buf)
+
+/**
+ * fimc_active_queue_add - add buffer to the capture active buffers queue
+ * @buf: buffer to add to the active buffers list
+ */
+static inline void fimc_active_queue_add(struct fimc_vid_cap *vid_cap,
+					 struct fimc_vid_buffer *buf)
 {
 	list_add_tail(&buf->list, &vid_cap->active_buf_q);
 	vid_cap->active_buf_cnt++;
 }
 
-/*
- * Pop a video buffer from the capture active buffers queue
- * Locking: Need to be called with fimc_dev::slock held.
+/**
+ * fimc_active_queue_pop - pop buffer from the capture active buffers queue
+ *
+ * The caller must assure the active_buf_q list is not empty.
  */
-static inline struct fimc_vid_buffer *
-active_queue_pop(struct fimc_vid_cap *vid_cap)
+static inline struct fimc_vid_buffer *fimc_active_queue_pop(
+				    struct fimc_vid_cap *vid_cap)
 {
 	struct fimc_vid_buffer *buf;
 	buf = list_entry(vid_cap->active_buf_q.next,
@@ -771,16 +776,23 @@ active_queue_pop(struct fimc_vid_cap *vid_cap)
 	return buf;
 }
 
-/* Add video buffer to the capture pending buffers queue */
+/**
+ * fimc_pending_queue_add - add buffer to the capture pending buffers queue
+ * @buf: buffer to add to the pending buffers list
+ */
 static inline void fimc_pending_queue_add(struct fimc_vid_cap *vid_cap,
 					  struct fimc_vid_buffer *buf)
 {
 	list_add_tail(&buf->list, &vid_cap->pending_buf_q);
 }
 
-/* Add video buffer to the capture pending buffers queue */
-static inline struct fimc_vid_buffer *
-pending_queue_pop(struct fimc_vid_cap *vid_cap)
+/**
+ * fimc_pending_queue_pop - pop buffer from the capture pending buffers queue
+ *
+ * The caller must assure the pending_buf_q list is not empty.
+ */
+static inline struct fimc_vid_buffer *fimc_pending_queue_pop(
+				     struct fimc_vid_cap *vid_cap)
 {
 	struct fimc_vid_buffer *buf;
 	buf = list_entry(vid_cap->pending_buf_q.next,
-- 
1.7.5.4

[PATCH v3 19/19] s5p-fimc: Add runtime PM support in the camera capture driver

[Sylwester Nawrocki]

Add support for whole pipeline suspend/resume. Sensors must support
suspend/resume through s_power subdev operation.

Signed-off-by: Sylwester Nawrocki <s.nawrocki@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>
---
 drivers/media/video/s5p-fimc/fimc-capture.c |   77 ++++++++++++++++++++-------
 drivers/media/video/s5p-fimc/fimc-core.c    |   10 ++--
 drivers/media/video/s5p-fimc/fimc-core.h    |    1 +
 3 files changed, 64 insertions(+), 24 deletions(-)

diff --git a/drivers/media/video/s5p-fimc/fimc-capture.c b/drivers/media/video/s5p-fimc/fimc-capture.c
index 3b64788..33776ff 100644
--- a/drivers/media/video/s5p-fimc/fimc-capture.c
+++ b/drivers/media/video/s5p-fimc/fimc-capture.c
@@ -73,7 +73,7 @@ static int fimc_start_capture(struct fimc_dev *fimc)
 	return ret;
 }
 
-static int fimc_stop_capture(struct fimc_dev *fimc)
+static int fimc_stop_capture(struct fimc_dev *fimc, bool suspend)
 {
 	struct fimc_vid_cap *cap = &fimc->vid_cap;
 	struct fimc_vid_buffer *buf;
@@ -89,26 +89,28 @@ static int fimc_stop_capture(struct fimc_dev *fimc)
 
 	wait_event_timeout(fimc->irq_queue,
 			   !test_bit(ST_CAPT_SHUT, &fimc->state),
-			   FIMC_SHUTDOWN_TIMEOUT);
+			   (2*HZ/10)); /* 200 ms */
 
 	spin_lock_irqsave(&fimc->slock, flags);
-	fimc->state &= ~(1 << ST_CAPT_RUN | 1 << ST_CAPT_PEND |
-			 1 << ST_CAPT_SHUT | 1 << ST_CAPT_STREAM |
-			 1 << ST_CAPT_ISP_STREAM);
+	fimc->state &= ~(1 << ST_CAPT_RUN | 1 << ST_CAPT_SHUT |
+			 1 << ST_CAPT_STREAM | 1 << ST_CAPT_ISP_STREAM);
+	if (!suspend)
+		fimc->state &= ~(1 << ST_CAPT_PEND | 1 << ST_CAPT_SUSPENDED);
 
-	fimc->vid_cap.active_buf_cnt = 0;
-
-	/* Release buffers that were enqueued in the driver by videobuf2. */
-	while (!list_empty(&cap->pending_buf_q)) {
+	/* Release unused buffers */
+	while (!suspend && !list_empty(&cap->pending_buf_q)) {
 		buf = fimc_pending_queue_pop(cap);
 		vb2_buffer_done(&buf->vb, VB2_BUF_STATE_ERROR);
 	}
-
+	/* If suspending put unused buffers onto pending queue */
 	while (!list_empty(&cap->active_buf_q)) {
 		buf = fimc_active_queue_pop(cap);
-		vb2_buffer_done(&buf->vb, VB2_BUF_STATE_ERROR);
+		if (suspend)
+			fimc_pending_queue_add(cap, buf);
+		else
+			vb2_buffer_done(&buf->vb, VB2_BUF_STATE_ERROR);
 	}
-
+	set_bit(ST_CAPT_SUSPENDED, &fimc->state);
 	spin_unlock_irqrestore(&fimc->slock, flags);
 	dbg("state: 0x%lx", fimc->state);
 
@@ -117,11 +119,39 @@ static int fimc_stop_capture(struct fimc_dev *fimc)
 
 int fimc_capture_suspend(struct fimc_dev *fimc)
 {
-	return -EBUSY;
+	bool suspend = fimc_capture_in_use(fimc);
+
+	int ret = fimc_stop_capture(fimc, suspend);
+	if (ret)
+		return ret;
+	return fimc_pipeline_shutdown(fimc);
 }
 
+static void buffer_queue(struct vb2_buffer *vb);
+
 int fimc_capture_resume(struct fimc_dev *fimc)
 {
+	struct fimc_vid_cap *vid_cap = &fimc->vid_cap;
+	struct fimc_vid_buffer *buf;
+	int i;
+
+	if (!test_and_clear_bit(ST_CAPT_SUSPENDED, &fimc->state))
+		return 0;
+
+	INIT_LIST_HEAD(&fimc->vid_cap.active_buf_q);
+	vid_cap->buf_index = 0;
+	fimc_pipeline_initialize(fimc, &fimc->vid_cap.vfd->entity,
+				 false);
+	fimc_start_capture(fimc);
+
+	clear_bit(ST_CAPT_SUSPENDED, &fimc->state);
+
+	for (i = 0; i < vid_cap->reqbufs_count; i++) {
+		if (list_empty(&vid_cap->pending_buf_q))
+			break;
+		buf = fimc_pending_queue_pop(vid_cap);
+		buffer_queue(&buf->vb);
+	}
 	return 0;
 }
 
@@ -181,7 +211,7 @@ static int stop_streaming(struct vb2_queue *q)
 	if (!fimc_capture_active(fimc))
 		return -EINVAL;
 
-	return fimc_stop_capture(fimc);
+	return fimc_stop_capture(fimc, false);
 }
 
 static unsigned int get_plane_size(struct fimc_frame *fr, unsigned int plane)
@@ -248,8 +278,9 @@ static void buffer_queue(struct vb2_buffer *vb)
 	spin_lock_irqsave(&fimc->slock, flags);
 	fimc_prepare_addr(ctx, &buf->vb, &ctx->d_frame, &buf->paddr);
 
-	if (!test_bit(ST_CAPT_STREAM, &fimc->state)
-	     && vid_cap->active_buf_cnt < FIMC_MAX_OUT_BUFS) {
+	if (!test_bit(ST_CAPT_SUSPENDED, &fimc->state) &&
+	    !test_bit(ST_CAPT_STREAM, &fimc->state) &&
+	    vid_cap->active_buf_cnt < FIMC_MAX_OUT_BUFS) {
 		/* Setup the buffer directly for processing. */
 		int buf_id = (vid_cap->reqbufs_count == 1) ? -1 :
 				vid_cap->buf_index;
@@ -337,6 +368,7 @@ static int fimc_capture_open(struct file *file)
 	if (fimc_m2m_active(fimc))
 		return -EBUSY;
 
+	set_bit(ST_CAPT_INUSE, &fimc->state);
 	pm_runtime_get_sync(&fimc->pdev->dev);
 
 	if (++fimc->vid_cap.refcnt == 1) {
@@ -348,6 +380,7 @@ static int fimc_capture_open(struct file *file)
 			pm_runtime_put_sync(&fimc->pdev->dev);
 			fimc->vid_cap.refcnt--;
 			v4l2_fh_release(file);
+			clear_bit(ST_CAPT_INUSE, &fimc->state);
 			return ret;
 		}
 		ret = fimc_capture_ctrls_create(fimc);
@@ -362,12 +395,18 @@ static int fimc_capture_close(struct file *file)
 	dbg("pid: %d, state: 0x%lx", task_pid_nr(current), fimc->state);
 
 	if (--fimc->vid_cap.refcnt == 0) {
-		fimc_stop_capture(fimc);
+		clear_bit(ST_CAPT_INUSE, &fimc->state);
+		fimc_stop_capture(fimc, false);
 		fimc_pipeline_shutdown(fimc);
-		fimc_ctrls_delete(fimc->vid_cap.ctx);
-		vb2_queue_release(&fimc->vid_cap.vbq);
+		clear_bit(ST_CAPT_SUSPENDED, &fimc->state);
 	}
+
 	pm_runtime_put_sync(&fimc->pdev->dev);
+
+	if (fimc->vid_cap.refcnt == 0) {
+		vb2_queue_release(&fimc->vid_cap.vbq);
+		fimc_ctrls_delete(fimc->vid_cap.ctx);
+	}
 	return v4l2_fh_release(file);
 }
 
diff --git a/drivers/media/video/s5p-fimc/fimc-core.c b/drivers/media/video/s5p-fimc/fimc-core.c
index 66d1af5..6044459 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.c
+++ b/drivers/media/video/s5p-fimc/fimc-core.c
@@ -334,6 +334,11 @@ void fimc_capture_irq_handler(struct fimc_dev *fimc, bool rel_buf)
 	struct timeval *tv;
 	struct timespec ts;
 
+	if (test_and_clear_bit(ST_CAPT_SHUT, &fimc->state)) {
+		wake_up(&fimc->irq_queue);
+		return;
+	}
+
 	if (!list_empty(&cap->active_buf_q) &&
 	    test_bit(ST_CAPT_RUN, &fimc->state) && rel_buf) {
 		ktime_get_real_ts(&ts);
@@ -348,11 +353,6 @@ void fimc_capture_irq_handler(struct fimc_dev *fimc, bool rel_buf)
 		vb2_buffer_done(&v_buf->vb, VB2_BUF_STATE_DONE);
 	}
 
-	if (test_and_clear_bit(ST_CAPT_SHUT, &fimc->state)) {
-		wake_up(&fimc->irq_queue);
-		return;
-	}
-
 	if (!list_empty(&cap->pending_buf_q)) {
 
 		v_buf = fimc_pending_queue_pop(cap);
diff --git a/drivers/media/video/s5p-fimc/fimc-core.h b/drivers/media/video/s5p-fimc/fimc-core.h
index 879d5dd..72c6799 100644
--- a/drivers/media/video/s5p-fimc/fimc-core.h
+++ b/drivers/media/video/s5p-fimc/fimc-core.h
@@ -63,6 +63,7 @@ enum fimc_dev_flags {
 	ST_CAPT_RUN,
 	ST_CAPT_STREAM,
 	ST_CAPT_ISP_STREAM,
+	ST_CAPT_SUSPENDED,
 	ST_CAPT_SHUT,
 	ST_CAPT_INUSE,
 	ST_CAPT_APPLY_CFG,
-- 
1.7.5.4

Re: [PATCH v3 08/19] s5p-fimc: Add the media device driver

[Subash Patel]

Hi Sylwester,

On 07/04/2011 11:24 PM, Sylwester Nawrocki wrote:
> The fimc media device driver is hooked onto "s5p-fimc-md" platform device.
> Such a platform device need to be added in a board initialization code
> and then camera sensors need to be specified as it's platform data.
> The "s5p-fimc-md" device is a top level entity for all FIMC, mipi-csis
> and sensor devices.
>
> Signed-off-by: Sylwester Nawrocki<s.nawrocki@samsung.com>
> Signed-off-by: Kyungmin Park<kyungmin.park@samsung.com>
> ---
>   drivers/media/video/Kconfig                 |    2 +-
>   drivers/media/video/s5p-fimc/Makefile       |    2 +-
...
>
>   /* -----------------------------------------------------*/
>   /* fimc-capture.c					*/
> diff --git a/drivers/media/video/s5p-fimc/fimc-mdevice.c b/drivers/media/video/s5p-fimc/fimc-mdevice.c
> new file mode 100644
> index 0000000..10c8d5d
> --- /dev/null
> +++ b/drivers/media/video/s5p-fimc/fimc-mdevice.c
> @@ -0,0 +1,804 @@
...
> +
> +static int fimc_md_register_sensor_entities(struct fimc_md *fmd)
> +{
> +	struct s5p_platform_fimc *pdata = fmd->pdev->dev.platform_data;
> +	struct fimc_dev *fd = NULL;
> +	int num_clients, ret, i;
> +
> +	/*
> +	 * Runtime resume one of the FIMC entities to make sure
> +	 * the sclk_cam clocks are not globally disabled.
> +	 */
> +	for (i = 0; !fd&&  i<  ARRAY_SIZE(fmd->fimc); i++)
> +		if (fmd->fimc[i])
> +			fd = fmd->fimc[i];
> +	if (!fd)
> +		return -ENXIO;
> +	ret = pm_runtime_get_sync(&fd->pdev->dev);
> +	if (ret<  0)
> +		return ret;
> +
> +	WARN_ON(pdata->num_clients>  ARRAY_SIZE(fmd->sensor));
> +	num_clients = min_t(u32, pdata->num_clients, ARRAY_SIZE(fmd->sensor));
> +
> +	fmd->num_sensors = num_clients;
> +	for (i = 0; i<  num_clients; i++) {
> +		fmd->sensor[i].pdata =&pdata->isp_info[i];
> +		ret = __fimc_md_set_camclk(fmd,&fmd->sensor[i], true);
> +		if (ret)
> +			break;
> +		fmd->sensor[i].subdev =
> +			fimc_md_register_sensor(fmd,&fmd->sensor[i]);

There is an issue here. Function fimc_md_register_sensor(), can return 
subdev, as also error codes when i2c_get_adapter() or NULL when 
v4l2_i2c_new_subdev_board() fail. But we do not invalidate, and assume 
the return value is valid subdev. It will cause kernel NULL pointer 
exception later in fimc_md_create_links().

> +		ret = __fimc_md_set_camclk(fmd,&fmd->sensor[i], false);
> +		if (ret)
> +			break;
> +	}
> +	pm_runtime_put(&fd->pdev->dev);
> +	return ret;
> +}
> +
> +/*
> + * MIPI CSIS and FIMC platform devices registration.
> + */

Regards,
Subash
SISO-SLG

Re: [PATCH v3 08/19] s5p-fimc: Add the media device driver

[Sylwester Nawrocki]

Hi Subash,

On 07/22/2011 08:26 AM, Subash Patel wrote:
> Hi Sylwester,
> 
> On 07/04/2011 11:24 PM, Sylwester Nawrocki wrote:
>> The fimc media device driver is hooked onto "s5p-fimc-md" platform device.
>> Such a platform device need to be added in a board initialization code
>> and then camera sensors need to be specified as it's platform data.
>> The "s5p-fimc-md" device is a top level entity for all FIMC, mipi-csis
>> and sensor devices.
>>
>> Signed-off-by: Sylwester Nawrocki<s.nawrocki@samsung.com>
>> Signed-off-by: Kyungmin Park<kyungmin.park@samsung.com>
>> ---
>> drivers/media/video/Kconfig | 2 +-
>> drivers/media/video/s5p-fimc/Makefile | 2 +-
> ...
>>
>> /* -----------------------------------------------------*/
>> /* fimc-capture.c */
>> diff --git a/drivers/media/video/s5p-fimc/fimc-mdevice.c b/drivers/media/video/s5p-fimc/fimc-mdevice.c
>> new file mode 100644
>> index 0000000..10c8d5d
>> --- /dev/null
>> +++ b/drivers/media/video/s5p-fimc/fimc-mdevice.c
>> @@ -0,0 +1,804 @@
> ...
>> +
>> +static int fimc_md_register_sensor_entities(struct fimc_md *fmd)
>> +{
>> + struct s5p_platform_fimc *pdata = fmd->pdev->dev.platform_data;
>> + struct fimc_dev *fd = NULL;
>> + int num_clients, ret, i;
>> +
>> + /*
>> + * Runtime resume one of the FIMC entities to make sure
>> + * the sclk_cam clocks are not globally disabled.
>> + */
>> + for (i = 0; !fd&& i< ARRAY_SIZE(fmd->fimc); i++)
>> + if (fmd->fimc[i])
>> + fd = fmd->fimc[i];
>> + if (!fd)
>> + return -ENXIO;
>> + ret = pm_runtime_get_sync(&fd->pdev->dev);
>> + if (ret< 0)
>> + return ret;
>> +
>> + WARN_ON(pdata->num_clients> ARRAY_SIZE(fmd->sensor));
>> + num_clients = min_t(u32, pdata->num_clients, ARRAY_SIZE(fmd->sensor));
>> +
>> + fmd->num_sensors = num_clients;
>> + for (i = 0; i< num_clients; i++) {
>> + fmd->sensor[i].pdata =&pdata->isp_info[i];
>> + ret = __fimc_md_set_camclk(fmd,&fmd->sensor[i], true);
>> + if (ret)
>> + break;
>> + fmd->sensor[i].subdev =
>> + fimc_md_register_sensor(fmd,&fmd->sensor[i]);
> 
> There is an issue here. Function fimc_md_register_sensor(),
> can return subdev, as also error codes when i2c_get_adapter()
> or NULL when v4l2_i2c_new_subdev_board() fail. But we do not
> invalidate, and assume the return value is valid subdev. It
> will cause kernel NULL pointer exception later in fimc_md_create_links().

Thanks for letting know.
I remember fixing this issue in v2 of the patch set by making 
fimc_md_register_sensor() return NULL on any error, also when
i2c_get_adapter() fails, rather than ERR_PTR value.

Do you really mean that there is a NULL or _invalid_ pointer
dereference in fimc_md_create_links() ?
An oops on a NULL subdev pointer in fmd->sensor[] array seems
impossible as there is a check at the beginning of the loop:

+static int fimc_md_create_links(struct fimc_md *fmd)
+{
+	struct v4l2_subdev *sensor, *csis;
+	struct s5p_fimc_isp_info *pdata;
+	struct fimc_sensor_info *s_info;
+	struct media_entity *source;
+	int fimc_id = 0;
+	int i, pad;
+	int rc = 0;
+
+	for (i = 0; i < fmd->num_sensors; i++) {
+		if (fmd->sensor[i].subdev == NULL)
+			continue;
...

Sorry, I'll be able to only make more test of this on Monday.

--
Thanks,
Sylwester

Re: [PATCH v3 08/19] s5p-fimc: Add the media device driver

[Subash Patel]

Hi Sylwester,

On 07/22/2011 02:29 PM, Sylwester Nawrocki wrote:
> Hi Subash,
>
> On 07/22/2011 08:26 AM, Subash Patel wrote:
>> Hi Sylwester,
>>
>> On 07/04/2011 11:24 PM, Sylwester Nawrocki wrote:
>>> The fimc media device driver is hooked onto "s5p-fimc-md" platform device.
>>> Such a platform device need to be added in a board initialization code
>>> and then camera sensors need to be specified as it's platform data.
>>> The "s5p-fimc-md" device is a top level entity for all FIMC, mipi-csis
>>> and sensor devices.
>>>
>>> Signed-off-by: Sylwester Nawrocki<s.nawrocki@samsung.com>
>>> Signed-off-by: Kyungmin Park<kyungmin.park@samsung.com>
>>> ---
>>> drivers/media/video/Kconfig | 2 +-
>>> drivers/media/video/s5p-fimc/Makefile | 2 +-
>> ...
>>>
>>> /* -----------------------------------------------------*/
>>> /* fimc-capture.c */
>>> diff --git a/drivers/media/video/s5p-fimc/fimc-mdevice.c b/drivers/media/video/s5p-fimc/fimc-mdevice.c
>>> new file mode 100644
>>> index 0000000..10c8d5d
>>> --- /dev/null
>>> +++ b/drivers/media/video/s5p-fimc/fimc-mdevice.c
>>> @@ -0,0 +1,804 @@
>> ...
>>> +
>>> +static int fimc_md_register_sensor_entities(struct fimc_md *fmd)
>>> +{
>>> + struct s5p_platform_fimc *pdata = fmd->pdev->dev.platform_data;
>>> + struct fimc_dev *fd = NULL;
>>> + int num_clients, ret, i;
>>> +
>>> + /*
>>> + * Runtime resume one of the FIMC entities to make sure
>>> + * the sclk_cam clocks are not globally disabled.
>>> + */
>>> + for (i = 0; !fd&&  i<  ARRAY_SIZE(fmd->fimc); i++)
>>> + if (fmd->fimc[i])
>>> + fd = fmd->fimc[i];
>>> + if (!fd)
>>> + return -ENXIO;
>>> + ret = pm_runtime_get_sync(&fd->pdev->dev);
>>> + if (ret<  0)
>>> + return ret;
>>> +
>>> + WARN_ON(pdata->num_clients>  ARRAY_SIZE(fmd->sensor));
>>> + num_clients = min_t(u32, pdata->num_clients, ARRAY_SIZE(fmd->sensor));
>>> +
>>> + fmd->num_sensors = num_clients;
>>> + for (i = 0; i<  num_clients; i++) {
>>> + fmd->sensor[i].pdata =&pdata->isp_info[i];
>>> + ret = __fimc_md_set_camclk(fmd,&fmd->sensor[i], true);
>>> + if (ret)
>>> + break;
>>> + fmd->sensor[i].subdev =
>>> + fimc_md_register_sensor(fmd,&fmd->sensor[i]);
>>
>> There is an issue here. Function fimc_md_register_sensor(),
>> can return subdev, as also error codes when i2c_get_adapter()
>> or NULL when v4l2_i2c_new_subdev_board() fail. But we do not
>> invalidate, and assume the return value is valid subdev. It
>> will cause kernel NULL pointer exception later in fimc_md_create_links().
>
> Thanks for letting know.
> I remember fixing this issue in v2 of the patch set by making
> fimc_md_register_sensor() return NULL on any error, also when
> i2c_get_adapter() fails, rather than ERR_PTR value.
>
> Do you really mean that there is a NULL or _invalid_ pointer
> dereference in fimc_md_create_links() ?
> An oops on a NULL subdev pointer in fmd->sensor[] array seems
> impossible as there is a check at the beginning of the loop:

If you return NULL, then this check will block a crash. In my case, I 
failed to get the i2c_adapter, and ENODEV was returned, which is not 
NULL. Hence I pass through this check, and will crash in

              s_info = v4l2_get_subdev_hostdata(sensor);

I dont have access to your new patch-set. But if you have returned NULL, 
then thats should fix this.

>
> +static int fimc_md_create_links(struct fimc_md *fmd)
> +{
> +	struct v4l2_subdev *sensor, *csis;
> +	struct s5p_fimc_isp_info *pdata;
> +	struct fimc_sensor_info *s_info;
> +	struct media_entity *source;
> +	int fimc_id = 0;
> +	int i, pad;
> +	int rc = 0;
> +
> +	for (i = 0; i<  fmd->num_sensors; i++) {
> +		if (fmd->sensor[i].subdev == NULL)
> +			continue;
> ...
>
> Sorry, I'll be able to only make more test of this on Monday.
>
> --
> Thanks,
> Sylwester
>
>
>
Regards,
Subash
SISO-SLG

Re: [PATCH v3 08/19] s5p-fimc: Add the media device driver

[Sylwester Nawrocki]

On 07/22/2011 11:26 AM, Subash Patel wrote:
>>> ...
>>>> +
>>>> +static int fimc_md_register_sensor_entities(struct fimc_md *fmd)
>>>> +{
>>>> + struct s5p_platform_fimc *pdata = fmd->pdev->dev.platform_data;
>>>> + struct fimc_dev *fd = NULL;
>>>> + int num_clients, ret, i;
>>>> +
>>>> + /*
>>>> + * Runtime resume one of the FIMC entities to make sure
>>>> + * the sclk_cam clocks are not globally disabled.
>>>> + */
>>>> + for (i = 0; !fd&& i< ARRAY_SIZE(fmd->fimc); i++)
>>>> + if (fmd->fimc[i])
>>>> + fd = fmd->fimc[i];
>>>> + if (!fd)
>>>> + return -ENXIO;
>>>> + ret = pm_runtime_get_sync(&fd->pdev->dev);
>>>> + if (ret< 0)
>>>> + return ret;
>>>> +
>>>> + WARN_ON(pdata->num_clients> ARRAY_SIZE(fmd->sensor));
>>>> + num_clients = min_t(u32, pdata->num_clients, ARRAY_SIZE(fmd->sensor));
>>>> +
>>>> + fmd->num_sensors = num_clients;
>>>> + for (i = 0; i< num_clients; i++) {
>>>> + fmd->sensor[i].pdata =&pdata->isp_info[i];
>>>> + ret = __fimc_md_set_camclk(fmd,&fmd->sensor[i], true);
>>>> + if (ret)
>>>> + break;
>>>> + fmd->sensor[i].subdev =
>>>> + fimc_md_register_sensor(fmd,&fmd->sensor[i]);
>>>
>>> There is an issue here. Function fimc_md_register_sensor(),
>>> can return subdev, as also error codes when i2c_get_adapter()
>>> or NULL when v4l2_i2c_new_subdev_board() fail. But we do not
>>> invalidate, and assume the return value is valid subdev. It
>>> will cause kernel NULL pointer exception later in fimc_md_create_links().
>>
>> Thanks for letting know.
>> I remember fixing this issue in v2 of the patch set by making
>> fimc_md_register_sensor() return NULL on any error, also when
>> i2c_get_adapter() fails, rather than ERR_PTR value.
>>
>> Do you really mean that there is a NULL or _invalid_ pointer
>> dereference in fimc_md_create_links() ?
>> An oops on a NULL subdev pointer in fmd->sensor[] array seems
>> impossible as there is a check at the beginning of the loop:
> 
> If you return NULL, then this check will block a crash. In my case,
> I failed to get the i2c_adapter, and ENODEV was returned, which is
> not NULL. Hence I pass through this check, and will crash in
> 
> s_info = v4l2_get_subdev_hostdata(sensor);

I see, I have also gone through this during the tests and this problem
should be fixed already in v3 of the patch-set you've originally replied to.

> 
> I dont have access to your new patch-set. But if you have returned
> NULL, then thats should fix this.

To make sure you use the most up to date version you could pull from:

 git://git.infradead.org/users/kmpark/linux-2.6-samsung s5p-fimc-next

Or here is a gitweb link:
http://git.infradead.org/users/kmpark/linux-2.6-samsung/shortlog/refs/heads/s5p-fimc-next

I intend to keep using the "s5p-fimc-next" branch name for any new
patches for next kernel version. So in case you get any crashes
it might be a good idea to also give the sources there a try.


--
Regards,
Sylwester